Analyzing the posterior prediction samples

Modeling
21

this error means that CmdStan ran and the sampler ran produced a set of Stan CSV files.
CmdStanPy expected that all CSV files would have a certain number of rows, columns, but they didn’t for some reason.

did you specify an output_dir when you ran the sample method? or do you still have the Python session running? the question is what’s going on with the CSV files and is this related to problems in the generated quantities block.

if you re-run the sample method, please specify show_console=True - this will give more information.

22

the data being fed by a Python dictionary doesn’t matter, except that it’s a big dictionary to keep in memory. CmdStanPy’s method write_stan_json is being called under-the-hood to create the JSON data file for CmdStan.

23

I did not specific an output_dir. should i have? It also says show_console is an unexpected argument.

24

yes, you should specify an output dir - which will be created if it doesn’t exist - and the result will contain both the Stan CSV output files and the corresponding CmdStan messages to stdout and stderr.

e.g.:

fit = model.sample(data=data, output_dir=os.path.join('.', 'bar'))

creates directory bar, which has contains a bunch of output files (here running bernoulli test model):

bernoulli-20211026164706-1-stdout.txt
bernoulli-20211026164706-1.csv
bernoulli-20211026164706-2-stdout.txt
bernoulli-20211026164706-2.csv
bernoulli-20211026164706-3-stdout.txt
bernoulli-20211026164706-3.csv
bernoulli-20211026164706-4-stdout.txt
bernoulli-20211026164706-4.csv

the argument show_console has been added as part of release 1.0.0.

with the Stan CSV files and the console outputs, check that:

  • all the .csv files have the same number of lines - (unix utility wc): wc -l some_output_dir/*.csv
  • all the .csv files end with series of comments giving timings
# 
#  Elapsed Time: 0.004 seconds (Warm-up)
#                0.012 seconds (Sampling)
#                0.016 seconds (Total)
#
  • the .txt files don’t have any error messages
25

Putting the output directory had it run with no errors.

26

what’s the total amount of storage for the Stan CSV output files?

I’m not sure what’s going on - filed an issue to investigate: check/warn about filling up _TMPDIR · Issue #485 · stan-dev/cmdstanpy · GitHub

maybe tmp storage truncated or deleted some of the output files? that’s kinda rude - not sure how we can check for this - by the time it happens, it’ll be too late to do anything, in which case, all we can do is add something to the docs.

glad you can run and fit your model!

27

Over 400 MB’s. I had to use my other hard drive because my c drive was full. I also tried to use box but python/Stan kept erroring out.

Jordan