I am fitting a hierarchical Hidden-Markov model with cmdstanr. I am using data from 20 subjects, each with around a 1000 trials (distributed across 20 block). The forward algorithm is implemented directly, hence I have to store about 20 x 20 x 100 variables for the forward-variables (and an equal number for the backward- and forward-backward smoothed ones as I wish to estimate by-trial probabilities).
Fitting the model works just fine and with lower number of subjects/blocks/trials, there is no issue whatsoever. However, when using the full dataset, cmdstanr cannot read back its own output files. In fact, it gets stuck in some obscure computation when trying to access any of the fitted model variables (such as using fit$draws() and even if trying to use fit$save_object(). This is the case, even if I use fit$optimize() instead of fit$sample(), even though the resulting output file from the fit (attached) is has only a single line (but a lot of variables, obviously) and is only about 9 MB in size.
@ihrke I was able to reproduce this using the file you shared (thanks for that). The problem is happening when CmdStanR calls posterior::subset_draws() towards the end of cmdstanr::read_cmdstan_csv.
I made a branch that has a temporary fix for this when reading in the csv after optimization (I think ultimately we need to fix posterior::subset_draws()):
This should get it to work with optimization (at least it allows me to use read_cmdstan_csv() successfully with the file you provided). Unfortunately Iām not sure where the problem is happening when youāre using sampling but if you share that csv I can probably track it down.
Edit: @ihrke I updated the branch to avoid using subset_draws also for sampling so perhaps it will solve that for you too but Iām not 100% sure.
Yeah, this is a known problem. A workaround for now is to use cmdstanr::read_cmdstan_csv() to read the csv files directly instead of using $draws().
Edit: @jonah notes that Iām wrong about this. Just for posterity, it is also currently the case that read_cmdstan_csv works for sampling fits with large numbers of parameters, but $draws() does not.
I tried using the branch with your fix and reading the csv now seems to work fine. I use the following convenience function to read the whole fit into memory (before storing it as an .RData file) and it is lightning fast (as opposed to taking ages before your fix).
However, I cannot use the fit$summary() or fit$draws() function for this object as I used to. The error I get is
> mod_opt_probed.r$draws("gmu")
Error in `[.default`(private$draws_, , variables, drop = FALSE) :
subscript out of bounds
> mod_opt_probed.r$summary()
Error: Can't subset columns that don't exist.
x Columns `variable` and `mean` don't exist.
Run `rlang::last_error()` to see where the error occurred.
@jsocolar Iām hopeful that with posterior::subset_draws() now fixed this will drastically improve the speed of $draws() with many parameters. I havenāt done any rigorous testing yet though.
My hunch is that youāll need to rebuild the R6 object. Unfortunately if we update a method it doesnāt update the methods associated with existing R6 objects.
However, there may be an alternative: do you by any chance still have the CSV files or were those just written to temp files? If you still have the CSV files associated with the old R6 object then you can recreate the R6 object without having to rerun the model using as_cmdstan_fit(paths_to_csv_files). Then the resulting fit object would use the latest draws method.
Cool, thanks for trying. You might also try using format = "draws_list" when running as_cmdstan_fit. According to @rok_cesnovar thatās the most efficient format to use if there are a ton of parameters. (That will just affect how the draws are stored internally. If you then use draws() it will use the regular default of ādraws_arrayā unless you specify a different format.)
$draws() is now blazing fast on a fit where it was previously unusable (250K parameters, now takes about 30 seconds, previously I killed it after 90 minutes).