Hello! Is there a “current” best practice to limit the amount of data written to .csv files?
In particular, in searching the Forums, there (in R) is an include = True, pars = "..." capability that only returns specific parameters from the model. Further sleuthing seems to show that this doesn’t prevent the data from being written, but only prevents which data is returned to R.
Is that correct? Also, is there a way to prevent/limit what is written to csv?
Context: I am running a very large model in a docker container and writing out all of the parameter values is killing the memory of the container (I could spin up more memory, but that is $$$ for memory I do not need if I can suppress the output).
I’m also curious about this; I haven’t seen anything in the cmdstan documentation to indicate how to do it.
If you have a lot of transformed parameters, one possible option may be to move them to the model block , and re-compute the transformations after the fact in R/python.
correct, the core Stan services always write all block-level variables in the parameters, transformed paramters, and generated quantities blocks to the CSV file.
in addition to the above suggestion re transformed params,
if you’re using the generated quantities block to get additional quantities of interest, you should be similarly careful to use local blocks when doing any computation where you’re declaring intermediate or housekeeping variables.
1 Like