Saving posterior in a binary format to overcome reading overflows for huge generated quantities? Anything close to pdocution grade?

Developers
1

Here is the inconvenience I ran into lately:

  • I run a proprietary ODE chemical reaction network. So my output are multiresponses.
  • Each draw of my generated quantities have 3 arrays of [5k, 4, 4]
  • .csv files are huge to a point where I run into memory issues when importing them back (each chain 5GB).

Anyone who tried to fork/branch the stan compiler or extend it such that a binary save would be possible? Reason: double precision encoding instead of flat character strings should save on filesize considerably.

I am barely at script kiddie level when it comes to C++ coding.

2

The nutpie sampler from PyMC supports Stan models and can save the samples to a couple binary formats. There is also a community developed R wrapper, though I’m not sure if it supports all the same options

3

You could try converting the CSV to parquet (can be done on-disk):

Then from the parquet file you can use arrow to read subsets of variables, do on-disk processing, etc.