Saving stanfit while fitting

Hi, I am running stan for long time on slurm and sometimes it gets terminated due to time limit (forced termination). Can I save stanfit while running so that I have information and run again from where it stopped?
Thanks a lot for your help!

For R or Python?

Edit. For long I would recommend CmdStan.

It is for R.
I was wondering if there is an option/function that can write while fitting in rstan.

For RStan, I don’t think you set inverse metric yet. For CmdStan and PyStan you can.

CmdStan saves the samples to a csv file, so if you can start sampling, and after the timeout you can “restart” by giving the inv_metric, stepsize and last location (+ random seed). This does need some manual editing.

And it is not the same as doing the sampling in one sweep, because the state of the random number generator is reset. But this should not affect the samples. And it is reproducible if you use the deterministic seeds.

There is similar idea in https://github.com/ahartikainen/fit_check_fit_loop/blob/master/MoreSamples_round1/increase_draws_in_steps.ipynb

(done with PyStan + ArviZ)

1 Like

Interesting! I didn’t know cmdstan could resume sampling; makes me wonder if I should switch to using cmdstan behind the scenes in ezStan to enable this feature for R (presumably what the fabled cmdStanR is going to do anyway?)

CmdStanPy can do this currently.

But if you or someone else could implement this for RStan, it would be great.

For PyStan I implemented this so that metric info --> tempfile --> pathinfo for sampler

I think that could be doable for RStan too.

3 Likes