I’m running lots of rstan jobs on a linux cluster. As with most HPC environments, the administrators prefer short jobs, i.e. with walltimes < 12hr, and rarely allow longer walltimes. If you have a potentially long job, the recommended strategy is often to break it into a series of smaller jobs, with…

It isn’t possible in Stan yet, but I know the developers are working on it. In the mean time, you might look into a more generic process checkpointing solution like: [image] CRIU Welcome to CRIU, a project to implement checkpoint/restore functionality for Linux. …

[image] dfalster: Is there anyway to in rstan to save the state of the sampler intermittently, and restart sampling of a previously saved job? No

Thanks aaronjig interesting idea. Have you tried this with rstan?

Not yet, but it should be available soon.

Thanks everyone for the response. Glad to hear that this feature is in dev!

Just checking in on this. Has saving state been implemented in stan yet? Has it been implemented in brms as well? If so could someone please refer me to the reference documents for? I have not found them. Thank you.

[image] KAtkin: Has saving state been implemented in stan yet? Not in rstan.

I forget, what are we waiting on for this? @Bob_Carpenter ’s comment above from 2017 suggested that we were close to having this a few years ago. Did something derail this?

@bbbales2 does campfire do this? If so how did you get that working?

Saving state and restarting sampling on cluster computer with strict walltimes

Interfaces RStan

stevebronder September 7, 2020, 8:53pm 10

Didn’t @bbbales2 do something like this with campfire ?

Topic		Replies	Views
Saving stanfit while fitting RStan	5	790	August 22, 2019
Current state of checkpointing in Stan Developers features	27	3371	November 18, 2020
Which Stan interface supports checkpointing? General cmdstan , rstan , stan , cmdstanr	6	1029	May 5, 2021
Saving progress and sampling for more iterations General rstan , brms	1	1329	January 19, 2021
Checkpointing CmdStan sampling Interfaces	24	2438	June 30, 2020

Saving state and restarting sampling on cluster computer with strict walltimes

Related topics