# Checkpointing CmdStan sampling

**URL:** <https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132>\
**Category:** Interfaces\
**Created:** [October 6, 2017, 7:41am UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132 "2017-10-06T07:41:27Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![maedoc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/maedoc/32/205_2.png) [@maedoc](https://discourse.mc-stan.org/u/maedoc)\
**Post date:** [October 6, 2017, 7:41am UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/1 "2017-10-06T07:41:27Z")

</div>

An HPC system I’m using has a walltime limit of 12 hours on batch jobs, which is barely enough for a model I’m working on to get out of warmup. I’d like to checkpoint the CmdStan process and restart it in a second job, but I’m not sure what is the best approach. Some alternatives I’ve thought of

- Use a generic checkpoint & restore utility, but requires more recent Linux kernel than available on the systems in question

- Dump Stan’s internal state to file and read back (not implemented AFAIK)

- Reformat last sample in CSV to rdump format and pass as init=

The last option is easiest, but does not include the valuable information obtained by NUTS during warmup such as the mass matrix, and CmdStan doesn’t expose options to dump or load those. Could this be compensated by a short warmup step?

---

<div class="post-metadata">

**Author:** ![sakrejda](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/sakrejda/32/845_2.png) [@sakrejda](https://discourse.mc-stan.org/u/sakrejda)\
**Post date:** [October 6, 2017, 12:24pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/2 "2017-10-06T12:24:51Z")

</div>

This is close but not implemented yet, I think the mass matrix and stepsizemay be the last that need to be plugged through the interfaces. If they were read from a file implementing it for cmdstan would be pretty easy for someone with a day or two to spare…

---

<div class="post-metadata">

**Author:** ![bbbales2](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bbbales2/32/77_2.png) [@bbbales2](https://discourse.mc-stan.org/u/bbbales2)\
**Post date:** [October 8, 2017, 7:41pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/3 "2017-10-08T19:41:30Z")

</div>

> If they were read from a file implementing it for cmdstan would be pretty easy for someone with a day or two to spare…

I’m down to try (if nobody else is doing yet). I want this feature.

Had a look at the cmdStan source. Looks like the main entry point is [https://github.com/stan-dev/cmdstan/blob/develop/src/cmdstan/command.hpp](https://github.com/stan-dev/cmdstan/blob/develop/src/cmdstan/command.hpp)?

If I did:

1. Add an argument to specify a mass matrix/vector file
2. Read in the mass matrix as a context using get\_var\_context (which looks like it reads the cmdstan Rdump file formats)
3. Pass that context to the hmc\_nuts\_dense\_e/whatever constructors
4. Some sorta tests?

You (or @mitzimorris) see anything missing from that?

And what would tests for this look like?

---

<div class="post-metadata">

**Author:** ![bbbales2](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bbbales2/32/77_2.png) [@bbbales2](https://discourse.mc-stan.org/u/bbbales2)\
**Post date:** [October 8, 2017, 7:50pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/4 "2017-10-08T19:50:29Z")

</div>

And were the mass matrix/stepsize either both gonna be adapted or both set? Or these are separate options (I’d certainly like the second thing)?

---

<div class="post-metadata">

**Author:** ![maedoc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/maedoc/32/205_2.png) [@maedoc](https://discourse.mc-stan.org/u/maedoc)\
**Post date:** [October 8, 2017, 7:53pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/5 "2017-10-08T19:53:34Z")

</div>

It’d probably be useful to reify the init arg to a section, e.g.

```
./model init data=init.R \
        output data=first.csv state=hmc.R \
        sample

./model init data=init.R state=hmc.R \
        output data=second.csv
        sample

```

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [October 8, 2017, 8:12pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/6 "2017-10-08T20:12:56Z")

</div>

And what would tests for this look like?

take a look at the services/sample tests:

- [https://github.com/stan-dev/stan/blob/develop/src/test/unit/services/sample/hmc\_nuts\_diag\_inv\_metric\_test.cpp](https://github.com/stan-dev/stan/blob/develop/src/test/unit/services/sample/hmc_nuts_diag_inv_metric_test.cpp)
- [https://github.com/stan-dev/stan/blob/develop/src/test/unit/services/sample/hmc\_static\_diag\_inv\_metric\_test.cpp](https://github.com/stan-dev/stan/blob/develop/src/test/unit/services/sample/hmc_static_diag_inv_metric_test.cpp)
- [https://github.com/stan-dev/stan/blob/develop/src/test/unit/services/sample/hmc\_nuts\_dense\_inv\_metric\_test.cpp](https://github.com/stan-dev/stan/blob/develop/src/test/unit/services/sample/hmc_nuts_dense_inv_metric_test.cpp)
- [https://github.com/stan-dev/stan/blob/develop/src/test/unit/services/sample/hmc\_static\_dense\_inv\_metric\_test.cpp](https://github.com/stan-dev/stan/blob/develop/src/test/unit/services/sample/hmc_static_dense_inv_metric_test.cpp)

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [October 8, 2017, 8:15pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/7 "2017-10-08T20:15:35Z")

</div>

> [@bbbales2](#):
>
> Had a look at the cmdStan source. Looks like the main entry point is [https://github.com/stan-dev/cmdstan/blob/develop/src/cmdstan/command.hpp](https://github.com/stan-dev/cmdstan/blob/develop/src/cmdstan/command.hpp)

this is correct.  
let me know if you want help on this,  
cheers,  
Mitzi

---

<div class="post-metadata">

**Author:** ![sakrejda](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/sakrejda/32/845_2.png) [@sakrejda](https://discourse.mc-stan.org/u/sakrejda)\
**Post date:** [October 8, 2017, 9:47pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/8 "2017-10-08T21:47:00Z")

</div>

That’s sounds about right to me. If you do have time I’m happy to help  
figure out any problems and do code review.

---

<div class="post-metadata">

**Author:** ![sakrejda](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/sakrejda/32/845_2.png) [@sakrejda](https://discourse.mc-stan.org/u/sakrejda)\
**Post date:** [October 8, 2017, 9:48pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/9 "2017-10-08T21:48:09Z")

</div>

We definitely want tests to make sure that the matrix is read in correctly  
otherwise we’d be getting set up for subtle bud later.

---

<div class="post-metadata">

**Author:** ![bbbales2](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bbbales2/32/77_2.png) [@bbbales2](https://discourse.mc-stan.org/u/bbbales2)\
**Post date:** [October 9, 2017, 3:51pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/10 "2017-10-09T15:51:28Z")

</div>

@mitzimorris @sakrejda cool beans, I’ll have a go at this later in the week.

---

<div class="post-metadata">

**Author:** ![maedoc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/maedoc/32/205_2.png) [@maedoc](https://discourse.mc-stan.org/u/maedoc)\
**Post date:** [June 26, 2018, 12:45pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/11 "2018-06-26T12:45:31Z")

</div>

> [@bbbales2](#):
>
> cool beans, I’ll have a go at this later in the week.

Did this go anywhere? If there’s an old branch lying around, I might be able to finish up / test.

_edit_ oops I see that one can provide a step size and metric for a NUTS run without adaptation. I guess that’s all that’s required, based on the previous posts in this topic.

---

<div class="post-metadata">

**Author:** ![bbbales2](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bbbales2/32/77_2.png) [@bbbales2](https://discourse.mc-stan.org/u/bbbales2)\
**Post date:** [June 26, 2018, 1:16pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/12 "2018-06-26T13:16:30Z")

</div>

Yup, it should be there and working in 2.17.1. Lemme know if you have any trouble getting it to work.

---

<div class="post-metadata">

**Author:** ![philophthalmus](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/philophthalmus/32/3704_2.png) [@philophthalmus](https://discourse.mc-stan.org/u/philophthalmus)\
**Post date:** [February 7, 2019, 12:39am UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/13 "2019-02-07T00:39:53Z")

</div>

I understand that this is an old conversation, but I could not find any newly information on this.

Is there any documentation on checkpointing on CmdStan?

Thank you,

---

<div class="post-metadata">

**Author:** ![maedoc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/maedoc/32/205_2.png) [@maedoc](https://discourse.mc-stan.org/u/maedoc)\
**Post date:** [February 7, 2019, 7:13am UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/14 "2019-02-07T07:13:21Z")

</div>

It’s now possible to parse the mass matrix from a CSV file and use it to initialize sampling in a new chain, but there doesn’t seem to be any tool to automate that (we wrote some custom scripts). Also, this only works after warmup is done; restarting warmup would require doing the warmup schedule in the script outside of CmdStan, if it’s even possible.

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [February 7, 2019, 9:09am UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/15 "2019-02-07T09:09:38Z")

</div>

That’s right… I don’t think that you can checkpoint right now in the sense of exactly restarting sampling. However, you can run one decent warmup and then fire off a few chains which sample using the already computed mass diagonal. This is obviously to some extent dangerous since the convergence diagnostics assume full independence of the chains which is obviously lost to some degree.

---

<div class="post-metadata">

**Author:** ![maedoc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/maedoc/32/205_2.png) [@maedoc](https://discourse.mc-stan.org/u/maedoc)\
**Post date:** [February 7, 2019, 9:11am UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/16 "2019-02-07T09:11:43Z")

</div>

> [@wds15](#):
>
> I don’t think that you can checkpoint right now in the sense of exactly restarting sampling

Hm, this is exactly what I thought we could do: run warmup completely, save a CSV file, parse the mass matrix and last accepted sample, and then start a new chain without warmup, and initialize it with the mass matrix and last sample. Isn’t this sufficient?

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [February 7, 2019, 9:17am UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/17 "2019-02-07T09:17:19Z")

</div>

Last time I looked at this some bits were missing. I think the issue is the state of the random number generator which will be different depending wether you stop or not. However, if you define that all of your Stan runs are stopped at the end of warmup and then you restart as you say, then you can say that things are correctly restarted, yes - it just won’t be the same result without the stop in the middle.

(at least this was my last update on this)

---

<div class="post-metadata">

**Author:** ![philophthalmus](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/philophthalmus/32/3704_2.png) [@philophthalmus](https://discourse.mc-stan.org/u/philophthalmus)\
**Post date:** [February 7, 2019, 3:54pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/18 "2019-02-07T15:54:36Z")

</div>

Thank you, [maedoc](https://discourse.mc-stan.org/u/maedoc) and [wds15](https://discourse.mc-stan.org/u/wds15) for a quick update on this issue.

As I correct in thinking that the custom scripts that [maedoc](https://discourse.mc-stan.org/u/maedoc) refers to are the source cited in a previous reply by [bbbales2](https://discourse.mc-stan.org/u/bbbales2)?

> [@Checkpointing CmdStan sampling](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/3):
>
> If they were read from a file implementing it for cmdstan would be pretty easy for someone with a day or two to spare… I’m down to try (if nobody else is doing yet). I want this feature. Had a look at the cmdStan source. Looks like the main entry point is [https://github.com/stan-dev/cmdstan/blob/develop/src/cmdstan/command.hpp](https://github.com/stan-dev/cmdstan/blob/develop/src/cmdstan/command.hpp)? If I did: Add an argument to specify a mass matrix/vector file Read in the mass matrix as a context using get\_var\_context (which looks like it reads the cmdstan Rd…

I’m just not quite sure how to :

> parse the mass matrix from a CSV file and use it to initialize sampling in a new chain

Thank you,

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [February 7, 2019, 3:58pm UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/19 "2019-02-07T15:58:44Z")

</div>

are we talking about the discussion that went down for this PR? [https://github.com/stan-dev/cmdstan/pull/604](https://github.com/stan-dev/cmdstan/pull/604)

you can run CmdStan with “adapt\_engaged=0” (false) and “num\_warmup=0” plus step size, mass matrix, param inits - but you won’t be at the same place in the RNG. you could use the same seed and use the ‘id’ as a way to advance it many skips down the road… hack hack hack

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [February 8, 2019, 8:18am UTC](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132/20 "2019-02-08T08:18:14Z")

</div>

To get the mass matrix you have to extract a special comment which is written into the csv outputs files. Another way to get this is to use RStan which can output the mass diagonal.

[Next page](https://discourse.mc-stan.org/t/checkpointing-cmdstan-sampling/2132.md?page=2)