# Pooled warmup

**URL:** <https://discourse.mc-stan.org/t/pooled-warmup/11796>\
**Category:** Developers\
**Tags:** stan\
**Created:** [November 7, 2019, 6:45pm UTC](https://discourse.mc-stan.org/t/pooled-warmup/11796 "2019-11-07T18:45:56Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [November 7, 2019, 6:45pm UTC](https://discourse.mc-stan.org/t/pooled-warmup/11796/1 "2019-11-07T18:45:56Z")

</div>

@stevebronder asked me to share my current prototype on parallel warmup … maybe this info here is useful for others as well (@bbbales2, @Bob_Carpenter, @yizhang, @betanalpha)

I just pushed to cmdstan and stan repos my pooling warmup prototype.

Right now I am abusing the `STAN_NUM_THREADS` variable to specify the number of chains which are being run. Nothing runs in parallel though yet. I figured thats not yet necessary to do so as I would first like to learn how we should pool the adaptation info.

At the moment I am just pooling the covariances which each chain learns in each window. The stan services framework isn’t really setup to share information between chains such that I had to trick a bit to get the window information.

I got side-tracked with rstan emergency which is why I was a bit silent on this front.

To me the next steps would be: Figure out a benchmark; possibly along the lines I outlined on discourse. Then we tweak the warmup pooling to the point where this is beneficial =\> so the total numerical effort must be lower when we use the pooling the info - otherwise we won’t see any speedups as going parallel means to add some friction inevitably.

Then we can go crazy on specs / design docs and finally we implement it.

Let me know what you think.

Sebastian

---

<div class="post-metadata">

**Author:** ![betanalpha](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/betanalpha/32/4_2.png) [@betanalpha](https://discourse.mc-stan.org/u/betanalpha)\
**Post date:** [November 12, 2019, 4:44am UTC](https://discourse.mc-stan.org/t/pooled-warmup/11796/2 "2019-11-12T04:44:38Z")

</div>

What are the branches?

As I was saying before I think it will be more productive to work out the changes to the services first before worrying about specific adaptation strategies (or parallelizaiton technologies, for that matter).

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [November 12, 2019, 8:07am UTC](https://discourse.mc-stan.org/t/pooled-warmup/11796/3 "2019-11-12T08:07:52Z")

</div>

I never worked on stan services. It took me a while to figure out the logic. Without a prototype I would not be able to make sensible designs.

> **[stan-dev/cmdstan](https://github.com/stan-dev/cmdstan/tree/feature/parallel-warmup)**
>
> CmdStan, the command line interface to Stan. Contribute to stan-dev/cmdstan development by creating an account on GitHub.

> **[stan-dev/stan](https://github.com/stan-dev/stan/tree/proto-parallel-warmup)**
>
> Stan development repository. The master branch contains the current release. The develop branch contains the latest stable development. See the Developer Process Wiki for details. - stan-dev/stan

---

<div class="post-metadata">

**Author:** ![Bob\_Carpenter](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bob_carpenter/32/9230_2.png) [@Bob\_Carpenter](https://discourse.mc-stan.org/u/Bob_Carpenter)\
**Post date:** [November 27, 2019, 7:30pm UTC](https://discourse.mc-stan.org/t/pooled-warmup/11796/4 "2019-11-27T19:30:02Z")

</div>

> [@betanalpha](#):
>
> work out the changes to the services first

You mean for things like specifying number of chains? Right now, each service call is an independent chain. I think the main change we’ll need from the services is a way to deal with multiple chains of output. The input and config should be straightforward assuming it’s shared among the chains.

---

<div class="post-metadata">

**Author:** ![betanalpha](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/betanalpha/32/4_2.png) [@betanalpha](https://discourse.mc-stan.org/u/betanalpha)\
**Post date:** [December 3, 2019, 2:59pm UTC](https://discourse.mc-stan.org/t/pooled-warmup/11796/5 "2019-12-03T14:59:58Z")

</div>

Correct – see also the progression I suggested in the other thread,

> [@MPI framework for parallelized warmups](https://discourse.mc-stan.org/t/mpi-framework-for-parallelized-warmups/11671/23):
>
> Personally I think that we want to prioritize chain parallelization first because it is beneficial in of itself.
> 
> 1. Service routes that run multiple chains and create multiple streams of output.
> 2. Parallelizing those routes, allowing all of the interfaces consistent parallelization without having to implement it themselves.
> 3. Progressive warmup to allow for intra chain communication.
> 4. Adaptive warmup through that intra chain communication.

In addition to setting up new adaptation strategies 1 and 2 alone would allow the interfaces to simplify quite a bit and provide a more coherent user experience.

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [December 3, 2019, 6:44pm UTC](https://discourse.mc-stan.org/t/pooled-warmup/11796/6 "2019-12-03T18:44:37Z")

</div>

Sure…but 1 and 2 are already implemented by all of our interfaces. It’s obviously a win to do it consistent, but it is no win in terms of new functionality.

---

<div class="post-metadata">

**Author:** ![stevebronder](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/stevebronder/32/17242_2.png) [@stevebronder](https://discourse.mc-stan.org/u/stevebronder)\
**Post date:** [December 3, 2019, 7:43pm UTC](https://discourse.mc-stan.org/t/pooled-warmup/11796/7 "2019-12-03T19:43:15Z")

</div>

Don’t we need to do (1) and (2) at the C++ level with tbb to also imbed (3) and (4)?

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [December 3, 2019, 8:43pm UTC](https://discourse.mc-stan.org/t/pooled-warmup/11796/8 "2019-12-03T20:43:55Z")

</div>

We do need them, yes…these are just a lot of work without too much net benefit…but we need them, of course.