# Multithreading with pystan3

**URL:** <https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946>\
**Category:** General\
**Created:** [July 24, 2020, 11:09am UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946 "2020-07-24T11:09:48Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![nerpa](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@nerpa](https://discourse.mc-stan.org/u/nerpa)\
**Post date:** [July 24, 2020, 11:09am UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/1 "2020-07-24T11:09:48Z")

</div>

Hello dear forum,

I am trying to run my Stan model with multithreading, but I saw that it’s a bit complicated with pystan2.18+. From this and other forums, seems like official multithreading is introduced with pystan3 (still Beta version), so I’ve decided to try it. Since my model is really really big, my goal is to parallelize each chain and to submit each chain to multiple cpus. However I couldn’t find anywhere how to implement it in pystan3.  
Can someone here point me to the right direction?

Thank you!

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [July 24, 2020, 11:52am UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/2 "2020-07-24T11:52:25Z")

</div>

We currently are using multiple processes in pystan3.

Do you want to use Stan multithreading functions?

---

<div class="post-metadata">

**Author:** ![nerpa](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@nerpa](https://discourse.mc-stan.org/u/nerpa)\
**Post date:** [July 24, 2020, 11:57am UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/3 "2020-07-24T11:57:25Z")

</div>

That’s my goal and I can allocate a generous number of cpus for each chain. In pystan2.19, even if I follow [this](https://pystan.readthedocs.io/en/latest/threading_support.html) example, I still have only 4 cpus running (1 per chain). So something doesn’t work right…  
Update - I use pystan on CentOS7 linux server.

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [July 24, 2020, 12:10pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/4 "2020-07-24T12:10:01Z")

</div>

So basically you need to define `STAN_NUM_THREADS` environmental variable and use `map_rect` function to use multiple cpu.

And in the compile step add

```
extra_compile_args = ['-pthread', '-DSTAN_THREADS']
```

---

<div class="post-metadata">

**Author:** ![nerpa](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@nerpa](https://discourse.mc-stan.org/u/nerpa)\
**Post date:** [July 24, 2020, 12:13pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/5 "2020-07-24T12:13:56Z")

</div>

So there is no way around rewriting my model to do multithreading? Even in pystan3?

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [July 24, 2020, 1:00pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/6 "2020-07-24T13:00:38Z")

</div>

No.

You need to use `map_rect` or `reduce_sum` (this is possible with CmdStanPy).

---

<div class="post-metadata">

**Author:** ![nerpa](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@nerpa](https://discourse.mc-stan.org/u/nerpa)\
**Post date:** [July 24, 2020, 1:05pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/7 "2020-07-24T13:05:33Z")

</div>

ok thanks! One last question - from reading more I saw that some folks suggest to increase the number of chains and to reduce the number of iterations in each chain to speed up. Is it really recommended?

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [July 24, 2020, 1:13pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/8 "2020-07-24T13:13:32Z")

</div>

between-chain parallelism is always more efficient than within-chain… but you have to go through the warmup for every chain…

---

<div class="post-metadata">

**Author:** ![nerpa](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@nerpa](https://discourse.mc-stan.org/u/nerpa)\
**Post date:** [July 24, 2020, 1:46pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/9 "2020-07-24T13:46:50Z")

</div>

Yes, I thought so. Thanks so much! Rewriting the model it is then!

---

<div class="post-metadata">

**Author:** ![nerpa](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@nerpa](https://discourse.mc-stan.org/u/nerpa)\
**Post date:** [July 24, 2020, 7:28pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/10 "2020-07-24T19:28:13Z")

</div>

I hope someone will pick it up - my really large model is stuck in the first set of warmup iterations for 8 hours. Any ideas what it means and how can I solve it?

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [July 24, 2020, 9:00pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/11 "2020-07-24T21:00:46Z")

</div>

How large is your model?

Is it large in parameters or data?

---

<div class="post-metadata">

**Author:** ![nerpa](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@nerpa](https://discourse.mc-stan.org/u/nerpa)\
**Post date:** [July 24, 2020, 9:16pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/12 "2020-07-24T21:16:17Z")

</div>

Thanks, Ari!  
It’s not huge in parameters - I am fitting ~40 parameters. But the data is pretty large - it’s ~1200 observations (including missing data), each with ~10 different tasks with hundreds of trials each. So I guess that’s a lot.  
Now the weird part is that in the original model I am looping over trials in each task and to make it more efficient, I vectorized it - BUT it takes **more** time after vectorization… hmmmm.

---

<div class="post-metadata">

**Author:** ![nerpa](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@nerpa](https://discourse.mc-stan.org/u/nerpa)\
**Post date:** [July 24, 2020, 9:21pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/13 "2020-07-24T21:21:42Z")

</div>

I see this since 8:40am EST

Gradient evaluation took 0.39 seconds  
1000 transitions using 10 leapfrog steps per transition would take 3900 seconds.  
Adjust your expectations accordingly!

Iteration: 1 / 1000 [0%] (Warmup)  
Iteration: 1 / 1000 [0%] (Warmup)  
Iteration: 1 / 1000 [0%] (Warmup)  
Iteration: 1 / 1000 [0%] (Warmup)

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [July 24, 2020, 10:31pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/14 "2020-07-24T22:31:23Z")

</div>

That sounds interesting.

Maybe ask how to improve your model in a new thread. It is usually a good idea to debug a hard model with others.

Loops in Stan are C++ loops so sometimes vectorization doesn’t help.

---

<div class="post-metadata">

**Author:** ![nerpa](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@nerpa](https://discourse.mc-stan.org/u/nerpa)\
**Post date:** [July 24, 2020, 11:15pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/15 "2020-07-24T23:15:27Z")

</div>

I wouldn’t even know where to begin - it’s a 600 lines code… I’ll start with re-parametrizing it and with implementing the within-chain parallelization. Thanks a lot!

---

<div class="post-metadata">

**Author:** ![dherrera1911](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/dherrera1911/32/16512_2.png) [@dherrera1911](https://discourse.mc-stan.org/u/dherrera1911)\
**Post date:** [July 19, 2023, 6:34pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/16 "2023-07-19T18:34:22Z")

</div>

Could you explain how an environmental variable is set? Is it a Python environment variable? Stan environment variable? OS environment variable?

---

<div class="post-metadata">

**Author:** ![Ezequiel\_Alvarez](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ezequiel_alvarez/32/17362_2.png) [@Ezequiel\_Alvarez](https://discourse.mc-stan.org/u/Ezequiel_Alvarez)\
**Post date:** [September 22, 2024, 10:11pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/17 "2024-09-22T22:11:11Z")

</div>

Hi, this works:

````no-highlight
import os
import sys
os.environ['STAN_NUM_THREADS'] = "16"```

Now a question for you, Ari, or whomever....

I cannot make it to work by including 

````

`extra_compile_args = ['-pthread', '-DSTAN_THREADS']`

```no-highlight

in the arguments of model.build() or model.sample() and make it work. It tells me that

```

TypeError: build() got an unexpected keyword argument ‘extra\_compile\_args’

```no-highlight
or 

```

ValueError: {‘json’: {‘extra\_compile\_args’: [‘Unknown field.’]}}

```no-highlight

respectively.

Someone knows how to make it actually copmile and sample with many threads?
(of course that I've the model with all the 'reduce_sum()' structure, that already works in cmdstan. But I need it to work in pystan.

Thank yoiu!
```

---

<div class="post-metadata">

**Author:** ![Ezequiel\_Alvarez](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ezequiel_alvarez/32/17362_2.png) [@Ezequiel\_Alvarez](https://discourse.mc-stan.org/u/Ezequiel_Alvarez)\
**Post date:** [September 22, 2024, 11:02pm UTC](https://discourse.mc-stan.org/t/multithreading-with-pystan3/16946/18 "2024-09-22T23:02:05Z")

</div>

Hi!

Can you plese clarify this statement: reduce\_sum can be used with pystan3? Or just map\_rect can be used with pystan3?

Thank you!