# Pystan on multiprocessing

**URL:** <https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561>\
**Category:** Modeling\
**Tags:** specification\
**Created:** [November 15, 2018, 11:16pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561 "2018-11-15T23:16:50Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chao](https://avatars.discourse-cdn.com/v4/letter/c/839c29/32.png) [@Chao](https://discourse.mc-stan.org/u/Chao)\
**Post date:** [November 15, 2018, 11:16pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/1 "2018-11-15T23:16:50Z")

</div>

Hi,  
Here I am generating mixture Gaussian model data and using pystan as a model. I want to run pystan sampling part in multiprocessing, and I put the ‘n\_jobs=20’ since my computer has 10 cores. But on the ‘task manager’, the CPU is still something like 5.9%, which means the code did not run in multiprocessing. So anyone can help me to write the code in multiprocessing? Thanks. Below is my code.

from **future** import division  
import pystan  
import pymc3 as pm  
from scipy import stats  
import matplotlib.pyplot as plt  
import theano.tensor as tt  
import seaborn as sns  
import numpy as np  
import os  
import sys

os.environ[‘STAN\_NUM\_THREADS’] = “20”

# generate the data

component=np.random.choice([0,1],size=50)  
x=[]  
loc=[[1,5],[5,2]]  
s1=1  
s2=1  
ro=0.5  
scale=[[s1\*s1,ro\*s1\*s2],[ro_s1_s2,s2\*s2]]  
scale=[scale,scale]  
for i in component:  
x.append(stats.multivariate\_normal.rvs(mean=loc[i],cov=scale[i]))  
x=np.array(x)

model=’’’  
data {  
int D; //number of dimensions  
int K; //number of gaussians  
int N; //number of data  
vector[D] y[N]; //data  
//real con[N]; //co ncentration

}

parameters {  
simplex[K] theta; //mixing proportions  
ordered[D] mu[K]; //mixture component means  
cholesky\_factor\_corr[D] L[K]; //cholesky factor of covariance  
}

transformed parameters{  
cholesky\_factor\_corr[D] cov[K];  
real sx[K];  
real sy[K];  
real ro[K];  
// vector[N] lamba;  
cov=L;  
for (i in 1:K){  
sx[i]=sqrt(cov[i,1,1]);  
sy[i]=sqrt(cov[i,2,2]);  
ro[i]=cov[i,1,2]/sqrt(cov[i,1,1])/sqrt(cov[i,2,2]);  
}  
// for (i in 1:N){  
// lamba[i]=1/(theta[1]_(1./2./sx[1]/sy[1]/sqrt(1-ro[1]ro[1])exp(-1./2./(1-ro[1]ro[1])((y[i,1]-mu[1,1])(y[i,1]-mu[1,1])/sx[1]/sx[1])+(y[i,2]-mu[1,2])(y[i,2]-mu[1,2])/sy[1]/sy[1]-2.ro[1](y[i,1]-mu[1,1])_(y[i,2]-mu[1,2])/sx[1]/sy[1]))+theta[2]_(1./2./sx[2]/sy[2]/sqrt(1-ro[2]ro[2])exp(-1./2./(1-ro[2]ro[2])((y[i,1]-mu[2,1])(y[i,1]-mu[2,1])/sx[2]/sx[2])+(y[i,2]-mu[2,2])(y[i,2]-mu[2,2])/sy[2]/sy[2]-2.ro[2](y[i,1]-mu[2,1])_(y[i,2]-mu[2,2])/sx[2]/sy[2])));  
//}  
}

model {  
real ps[K];

for(k in 1:K){  
mu[k] ~ uniform(-10,10);  
L[k] ~ lkj\_corr\_cholesky(2);  
//con ~ exponential(lamba);

}

for (n in 1:N){  
for (k in 1:K){  
ps[k] = log(theta[k])+multi\_normal\_cholesky\_lpdf(y[n] | mu[k], L[k]); //increment log probability of the gaussian  
}  
target += log\_sum\_exp(ps);  
}  
// for(i in 1:N){  
// target += - lamba[i]\*con[i];  
//}

}

‘’’  
if **name** == ‘ **main** ’:  
dat={‘N’:50,‘y’:x,‘D’:2,‘K’:2}

fit=pystan.stan(model\_code=model, data=dat, iter=5000, warmup=500, chains=1,n\_jobs=20)

print(fit)

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [November 15, 2018, 11:27pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/2 "2018-11-15T23:27:46Z")

</div>

For threading, you need to use map\_rect function. See [https://github.com/stan-dev/math/wiki/Threading-Support](https://github.com/stan-dev/math/wiki/Threading-Support)

And for a good tutorial [https://github.com/rmcelreath/cmdstan\_map\_rect\_tutorial](https://github.com/rmcelreath/cmdstan_map_rect_tutorial)

On PyStan n\_jobs and num\_stan\_threads are separate things.

```
n_jobs <= chains

```

See [https://pystan.readthedocs.io/en/latest/threading\_support.html](https://pystan.readthedocs.io/en/latest/threading_support.html)

ps. To format your multiline codeblock use 3 ``` at start and end.

---

<div class="post-metadata">

**Author:** ![Chao](https://avatars.discourse-cdn.com/v4/letter/c/839c29/32.png) [@Chao](https://discourse.mc-stan.org/u/Chao)\
**Post date:** [November 16, 2018, 2:40am UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/3 "2018-11-16T02:40:40Z")

</div>

Hi,  
I have added a map\_rect function (the bold code), but it gives me a error. Could you help me fix that? Thanks.  
My new code is below:

* * *

```stan
from __future__ import division
import pystan
import pymc3 as pm
from scipy import stats
import matplotlib.pyplot as plt
import theano.tensor as tt
import seaborn as sns
import numpy as np
component=np.random.choice([0,1],size=500)
x=[]
loc=[[1,5],[5,2]]
s1=1
s2=1
ro=0.5
scale=[[s1*s1,ro*s1*s2],[ro*s1*s2,s2*s2]]
scale=[scale,scale]

* List item

for i in component:
  x.append(stats.multivariate_normal.rvs(mean=loc[i],cov=scale[i]))
x=np.array(x)

c=[]
for i in range(len(x)):
 c.append(stats.expon.rvs(scale=1/(0.5*stats.multivariate_normal.pdf(x[i],mean=loc[0],cov=scale[0])+0.5*stats.multivariate_normal.pdf(x[i],mean=loc[1],cov=scale[1]))))

x=np.array(x)
**model='''**
 **functions {**
**vector bl_glm(real[] theta, vector[] mu, cholesky_factor_corr[] L, vector[] y) {**
**real K=theta.size();**
**real N=y.size();**
**real ps[K];**
**for (k in 1:K){**
**ps[k] = log(theta[k])+multi_normal_cholesky_lpdf(y | mu[k], L[k]); //increment log probability of the gaussian**
 **}**
**return [ps]';**
 **}**

data {
 int D; //number of dimensions
 int K; //number of gaussians
 int N; //number of data
 vector[D] y[N]; //data

}

parameters {
 real[K] theta; //mixing proportions
 vector[D] mu[K]; //mixture component means
 cholesky_factor_corr[D] L[K]; //cholesky factor of covariance
}

model {
 real ps[K];

 for(k in 1:K){
 mu[k] ~ normal(0,4);
 L[k] ~ lkj_corr_cholesky(4);
 }

 
  target += sum(map_rect(bl_glm, mu, L, y, con);
}

'''

dat={'N':500,'y':x,'D':2,'K':2}#,'con':c}
extra_compile_args = ['-pthread', '-DSTAN_THREADS']

stan_model = pystan.StanModel(
   model_code=model,
   extra_compile_args=extra_compile_args)

fit = stan_model.sampling(data=dat, n_jobs=4)

print(fit)

The error is: 
Forget to mention the error is below:
ValueError: Failed to parse Stan model 'anon_model_fc39d0e82b2a948e69d5573a0f8031ea'. Error message:
SYNTAX ERROR, MESSAGE(S) FROM PARSER:

  error in 'unknown file name' at line 3, column 42
  -------------------------------------------------
     1: 
     2: functions {
     3: vector bl_glm(real[] theta, vector[] mu, cholesky_factor_corr[] L, vector[] y) {
                                                 ^
     4: real K=theta.size();
  -------------------------------------------------

PARSER EXPECTED: <argument declaration or close paren ) to end argument declarations>
```

---

<div class="post-metadata">

**Author:** ![Chao](https://avatars.discourse-cdn.com/v4/letter/c/839c29/32.png) [@Chao](https://discourse.mc-stan.org/u/Chao)\
**Post date:** [November 16, 2018, 6:33pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/5 "2018-11-16T18:33:07Z")

</div>

Hello,  
Could you help with map\_rect function? I have wrote below and it gives errors. Thanks.

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [November 16, 2018, 6:49pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/6 "2018-11-16T18:49:12Z")

</div>

Hi,

This is missing parenthesis.

`target += sum(map_rect(bl_glm, mu, L, y, con);`

But do you really need `bl_glm` function?

You need to create a function that calculates an item in `ps`.

For description of the issue see tutorial [https://github.com/rmcelreath/cmdstan\_map\_rect\_tutorial](https://github.com/rmcelreath/cmdstan_map_rect_tutorial)

and our Stan 2.18 User Guide (chapter 22)

> **[stan-dev/cmdstan](https://github.com/stan-dev/cmdstan/releases)**
>
> CmdStan, the command line interface to Stan. Contribute to stan-dev/cmdstan development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![Chao](https://avatars.discourse-cdn.com/v4/letter/c/839c29/32.png) [@Chao](https://discourse.mc-stan.org/u/Chao)\
**Post date:** [November 16, 2018, 7:05pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/7 "2018-11-16T19:05:20Z")

</div>

Hi,  
why you think bl\_glm function is not necessary? Is it used for multiprocess? I have add the parenthesis, but it still give the same error. Could you help check whether the way below is right?

* * *

```stan
functions {
vector bl_glm(real[] theta, vector[] mu, cholesky_factor_corr[] L, vector[] y) {real K=theta.size(); 
real N=y.size();
```

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [November 16, 2018, 7:09pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/8 "2018-11-16T19:09:33Z")

</div>

Please read the User guide for map reduce. Also, there are more discussion on the Discourse on the parallel computing.

In your original example there was no `bl_glm` but it was in the example showing `map_rect` usage.

---

<div class="post-metadata">

**Author:** ![Chao](https://avatars.discourse-cdn.com/v4/letter/c/839c29/32.png) [@Chao](https://discourse.mc-stan.org/u/Chao)\
**Post date:** [November 16, 2018, 7:11pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/9 "2018-11-16T19:11:20Z")

</div>

Also, just want to make sure we are on the same page, I am running the code on windows system and using python 2.7.13. I read some website says pystan could not work on multiprocessing on windows. Do you think could that be a issue? Thanks.

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [November 16, 2018, 7:17pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/10 "2018-11-16T19:17:42Z")

</div>

Yes, multithreading is not supported on Windows.

Multiprocessing (`n_jobs`) works with MingW-w64 by distributing chains on parallel cpu. (Python 3). Python 2 does not support multiprocessing on Windows.

Remember to call sampling under \_\_main\_\_ block

```
if __name__ == ' __main__':
    code here

```

see [https://pystan.readthedocs.io/en/latest/windows.html](https://pystan.readthedocs.io/en/latest/windows.html)

---

<div class="post-metadata">

**Author:** ![mjack](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mjack/32/3692_2.png) [@mjack](https://discourse.mc-stan.org/u/mjack)\
**Post date:** [November 16, 2018, 7:25pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/11 "2018-11-16T19:25:03Z")

</div>

So multithreading is for running iterations in different CPU’s right? Do you have any suggestions for running multithreading on Windows? Or is it not possible at all? As an alternative, is multithreading in stan in R possible on Windows?

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [November 16, 2018, 7:35pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/12 "2018-11-16T19:35:23Z")

</div>

Multiprocessing runs chains on individual cpu.

Multithreading parallelizes specific calculations to multiple cpu.

If you really want to do it on Windows there is a experimental (not official) tutorial that I made. It uses MSVC 2017 + clang to circumvent the thread\_local bug on MinGW-w64/winpthreads.

See [Multithreading on Windows with PyStan 2.18+](https://discourse.mc-stan.org/t/multithreading-on-windows-with-pystan-2-18/5726)

---

<div class="post-metadata">

**Author:** ![mjack](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mjack/32/3692_2.png) [@mjack](https://discourse.mc-stan.org/u/mjack)\
**Post date:** [December 3, 2018, 4:47pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/13 "2018-12-03T16:47:43Z")

</div>

Hi. I am wondering if it is possible to hand write a C++ code for multithreading in the pystan model=’’’ part of the code instead of trying the experimental way to be able to use multithreading in the module? When my code includes a lot of data, it takes ~30 minutes to run on Windows with only 10,000 iterations so I would like to try the best way to decrease the time. Thanks!

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [December 3, 2018, 5:23pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/14 "2018-12-03T17:23:52Z")

</div>

What do you mean by multithreading? Not `map_rect`?

Theading things are done in Stan side, so you would need to touch Stan source. For this I recommend that you clone our repo and install in place.

```
git clone --recursive https://github.com/stan-dev/pystan
cd pystan
pip install -e .

```

For threading discussion, please read other discussion going on this forum.

Ps. PyStan 3 uses threading to run chains parallel, but that is not probably the thing you are looking for.

Edit. You get cpp code with `stanc` and you can edit this after saving. Then you can read the modified cpp code and replace in stanc output that can then go to `StanModel` init.

---

<div class="post-metadata">

**Author:** ![mjack](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mjack/32/3692_2.png) [@mjack](https://discourse.mc-stan.org/u/mjack)\
**Post date:** [December 3, 2018, 5:31pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/15 "2018-12-03T17:31:56Z")

</div>

Yes I am concerned about running iterations on different CPU, not different chains on different CPUs. So there is no way to write the multithreading for running different iterations in the model code? I would have to edit the source code?

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [December 3, 2018, 7:59pm UTC](https://discourse.mc-stan.org/t/pystan-on-multiprocessing/6561/16 "2018-12-03T19:59:03Z")

</div>

I mean that currently `map_rect` is the way to go if your model can be parallelized.

Otherwise you need to handle C++ and Stan types.

There might be other ways to speed up your model (setting good priors etc).