# Cmdstanpy, mpi speedup

**URL:** <https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115>\
**Category:** Developers\
**Created:** [November 1, 2024, 11:26am UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115 "2024-11-01T11:26:24Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 1, 2024, 11:26am UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/1 "2024-11-01T11:26:24Z")

</div>

Hi all,

I have connected a stan program to MPI in hopes for a speedup. I have 10 nodes running (verified by seeing their CPUs max out), but I am not seeing a 10x speedup. I do have a small network latency, so not sure if that’s throttling me a bit. Here is the output of the running program:

Starting mpi process  
method = sample (Default)  
sample  
num\_samples = 1000 (Default)  
num\_warmup = 500  
save\_warmup = false (Default)  
thin = 1 (Default)  
adapt  
engaged = true (Default)  
gamma = 0.05 (Default)  
delta = 0.8 (Default)  
kappa = 0.75 (Default)  
t0 = 10 (Default)  
init\_buffer = 75 (Default)  
term\_buffer = 50 (Default)  
window = 25 (Default)  
save\_metric = false (Default)  
algorithm = hmc (Default)  
hmc  
engine = nuts (Default)  
nuts  
max\_depth = 10 (Default)  
metric = diag\_e (Default)  
metric\_file = (Default)  
stepsize = 1 (Default)  
stepsize\_jitter = 0 (Default)  
num\_chains = 1 (Default)  
id = 1 (Default)  
data  
file = stan\_data.json  
init = 2 (Default)  
random  
seed = 3950355111 (Default)  
output  
file = blah.csv  
diagnostic\_file = (Default)  
refresh = 100 (Default)  
sig\_figs = -1 (Default)  
profile\_file = profile.csv (Default)  
save\_cmdstan\_config = false (Default)  
num\_threads = 1 (Default)  
mpi\_enabled = 1

Iteration: 1 / 1500 [0%] (Warmup)

is there a specific config change I need to explore to benefit from a MPI speedup further? Thank you,

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 1, 2024, 2:49pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/2 "2024-11-01T14:49:21Z")

</div>

Did some reading… it looks like MPI is designed for parallelizing across multiple chains, not for speeding up a single chain. Is this correct?

If so, is there a way to speed up a single chain?..

---

<div class="post-metadata">

**Author:** ![jsocolar](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jsocolar/32/2486_2.png) [@jsocolar](https://discourse.mc-stan.org/u/jsocolar)\
**Post date:** [November 1, 2024, 5:10pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/3 "2024-11-01T17:10:31Z")

</div>

> [@e32432423](#):
>
> is there a way to speed up a single chain

Yes! See [Parallelization](https://mc-stan.org/docs/cmdstan-guide/parallelization.html)

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 1, 2024, 5:22pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/4 "2024-11-01T17:22:38Z")

</div>

thank you. the piece I may have missed is this:

In order to use multi-processing with MPI in a Stan model, the models must be rewritten to use the [`map_rect` function](https://mc-stan.org/docs/functions-reference/higher-order_functions.html#functions-map). By using MPI, the model can be parallelized across multiple cores or a cluster.

I will do this next

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 4, 2024, 7:00pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/5 "2024-11-04T19:00:48Z")

</div>

hello, I re-wrote my function to use map\_rect() function. MPI configured.

When it comes to this step:  
fit = model.sample(data=stan\_data, \*\*rc.sample\_kwargs)

I now replace it with:  
fit = cmdstanpy.from\_csv(‘output.csv’) (the output.csv is the output from the MPI process, seems to be populated with all the outputs)

I get this error:

Invalid or corrupt Stan CSV output file.

How do I get a MPI output file to use for fitting? thanks,

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 4, 2024, 8:42pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/6 "2024-11-04T20:42:30Z")

</div>

it appears that the from\_csv() is a bit brittle because the

subprocesses() generation of csv\_output.csv creates a csv file but it may not be exactly how cmdstanpy would expect (for example, there are true/false values instead of 1/0 values that the from\_csv() does not like)

so I stumbled on this:

> <https://github.com/stan-dev/cmdstanpy/issues/360>
>
> \#### Summary:
> CmdStan does not work with open MPI c++ compiler feature
> 
> 
> \###…# Description:
> When compiling a stan program to compile and use open MPI, as:
> 
> \`\`\`
> cpp\_options = {
> 'STAN\_MPI' : True ,      
> }
> 
> sm = CmdStanModel(stan\_file='./stan\_files/BNN.stan', cpp\_options = cpp\_options )
> 
> \`\`\`
> 
> The model fails to successfully compile the program (after installing open mpi)
> 
> \#### Additional Information:
> The solution was to add the following lines to the cmdstan/make/local file:
> 
> \`\`\`
> TBB\_CXX\_TYPE=gcc
> CXX=mpicxx
> \`\`\`
> 
> So in order to allow cmdstanpy to compile using mpi, the class \`CompilerOptions\` in file \`compiler\_opts\` should add those flags
> 
> \##### IMPORTANT NOTE
> 
> With this modification, the mpi compilation is invoqued. However, the program fails to compile but I think that is on the cmdstan side, and not the cmdstanpy. Will figure that out and come back.

here I found I can do

```no-highlight
cpp_options = {
                'STAN_MPI' : True ,      
                'CXX': 'mpicxx',
                'TBB_CXX_TYPE': 'gcc'
              }

sm = CmdStanModel(stan_file='./stan_files/BNN.stan', cpp_options = cpp_options, compile='force')

```

but I don’t have control of # of nodes for MPI to run,

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 4, 2024, 9:03pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/7 "2024-11-04T21:03:12Z")

</div>

using cpp\_options ^ as seen above, doesn’t run the other nodes in my MPI network.

the subprocess.run() worked as intended, but I can’t seem to get cmdstanpy to work with its output to create the fit. anyone able to get MPI working with this?

---

<div class="post-metadata">

**Author:** ![Bob\_Carpenter](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bob_carpenter/32/9230_2.png) [@Bob\_Carpenter](https://discourse.mc-stan.org/u/Bob_Carpenter)\
**Post date:** [November 4, 2024, 9:33pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/8 "2024-11-04T21:33:32Z")

</div>

Hi, @e32432423 (?—shouldn’t the last two digits be “32”?).

What platform are you running on? I believe this is much more challenging with Windows.

You can use multiple threads within a single machine, or MPI across machines. Usually it’s better to scale up on one machine until you run out of room and only then scale out using MPI, which can be much slower due to network latency. MPI is only going to pay off across machines if you have more compute to do than network latency, which will depend on the Stan program and the hardware setup.

It’s generally helpful to us debug if you provide a reproducible example. It looks like you’re running at least something through python?

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 4, 2024, 10:07pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/9 "2024-11-04T22:07:08Z")

</div>

I chose “23” at the end to throw you off Bob ;).

this is linux.

i found this:

> **[Run Stan's MCMC algorithms with MPI — model-method-sample\_mpi](https://mc-stan.org/cmdstanr/reference/model-method-sample_mpi.html)**
>
> The $sample\_mpi() method of a CmdStanModel object is
> identical to the $sample() method but with support for
> MPI (message passing interface). The target audience for MPI are
> those with large computer clusters. For other users, the
> $sample() method...

for cmdstanpy equivalent… it looks like I need to stick with cmdstanpy so I found the following:

```
cpp_options = {
    'STAN_MPI': True,
    'CXX': 'mpicxx',
    'TBB_CXX_TYPE': 'gcc'
}
model = cmdstanpy.CmdStanModel(stan_file=stan_file_path, cpp_options=cpp_options, force_compile=True)

```

in cmdstanr world, the next step would be:  
fit = model.sample\_mpi(data=stan\_data, mpi\_cmd = “mpiexec”,mpi\_args = “-n 4”)

for running  
mpiexec -n 4 model\_executable

but in python, any attempt at that i get  
AttributeError: ‘CmdStanModel’ object has no attribute ‘sample\_mpi’

as there doesn’t seem to be any matching documentation for cmdstanpy equivalents  
[https://mc-stan.org/cmdstanpy/](https://mc-stan.org/cmdstanpy/)

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 4, 2024, 10:28pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/10 "2024-11-04T22:28:17Z")

</div>

Instead of running `sample()` from Python, i tried to run the command manually:

subprocess.run([  
‘mpirun’, ‘-np’, ‘4’, ‘-f’, ‘node\_config\_file’, model.exe\_file,  
‘sample’, ‘num\_samples=1000’, ‘num\_warmup=500’,  
‘data’, ‘file={}’.format(stan\_data\_file),  
‘output’, ‘file=output.csv’  
])

and then  
fit = cmdstanpy.from\_csv(‘output.csv’) but the from\_csv() is brittle and the outputted csv from subprocess.run() doesn’t match exactly to what from\_csv() is looking for (e.g., true/false instead of 1/0, and other things)

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 4, 2024, 10:52pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/11 "2024-11-04T22:52:45Z")

</div>

I think this is a general cmdstanpy + mpi issue. I cannot change the title. There are 2 possible solutions:

1. ensure the community can run the command:  
mpiexec -n 4 -f node\_config\_file model.exe\_file

within the cmdstanpy syntaxes, where node\_config\_file tells how many threads per node to run, or

1. the cmdstanpy.from\_csv() needs to read the csv outputted from the subprocess.run() command in above post.

I would imagine addressing either of these will solve this general problem for anyone trying to use mpi+cmdstanpy. There is no obvious cmdstanpy equivalent of .sample\_mpi() (with mpi\_args argument) from cmdstanr world.

---

<div class="post-metadata">

**Author:** ![WardBrian](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wardbrian/32/12078_2.png) [@WardBrian](https://discourse.mc-stan.org/u/WardBrian)\
**Post date:** [November 5, 2024, 2:28pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/12 "2024-11-05T14:28:14Z")

</div>

> [@e32432423](#):
>
> but the from\_csv() is brittle and the outputted csv from subprocess.run() doesn’t match exactly to what from\_csv() is looking for (e.g., true/false instead of 1/0, and other things)

Are you using the latest cmdstanpy?

Have you been able to run any non-MPI sampling successfully?

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 5, 2024, 2:54pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/13 "2024-11-05T14:54:13Z")

</div>

my cmdstanpy version is  
1.2.2  
I notice there is 1.2.4. I will try it. But

1. what is the syntax such that I can run a multi-node mpi command such as this in cmdstanpy, like cmdstanr allows?  
mpiexec -n 4 -f node\_config\_file model.exe\_file  
i tried:  
cpp\_options = {  
‘STAN\_MPI’: True,  
‘CXX’: ‘mpicxx’,  
‘TBB\_CXX\_TYPE’: ‘gcc’,  
‘mpiexec’: ‘-n 4 -f node\_config\_file’  
}  
model = cmdstanpy.CmdStanModel(stan\_file=stan\_file\_path, cpp\_options=cpp\_options, force\_compile=True)  
fit = model.sample(data=stan\_data)  
and it doesn’t execute the MPI as expected.
2. If not possible, then will 1.2.4 resolve the cmdstanpy.from\_csv() being brittle? The subprocess.run() command of the executable (see above post) runs the MPI as expected, but the outputted ‘output.csv’ is not readable by cmdstanpy.from\_csv().

yes, non-MPI sampling works fine.

---

<div class="post-metadata">

**Author:** ![WardBrian](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wardbrian/32/12078_2.png) [@WardBrian](https://discourse.mc-stan.org/u/WardBrian)\
**Post date:** [November 5, 2024, 3:01pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/14 "2024-11-05T15:01:55Z")

</div>

> [@e32432423](#):
>
> what is the syntax such that I can run a multi-node mpi command such as this in cmdstanpy, like cmdstanr allows?

Cmdstanpy does not have this feature built in in the way that it appears cmdstanr does

> [@e32432423](#):
>
> If not possible, then will 1.2.4 resolve the cmdstanpy.from\_csv() being brittle?

The primary reason for the 1.2.4 release was to fix issues with reading the CSV files from recent cmdstans

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 5, 2024, 3:03pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/15 "2024-11-05T15:03:50Z")

</div>

1. May I ask why?
2. I am hoping 1.2.4 has from\_csv() fixed- trying now,

---

<div class="post-metadata">

**Author:** ![WardBrian](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wardbrian/32/12078_2.png) [@WardBrian](https://discourse.mc-stan.org/u/WardBrian)\
**Post date:** [November 5, 2024, 3:16pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/16 "2024-11-05T15:16:57Z")

</div>

> [@e32432423](#):
>
> May I ask why?

I don’t think there is any reason other than nobody wrote the code. sample\_mpi being added to cmdstanr predates my involvement with the project

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 5, 2024, 8:42pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/17 "2024-11-05T20:42:16Z")

</div>

cmdstanpy 1.2.4 seemed to read in the csv file using cmdstanpy.from\_csv() method. this is excellent, and considered closed at this time.

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 12, 2024, 5:36am UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/19 "2024-11-12T05:36:12Z")

</div>

here’s a long shot question.

the above has discussed running the .exe file using mpi command, parallelizing for num\_chains=1, via n\_shard and map\_rect. This helps with the gradient evaluation time.

1. is there a way to parallelize to decrease iteration completion time? Does an iteration include many serial gradient evaluations?
2. is there also a way to parallelize across num\_chains, where num\_chains\>1? Currently it runs serially, chain1 iterations…then chain2 iterations… I would imagine that is a parallelizable opportunity…

---

<div class="post-metadata">

**Author:** ![WardBrian](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wardbrian/32/12078_2.png) [@WardBrian](https://discourse.mc-stan.org/u/WardBrian)\
**Post date:** [November 12, 2024, 2:26pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/20 "2024-11-12T14:26:18Z")

</div>

> [@e32432423](#):
>
> 1. is there a way to parallelize to decrease iteration completion time? Does an iteration include many serial gradient evaluations?

If you’re already using map\_rect, there are not really any additional opportunities for parallelism within one iteration. There are some (potentially wasteful) options discussed [here](https://github.com/stan-dev/stan/pull/3103), but not implemented

> [@e32432423](#):
>
> is there also a way to parallelize across num\_chains, where num\_chains\>1? Currently it runs serially, chain1 iterations…then chain2 iterations… I would imagine that is a parallelizable opportunity…

Yes, definitely. [Parallelization](https://mc-stan.org/docs/cmdstan-guide/parallelization.html#multi-threading-with-tbb). You can also do what cmdstanpy and friends _used_ to do and wrap your subprocess call in something from python’s `multiprocessing` module

---

<div class="post-metadata">

**Author:** ![e32432423](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@e32432423](https://discourse.mc-stan.org/u/e32432423)\
**Post date:** [November 14, 2024, 4:27pm UTC](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115/21 "2024-11-14T16:27:55Z")

</div>

is it true that the pymc devs found a way to parallelize via cpu/gpu? If so, how can Stan do same?

> **[MCMC for big datasets: faster sampling with JAX and the GPU - PyMC Labs](https://www.pymc-labs.com/blog-posts/pymc-stan-benchmark/)**
>
> Scaling PyMC using JAX to sample on the GPU

it seems like the sampling can be parallelized?..

[Next page](https://discourse.mc-stan.org/t/cmdstanpy-mpi-speedup/37115.md?page=2)