# Parallel the same model fitting for differen data
**URL:** https://discourse.mc-stan.org/t/parallel-the-same-model-fitting-for-differen-data/26135
**Category:** CmdStan
**Tags:** techniques
**Created:** [January 28, 2022, 2:17am UTC](https://discourse.mc-stan.org/t/parallel-the-same-model-fitting-for-differen-data/26135 "2022-01-28T02:17:03Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Michelle](https://discourse.mc-stan.org/u/Michelle)
#### Post date: [January 28, 2022, 2:17am UTC](https://discourse.mc-stan.org/t/parallel-the-same-model-fitting-for-differen-data/26135/1 "2022-01-28T02:17:03Z")
Hi Stan community,
I am trying to fit the same model (using cmdstanr) for different data inputs `data_list_all[[idx]]`. In this case, I thought I could use parallel for loop to save some time. In the following code, I tried `foreach` and `%dopar%` from `library(parallel) library(foreach) library(doParallel)`. It works when I decrease the data length of each `data_list_all[[idx]]` in each loop (for testing), but when using full length data, the fitting results of a few `data_list_all[[idx]]` were not properly saved, because cmdstan output files were not found in the temp folder. If I just run sequentially with full data size, everyone fitted very well.
Do you know what’s going on here? Maybe we have another better choice? My gut feeling is that parallel chains in Stan may not fully completable with `foreach` loop such that one finished chain for `data_list_all[[2]]` was overwritten by a chain for `data_list_all[[6]]`, before other chains complete for `data_list_all[[2]]`.
Another way I am thinking is to index parameters in the model and feed all data with the same indexing, as long as I don’t pool parameters over the data set, it should be identical to the for loop solution. So you think in this case I can tell Stan to use 4 cores per `data_list_all[[idx]]`?
Thank you very much :)
```
fit_list_all <- foreach(
idx = fit_idx
) %dopar% {
mod$sample(
data = data_list_all[[idx]], iter_warmup = 1000, iter_sampling = 1000,
chains = 4, parallel_chains = 4, show_messages = F
)
}
```
---
### Author:  [@Bob\_Carpenter](https://discourse.mc-stan.org/u/Bob_Carpenter)
#### Post date: [February 1, 2022, 10:14pm UTC](https://discourse.mc-stan.org/t/parallel-the-same-model-fitting-for-differen-data/26135/2 "2022-02-01T22:14:12Z")
> [@Michelle](#):
>
> My gut feeling is that parallel chains in Stan may not fully completable with `foreach` loop such that one finished chain for `data_list_all[[2]]` was overwritten by a chain for `data_list_all[[6]]` , before other chains complete for `data_list_all[[2]]` .
You may be right, despite R randomly generating file names. You could try using the `output_dir` arguments or `output_basename` functions in the method `sample()` on a cmdstan\_model in cmdstanr.
I’m pinging @jgabry, who should know the answer here.
Also, you want to be careful to not spawn more jobs than you have cores. Even then, I find on my rather beefy Xeon-based iMac Pro that it can’t run 8 chains in parallel nearly as fast as 1 sequentially. So you might not be getting a lot of gains from parallelization if you’re getting close to or exceeding the number of cores you have.
---
### Author:  [@Michelle](https://discourse.mc-stan.org/u/Michelle)
#### Post date: [February 2, 2022, 2:08pm UTC](https://discourse.mc-stan.org/t/parallel-the-same-model-fitting-for-differen-data/26135/3 "2022-02-02T14:08:22Z")
> [@Bob\_Carpenter](#):
>
> You may be right, despite R randomly generating file names. You could try using the `output_dir` arguments or `output_basename` functions in the method `sample()` on a cmdstan\_model in cmdstanr.
>
> I’m pinging @jgabry, who should know the answer here.
I will try to specify cmdstan output folder. Hopefully @jonah has better solutions.
> [@Bob\_Carpenter](#):
>
> Also, you want to be careful to not spawn more jobs than you have cores. Even then, I find on my rather beefy Xeon-based iMac Pro that it can’t run 8 chains in parallel nearly as fast as 1 sequentially. So you might not be getting a lot of gains from parallelization if you’re getting close to or exceeding the number of cores you have.
You are absolutely right, this already happened to me. I do the following to avoid this issue.
```no-highlight
parallel::detectCores()
n.cores <- parallel::detectCores() - 2
```
---
### Author:  [@karimn](https://discourse.mc-stan.org/u/karimn)
#### Post date: [November 8, 2023, 7:38pm UTC](https://discourse.mc-stan.org/t/parallel-the-same-model-fitting-for-differen-data/26135/4 "2023-11-08T19:38:05Z")
@jonah Do you know why this might be happening? I’m seeing the same thing use `furrr::future_map`. I fit fine and I’m able to extract posterior samples fine _but only if I don’t do that in parallel as well_. So if I have a list of fit objects and run something like `furrr::future_map(list_of_fits, \(f) spread_rvars(f, x[i]))` I get errors that the csv files cannot be found.
---
### Author:  [@jsocolar](https://discourse.mc-stan.org/u/jsocolar)
#### Post date: [November 9, 2023, 12:09am UTC](https://discourse.mc-stan.org/t/parallel-the-same-model-fitting-for-differen-data/26135/5 "2023-11-09T00:09:58Z")
What happens if you pass the `output_basename` parameter to `cmstanr::sample` such that each iteration of the parallel map or loop writes to a deterministically unique csv filename?
---
### Author:  [@jonah](https://discourse.mc-stan.org/u/jonah)
#### Post date: [November 9, 2023, 1:10am UTC](https://discourse.mc-stan.org/t/parallel-the-same-model-fitting-for-differen-data/26135/6 "2023-11-09T01:10:01Z")
I like @jsocolar’s suggestion. Curious if that resolves the issue.
---
### Author:  [@karimn](https://discourse.mc-stan.org/u/karimn)
#### Post date: [November 17, 2023, 5:12pm UTC](https://discourse.mc-stan.org/t/parallel-the-same-model-fitting-for-differen-data/26135/7 "2023-11-17T17:12:06Z")
Trying it now. Will let you know if I hit any problems.
---
### Author:  [@karimn](https://discourse.mc-stan.org/u/karimn)
#### Post date: [November 22, 2023, 3:55pm UTC](https://discourse.mc-stan.org/t/parallel-the-same-model-fitting-for-differen-data/26135/8 "2023-11-22T15:55:17Z")
It works. Thanks, all!
---
### Author:  [@jonah](https://discourse.mc-stan.org/u/jonah)
#### Post date: [November 22, 2023, 4:54pm UTC](https://discourse.mc-stan.org/t/parallel-the-same-model-fitting-for-differen-data/26135/9 "2023-11-22T16:54:26Z")
Great, thanks for following up and letting us know!