I am trying to compute multiple models in parallel on my computer with 16 logical cores. To do so, I use brms::brm()
inside furrr::future_map()
, and using four workers to run furrr::future_map()
by setting plan(tweak(multisession, workers = 4))
. In addition, I use brms::brm()
with its backend set to cmdstanr
(i.e. brms::brm(..., backend = "cmdstanr")
), since cmdstanr
’s computation is fast. Moreover, I set the number of cores to use when executing the chains in parallel as four (i.e. brms::brm(..., cores = 4)
), as I have four chains per model.
However, when these settings are implemented, as shown below, and multiple models are computed in parallel using furrr::future_map()
, the error collect2.exe: error: ld returned 1 exit status
occurs and the execution halts. The problem is resolved when I use purrr::map()
instead of furrr::future_map()
but purrr::map()
does not parallelise the computation of multiple models.
Does the problem happen because I am trying to do nested parallelism where four workers compute models (by setting plan(tweak(multisession, workers = 4))
) and each worker deals with additional four cores set by brms::brm(..., cores = 4)
? Then, is it impossible to compute multiple models in parallel using brms as shown in here or @matti 's post on RPubs when backend = "cmdstanr"
is set?
bayes_samples_Nsim < samples_Nsim >
group_by(sample_id) >
nest(samples = samples) >
mutate(
Bayesian_model = future_map(
.options = furrr_options(
scheduling = 0,
seed = 7,
prefix = "prefix"
),
samples,
~ brm(
samples ~ 1,
data = .,
family = gaussian(),
# The prior can be more constrained
# then the following weak prior:
# prior = prior(normal(10, 5), class = Intercept),
prior = prior(normal(10, 1), class = Intercept),
chains = 4,
cores = 4,
control = list(adapt_delta = 0.95),
backend = "cmdstanr",
iter = 2000,
warmup = 1000
)
),
estimates =
map_dfr(
.x = Bayesian_model,
~ fixef(.x) >
as_tibble()
),
Bayesian_estimate = estimates$Estimate,
Bayesian_std_error = estimates$`Est.Error`,
Bayesian_lower = estimates$`Q2.5`,
Bayesian_upper = estimates$`Q97.5`
)
Error message
C
:\rtools42/x86_64w64mingw32.static.posix/bin/ld.exe:
src/cmdstan/main.o:main.cpp:(.text$_ZN4stan2io15stan_csv_reader13read_metadataERSiRNS0_17stan_csv_metadataEPSo[_ZN4stan2io15stan_csv_reader13read_metadataERSiRNS0_17stan_csv_metadataEPSo]+0xe3): undefined reference to `std::istream::seekg(std::fpos<int>)'
C:\rtools42/x86_64w64mingw32.static.posix/bin/ld.exe:
s
rc/cmdstan/main.o:main.cpp:(.text$_ZN4stan2io15stan_csv_reader15read_adaptationERSiRNS0_19stan_csv_adaptationEPSo[_ZN4stan2io15stan_csv_reader15read_adaptationERSiRNS0_19stan_csv_adaptationEPSo]+0x164): undefined reference to `std::istream::seekg(std::fpos<int>)'
C:\rtools42/x86_64w64mingw32.static.posix/bin/ld.exe:
s
rc/cmdstan/main.o:main.cpp:(.text$_ZN4stan2io15stan_csv_reader12read_samplesERSiRN5Eigen6MatrixIdLin1ELin1ELi0ELin1ELin1EEERNS0_15stan_csv_timingEPSo[_ZN4stan2io15stan_csv_reader12read_samplesERSiRN5Eigen6MatrixIdLin1ELin1ELi0ELin1ELin1EEERNS0_15stan_csv_timingEPSo]+0x74d): undefined reference to `std::istream::seekg(std::fpos<int>)'
collect2.exe: error: ld returned 1 exit status
mingw32make.exe: *** [make/program:59: C:\Users\my_username\AppData\Local\Temp\RtmpyKRff8\model2d24605a3f2e.exe] Error 1
Error in `mutate()`:
! Problem while computing `Bayesian_model = future_map(...)`.
ℹ The error occurred in group 1: sample_id = "1".
Caused by error:
! An error occured during compilation! See the message above for more information.
Run `rlang::last_error()` to see where the error occurred.
MWE
library("tidyverse")
library("brms")
library("cmdstanr")
library("future")
library("furrr")
plan(tweak(multisession, workers = 4))
alpha < 0.05
n < 100
Nsim < 10
mu < 10
sigma < 5
set.seed(20220718)
samples_Nsim < 1:Nsim >
map(
~ rnorm(
n, mu, sigma
)
) >
map_dfr(
.id = "sample_id",
~ {
sample_mean < mean(x = .x)
sample_sd < sd(x = .x)
half_interval < sample_sd / sqrt(n) * qt(1  alpha * 0.5, df = n  1)
upper < sample_mean + half_interval
lower < sample_mean  half_interval
tibble(
samples = .x,
sample_mean = sample_mean,
sample_sd = sample_sd,
upper = upper,
lower = lower,
mu_out_of_CI = upper < mu  lower > mu
)
}
)
tictoc::tic()
# 245.58 sec = 4 min 5 sec elapsed to calculate 10 model with iter = 2000 using purrr:map
# 2309.72 sec = 38 min 29 sec elapsed to calculate 100 model with iter = 2000 using purrr:map
bayes_samples_Nsim < samples_Nsim >
group_by(sample_id) >
nest(samples = samples) >
mutate(
Bayesian_model = future_map(
.options = furrr_options(
scheduling = 0,
seed = 7,
prefix = "prefix"
),
samples,
~ brm(
samples ~ 1,
data = .,
family = gaussian(),
# The prior can be more constrained
# then the following weak prior:
# prior = prior(normal(10, 5), class = Intercept),
prior = prior(normal(10, 1), class = Intercept),
chains = 4,
cores = 4,
control = list(adapt_delta = 0.95),
backend = "cmdstanr",
iter = 2000,
warmup = 1000
)
),
estimates =
map_dfr(
.x = Bayesian_model,
~ fixef(.x) >
as_tibble()
),
Bayesian_estimate = estimates$Estimate,
Bayesian_std_error = estimates$`Est.Error`,
Bayesian_lower = estimates$`Q2.5`,
Bayesian_upper = estimates$`Q97.5`
)
tictoc::toc()
Session Info
Operating System
 Windows 10 Pro (
x86_64w64mingw32
)  R version 4.2.0 (20220422 ucrt)
package version

brms
2.17.0 
cmdstanr
0.5.2 
future
1.25.0 
furrr
0.3.0 
tidyverse
1.3.1