I can't calling some of my cpu

Interfaces RStan
1

I’m lucky to use 80 cpu, so I’m trying to parallelize about 28 at once.
but, when I tried this, I got error message below.

error occurred during calling the sampler; sampling not done.

I configured that 20 cpu are working, and I got 8 error messages above.

below is my code.

rstan_options(auto_write = T)
options(mc.cores = 40)
cl <- makeCluster(40)
clusterExport(cl,c("IvT_model", "stan_data1", "stan_data2", "stan_data3", "stan_data4", "stan_data5", "stan_data6", "stan_data7"))
result <- parLapply(cl, c(1:7), first_node)

Also, when I tried to parallelize 16 sampling with different data, there comes different error below.

Error in unserialize(node$con) : error reading from connection Calls

As I searched the detail, this problem is come from memory issue. Is it right??
And is there any ways to solve the cpu calling problem??

2

Sorry that nobody responded initially. The thing to do is get one core working then try to scale up to more cores. You can run into memory issues on a single big server if you run too many parallel cores using multi-process, because each needs the memory to store all the data and all the output.

Using something like cmdstanr or just CmdStan straight up can stream analyses to file. I think this may also be possible with RStan, but I’m not entirely sure if you can turn off writing to memory at the same time.

if you want to store a large number of

3

Thanks Bob!
I’ve moved to cmdstanr couple days ago, and this problem was solved. But still not know what is the extact problem.