Using nested parallelization with Rstan?

Interfaces RStan
1

Hello everyone,

I am trying to run a simulation-based calibration and my goal is then to loop in parallel over many simulated data sets and fit the same model.

I can easily achieve this if I fit sample only from one chain ore if they are sampled sequentially with the same core. For example:

fits <- mclapply(1:N_data_sets, 
function(x)
  sampling(object = stan_model,
  data = data_set[[x]], 
  cores = 1, 
  chains = 1), 
  mc.cores = 4)

However, I would like to have a sort of nested parallelization. For example, if I have 4 cores, I want two groups of 2 cores, where in each group I sample two chains in parallel for 2 different data sets. Something like this:

fits <- mclapply(1:N_data_sets, 
function(x)
  sampling(object = stan_model,
  data = data_set[[x]], 
  cores = 2, 
  chains = 2), 
  mc.cores = 4)

The problem is that, with this type of code, I can only sample each data set sequentially, with two chains in parallel, instead of the 2 data sets with 2 parallel chains. I also get this warning:

5: In socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :
  localhost:11449 cannot be opened

My question is if this is possible at all with Rstan, or only with cmdStan?

Btw, I am using Ubuntu 18.04.

Many thanks for your help.

2

Could you try setting mc.cores to 2 instead of 4?

Another possibility is to run two one-chain runs for each dataset, then combine them with sflist2stanfit.

1 Like
3

Your computer is not letting you use sockets for some unknown reason. It may work if you run it from a shell rather than RStudio.

4

Thank you for your response.

If I do this, I get the same error, and chains will sampled sequentially.

Yes, maybe this is easiest way, to parallelize everything and then combine the chains using sflist2stanfit.

5

Thank you for your response.

Running the script from the shell seemed to work without any errors.

Do you have any idea why it does not work on RStudio?

6

Something is preventing you from running a R process in the background. Permissions maybe? Or maybe your computer is just locked down for a corporate environment.