Parallel chains with rstan fails in Rstudio

Interfaces RStan
1

Here’s the problem. In Rstudio, Stan runs one chain at a time but not in parallel. Below is an example. This example uses rstanarm but I had the same problem with regular rstan.

I’ve also tried it in the regular R console, and it works fine. The problem is only in Rstudio, and only with parallel chains.

Example below:

> y <- rnorm(50)
> group <- rep(1:10,5)
> library(rstanarm)
Loading required package: Rcpp
rstanarm (Version 2.19.3, packaged: 2020-05-03 05:05:42 UTC)
- Do not expect the default priors to remain the same in future rstanarm versions.
Thus, R scripts should specify priors explicitly, even if they are just the defaults.
- For execution on a local, multicore CPU with excess RAM we recommend calling
options(mc.cores = parallel::detectCores())
- bayesplot theme set to bayesplot::theme_default()
   * Does _not_ affect other ggplot2 plots
   * See ?bayesplot_theme_set for details on theme setting
> data <- data.frame(y,group)
> stan_glmer(y ~ (1 | group), data=data)

SAMPLING FOR MODEL 'continuous' NOW (CHAIN 1).
Chain 1: 
Chain 1: Gradient evaluation took 7.8e-05 seconds
Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 0.78 seconds.
Chain 1: Adjust your expectations accordingly!
Chain 1: 
Chain 1: 
Chain 1: Iteration:    1 / 2000 [  0%]  (Warmup)
Chain 1: Iteration:  200 / 2000 [ 10%]  (Warmup)
Chain 1: Iteration:  400 / 2000 [ 20%]  (Warmup)
Chain 1: Iteration:  600 / 2000 [ 30%]  (Warmup)
Chain 1: Iteration:  800 / 2000 [ 40%]  (Warmup)
Chain 1: Iteration: 1000 / 2000 [ 50%]  (Warmup)
Chain 1: Iteration: 1001 / 2000 [ 50%]  (Sampling)
Chain 1: Iteration: 1200 / 2000 [ 60%]  (Sampling)
Chain 1: Iteration: 1400 / 2000 [ 70%]  (Sampling)
Chain 1: Iteration: 1600 / 2000 [ 80%]  (Sampling)
Chain 1: Iteration: 1800 / 2000 [ 90%]  (Sampling)
Chain 1: Iteration: 2000 / 2000 [100%]  (Sampling)
Chain 1: 
Chain 1:  Elapsed Time: 0.204752 seconds (Warm-up)
Chain 1:                0.126084 seconds (Sampling)
Chain 1:                0.330836 seconds (Total)
Chain 1: 

SAMPLING FOR MODEL 'continuous' NOW (CHAIN 2).
Chain 2: 
Chain 2: Gradient evaluation took 1.4e-05 seconds
Chain 2: 1000 transitions using 10 leapfrog steps per transition would take 0.14 seconds.
Chain 2: Adjust your expectations accordingly!
Chain 2: 
Chain 2: 
Chain 2: Iteration:    1 / 2000 [  0%]  (Warmup)
Chain 2: Iteration:  200 / 2000 [ 10%]  (Warmup)
Chain 2: Iteration:  400 / 2000 [ 20%]  (Warmup)
Chain 2: Iteration:  600 / 2000 [ 30%]  (Warmup)
Chain 2: Iteration:  800 / 2000 [ 40%]  (Warmup)
Chain 2: Iteration: 1000 / 2000 [ 50%]  (Warmup)
Chain 2: Iteration: 1001 / 2000 [ 50%]  (Sampling)
Chain 2: Iteration: 1200 / 2000 [ 60%]  (Sampling)
Chain 2: Iteration: 1400 / 2000 [ 70%]  (Sampling)
Chain 2: Iteration: 1600 / 2000 [ 80%]  (Sampling)
Chain 2: Iteration: 1800 / 2000 [ 90%]  (Sampling)
Chain 2: Iteration: 2000 / 2000 [100%]  (Sampling)
Chain 2: 
Chain 2:  Elapsed Time: 0.157635 seconds (Warm-up)
Chain 2:                0.128213 seconds (Sampling)
Chain 2:                0.285848 seconds (Total)
Chain 2: 

SAMPLING FOR MODEL 'continuous' NOW (CHAIN 3).
Chain 3: 
Chain 3: Gradient evaluation took 1.4e-05 seconds
Chain 3: 1000 transitions using 10 leapfrog steps per transition would take 0.14 seconds.
Chain 3: Adjust your expectations accordingly!
Chain 3: 
Chain 3: 
Chain 3: Iteration:    1 / 2000 [  0%]  (Warmup)
Chain 3: Iteration:  200 / 2000 [ 10%]  (Warmup)
Chain 3: Iteration:  400 / 2000 [ 20%]  (Warmup)
Chain 3: Iteration:  600 / 2000 [ 30%]  (Warmup)
Chain 3: Iteration:  800 / 2000 [ 40%]  (Warmup)
Chain 3: Iteration: 1000 / 2000 [ 50%]  (Warmup)
Chain 3: Iteration: 1001 / 2000 [ 50%]  (Sampling)
Chain 3: Iteration: 1200 / 2000 [ 60%]  (Sampling)
Chain 3: Iteration: 1400 / 2000 [ 70%]  (Sampling)
Chain 3: Iteration: 1600 / 2000 [ 80%]  (Sampling)
Chain 3: Iteration: 1800 / 2000 [ 90%]  (Sampling)
Chain 3: Iteration: 2000 / 2000 [100%]  (Sampling)
Chain 3: 
Chain 3:  Elapsed Time: 0.169438 seconds (Warm-up)
Chain 3:                0.193983 seconds (Sampling)
Chain 3:                0.363421 seconds (Total)
Chain 3: 

SAMPLING FOR MODEL 'continuous' NOW (CHAIN 4).
Chain 4: 
Chain 4: Gradient evaluation took 1.9e-05 seconds
Chain 4: 1000 transitions using 10 leapfrog steps per transition would take 0.19 seconds.
Chain 4: Adjust your expectations accordingly!
Chain 4: 
Chain 4: 
Chain 4: Iteration:    1 / 2000 [  0%]  (Warmup)
Chain 4: Iteration:  200 / 2000 [ 10%]  (Warmup)
Chain 4: Iteration:  400 / 2000 [ 20%]  (Warmup)
Chain 4: Iteration:  600 / 2000 [ 30%]  (Warmup)
Chain 4: Iteration:  800 / 2000 [ 40%]  (Warmup)
Chain 4: Iteration: 1000 / 2000 [ 50%]  (Warmup)
Chain 4: Iteration: 1001 / 2000 [ 50%]  (Sampling)
Chain 4: Iteration: 1200 / 2000 [ 60%]  (Sampling)
Chain 4: Iteration: 1400 / 2000 [ 70%]  (Sampling)
Chain 4: Iteration: 1600 / 2000 [ 80%]  (Sampling)
Chain 4: Iteration: 1800 / 2000 [ 90%]  (Sampling)
Chain 4: Iteration: 2000 / 2000 [100%]  (Sampling)
Chain 4: 
Chain 4:  Elapsed Time: 0.216624 seconds (Warm-up)
Chain 4:                0.149475 seconds (Sampling)
Chain 4:                0.366099 seconds (Total)
Chain 4: 
stan_glmer
 family:       gaussian [identity]
 formula:      y ~ (1 | group)
 observations: 50
------
            Median MAD_SD
(Intercept) 0.0    0.1   

Auxiliary parameter(s):
      Median MAD_SD
sigma 1.0    0.1   

Error terms:
 Groups   Name        Std.Dev.
 group    (Intercept) 0.25    
 Residual             0.97    
Num. levels: group 10 

------
* For help interpreting the printed output see ?print.stanreg
* For info on the priors used see ?prior_summary.stanreg
> options(mc.cores = parallel::detectCores())
> stan_glmer(y ~ (1 | group), data=data)
starting worker pid=60598 on localhost:11497 at 15:18:21.818
starting worker pid=60599 on localhost:11497 at 15:18:21.818
starting worker pid=60597 on localhost:11497 at 15:18:21.819
starting worker pid=60600 on localhost:11497 at 15:18:21.819
Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> -> slaveLoop -> makeSOCKmaster
Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> -> slaveLoop -> makeSOCKmaster
Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> -> slaveLoop -> makeSOCKmaster
Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> -> slaveLoop -> makeSOCKmaster
Execution halted
Execution halted
Execution halted
Execution halted
Error in makePSOCKcluster(names = spec, ...) : 
  Cluster setup failed. 4 of 4 workers failed to connect.
>

Edited by @Max_Mantei: Added code tags

2

It was an RStudio bug: RStudio Viewer with rstan running in parrallel with RStudio 1.3

3 Likes
3

Thanks.

4

sure, thanks.

5

I will mark your reply as solution.

Edit: Unmarked, since the bug is apparently not fixed.

1 Like
6

The bug still exists. I’m in R 4.0.2 and Rstudio 1.2.5033 and it still is failing to run parallel chains in rstan. Again, it runs fine in the regular R console.

7

What OS?

9

Mac Catalina 10.15.5
R 4.0.2
RStudio 1.3.959

According to Jonah, it won’t work until a future version of Rstudio.

10

It should work with the latest “preview” version of RStudio but the bug was just fixed recently so it’s not in an official RStudio release yet.

1 Like
11

Ok. If using the preview version of RStudio doesn’t work and you’re desperate, you could try ezStan, which forks a new R process per chain so should be more robust (also has a nicer progress indicator IMO 😋).

12

Hi–I’m not desperate, as I can just use the non-Rstudio R GUI. I just wanted this to be cleared up before Regression and Other Stories comes out.
Andrew

1 Like