Rstan fails in multiple Rserve sessions

jzh · July 7, 2017, 9:59am

It looks like rstan does not support running in multiple Rserve sessions simultaneously. From the Rserve client I created 2 threads each working on a specific section of data over a specific Rserve session to the same Ubuntu (16.04.1LTS). As you can see below one of them failed with error message “Error in readRDS(file) : error reading from connection” right after the c++ code was compiled. It seems that the two session is trying to compete with the same file in the file system, so only one of them can get it locking everyone else out. The R version is 3.4.1 and rstan is 2.16.2.
Any suggestions?

Operating System: Ubuntu
Interface Version: 2.16.2
Output of writeLines(readLines(file.path(Sys.getenv(“HOME”), “.R/Makevars”))):
Output of devtools::session_info("rstan”):

///////////////////////////////////////////////////////////////////////////
Loading required package: ggplot2
Loading required package: StanHeaders
Loading required package: StanHeaders
rstan (Version 2.16.2, packaged: 2017-07-03 09:24:58 UTC, GitRev: 2e1f913d3ca3)
For execution on a local, multicore CPU with excess RAM we recommend calling
rstan_options(auto_write = TRUE)
options(mc.cores = parallel::detectCores())
rstan (Version 2.16.2, packaged: 2017-07-03 09:24:58 UTC, GitRev: 2e1f913d3ca3)
For execution on a local, multicore CPU with excess RAM we recommend calling
rstan_options(auto_write = TRUE)
options(mc.cores = parallel::detectCores())
DIAGNOSTIC(S) FROM PARSER:DIAGNOSTIC(S) FROM PARSER:

Warning (non-fatal):
Left-hand side of sampling statement (~) may contain a non-linear transform of a parameter or local variable.
If it does, you need to include a target += statement with the log absolute determinant of the Jacobian of the transform.
Left-hand-side of sampling statement:
e ~ student_t(…)
Warning (non-fatal):
Left-hand side of sampling statement (~) may contain a non-linear transform of a parameter or local variable.
If it does, you need to include a target += statement with the log absolute determinant of the Jacobian of the transform.
Left-hand-side of sampling statement:
e ~ student_t(…)

In file included from /usr/local/lib/R/site-library/BH/include/boost/config.hpp:39:0,
from /usr/local/lib/R/site-library/BH/include/boost/math/tools/config.hpp:13,
from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/core/var.hpp:7,
from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/core/gevv_vvv_vari.hpp:5,
from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/core.hpp:12,
from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/mat.hpp:4,
from /usr/local/lib/R/site-library/StanHeaders/include/stan/math.hpp:4,
from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/model/model_header.hpp:4,
from file35c7ed5684b.cpp:8:
/usr/local/lib/R/site-library/BH/include/boost/config/compiler/gcc.hpp:186:0: warning: “BOOST_NO_CXX11_RVALUE_REFERENCES” redefined

define BOOST_NO_CXX11_RVALUE_REFERENCES

^
:0:0: note: this is the location of the previous definition
In file included from /usr/local/lib/R/site-library/BH/include/boost/config.hpp:39:0,
from /usr/local/lib/R/site-library/BH/include/boost/math/tools/config.hpp:13,
from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/core/var.hpp:7,
from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/core/gevv_vvv_vari.hpp:5,
from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/core.hpp:12,
from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/mat.hpp:4,
from /usr/local/lib/R/site-library/StanHeaders/include/stan/math.hpp:4,
from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/model/model_header.hpp:4,
from file35d7bf9aac4.cpp:8:
/usr/local/lib/R/site-library/BH/include/boost/config/compiler/gcc.hpp:186:0: warning: “BOOST_NO_CXX11_RVALUE_REFERENCES” redefined

define BOOST_NO_CXX11_RVALUE_REFERENCES

^
:0:0: note: this is the location of the previous definition
Error in readRDS(file) : error reading from connection

SAMPLING FOR MODEL ‘634840b083abed8af10db9ed0d88ab99’ NOW (CHAIN 1).

Gradient evaluation took 0.000329 seconds
1000 transitions using 10 leapfrog steps per transition would take 3.29 seconds.
Adjust your expectations accordingly!

Iteration: 1 / 3000 [ 0%] (Warmup)

SAMPLING FOR MODEL ‘634840b083abed8af10db9ed0d88ab99’ NOW (CHAIN 2).

Gradient evaluation took 0.000307 seconds
1000 transitions using 10 leapfrog steps per transition would take 3.07 seconds.
Adjust your expectations accordingly!

Iteration: 1 / 3000 [ 0%] (Warmup)

SAMPLING FOR MODEL ‘634840b083abed8af10db9ed0d88ab99’ NOW (CHAIN 3).

Gradient evaluation took 0.000322 seconds
1000 transitions using 10 leapfrog steps per transition would take 3.22 seconds.
Adjust your expectations accordingly!

Iteration: 1 / 3000 [ 0%] (Warmup)

SAMPLING FOR MODEL ‘634840b083abed8af10db9ed0d88ab99’ NOW (CHAIN 4).

Gradient evaluation took 0.000307 seconds
1000 transitions using 10 leapfrog steps per transition would take 3.07 seconds.
Adjust your expectations accordingly!

Iteration: 1 / 3000 [ 0%] (Warmup)
Iteration: 300 / 3000 [ 10%] (Warmup)
Iteration: 300 / 3000 [ 10%] (Warmup)
Iteration: 300 / 3000 [ 10%] (Warmup)
Iteration: 300 / 3000 [ 10%] (Warmup)
Iteration: 600 / 3000 [ 20%] (Warmup)
Iteration: 600 / 3000 [ 20%] (Warmup)
Iteration: 600 / 3000 [ 20%] (Warmup)
Iteration: 600 / 3000 [ 20%] (Warmup)
Iteration: 900 / 3000 [ 30%] (Warmup)
Iteration: 900 / 3000 [ 30%] (Warmup)
Iteration: 900 / 3000 [ 30%] (Warmup)
Iteration: 900 / 3000 [ 30%] (Warmup)
Iteration: 1200 / 3000 [ 40%] (Warmup)
Iteration: 1200 / 3000 [ 40%] (Warmup)
Iteration: 1200 / 3000 [ 40%] (Warmup)
Iteration: 1200 / 3000 [ 40%] (Warmup)
Iteration: 1500 / 3000 [ 50%] (Warmup)
Iteration: 1501 / 3000 [ 50%] (Sampling)
Iteration: 1500 / 3000 [ 50%] (Warmup)
Iteration: 1501 / 3000 [ 50%] (Sampling)
Iteration: 1500 / 3000 [ 50%] (Warmup)
Iteration: 1501 / 3000 [ 50%] (Sampling)
Iteration: 1500 / 3000 [ 50%] (Warmup)
Iteration: 1501 / 3000 [ 50%] (Sampling)
Iteration: 1800 / 3000 [ 60%] (Sampling)
Iteration: 1800 / 3000 [ 60%] (Sampling)
Iteration: 1800 / 3000 [ 60%] (Sampling)
Iteration: 1800 / 3000 [ 60%] (Sampling)
Iteration: 2100 / 3000 [ 70%] (Sampling)
Iteration: 2100 / 3000 [ 70%] (Sampling)
Iteration: 2100 / 3000 [ 70%] (Sampling)
Iteration: 2100 / 3000 [ 70%] (Sampling)
Iteration: 2400 / 3000 [ 80%] (Sampling)
Iteration: 2400 / 3000 [ 80%] (Sampling)
Iteration: 2400 / 3000 [ 80%] (Sampling)
Iteration: 2400 / 3000 [ 80%] (Sampling)
Iteration: 2700 / 3000 [ 90%] (Sampling)
Iteration: 2700 / 3000 [ 90%] (Sampling)
Iteration: 2700 / 3000 [ 90%] (Sampling)
Error in extract(fit, “omega”) : object ‘fit’ not found
Iteration: 2700 / 3000 [ 90%] (Sampling)
Iteration: 3000 / 3000 [100%] (Sampling)

Elapsed Time: 5.97196 seconds (Warm-up)
6.57777 seconds (Sampling)
12.5497 seconds (Total)

Iteration: 3000 / 3000 [100%] (Sampling)

Elapsed Time: 6.91391 seconds (Warm-up)
6.22381 seconds (Sampling)
13.1377 seconds (Total)

Iteration: 3000 / 3000 [100%] (Sampling)

Elapsed Time: 7.14996 seconds (Warm-up)
6.55966 seconds (Sampling)
13.7096 seconds (Total)

Iteration: 3000 / 3000 [100%] (Sampling)

Elapsed Time: 6.35933 seconds (Warm-up)
6.61689 seconds (Sampling)
12.9762 seconds (Total)

bgoodri · July 8, 2017, 11:50am

How are you calling them?

jzh · July 11, 2017, 11:16am

Create several Rserver sessions, each session having the same stan model code.

jicun

bgoodri · July 11, 2017, 2:49pm

I am not too familiar with Rserve. If I start two tabs in my Konsole shell and from the first tab, do Edit -> Copy Input To the second tab. And then in the first tab, I execute

R CMD Rserve --RS-source test.R

it runs Stan in both but has an error when it exits

...
 Elapsed Time: 0.01088 seconds (Warm-up)
               0.010922 seconds (Sampling)
               0.021802 seconds (Total)

Rserv started in daemon mode.
##> SOCK_ERROR: bind error #98(address already in use)

jzh · July 12, 2017, 8:16am

You can not start multiple Rserve daemons, they will try to bind to the same port therefore the error you saw. You can however setup several Rserve TCP sessions and with in each session you can do whatever R code you want with name space isolation.

jicun

bgoodri · July 12, 2017, 7:53pm

I’m still not sure exactly how to test this, but I committed what might be a fix on GitHub.

jzh · July 13, 2017, 7:13am

I have found the solution to this. All you need to do is to include the thread id into the hash that is used to generate the temp file name you created. That way different sessions will have its own file to work with not conflicting each other.

jicun

Topic		Replies	Views
RDS issues in Rstan with parallel session RStan	8	1598	November 21, 2017
Rstan crashes after first iteration on one of two Windows machines RStan	1	1431	June 5, 2017
cmdstanR and rstan random run failures on computing cluster General rstan , cmdstanr	0	126	May 11, 2024
Issue running rstan models using foreach RStan rstan	5	598	July 12, 2021
Error running stan with linux RStan	2	395	July 12, 2021

Rstan fails in multiple Rserve sessions

define BOOST_NO_CXX11_RVALUE_REFERENCES

define BOOST_NO_CXX11_RVALUE_REFERENCES

Related topics