Troubleshooting multithreading with map_rect in RStan

Interfaces RStan
1

I have been running a model in RStan on an Ubuntu computer successfully. I am trying to enable multithreading with map_rect for the first time. After wading through several syntax errors, I now arrive at the following error message. It is the same message as seen in two posts here and here.
The solution in both prior posts was to update the BH package in R. But in my case, BH was already on version 1.72.0.3. Reinstalling it did not solve the problem.

Operating system: Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-78-generic x86_64)

I created the file ~/.R/Makevars and added this line.
CXX14FLAGS += -DSTAN_THREADS

I added this line to my R code
Sys.setenv(STAN_NUM_THREADS = 5)

Then I rewrote my model to fit map_rect. There’s probably a mathematical error somewhere in my model. I’m not sure if errors in calculating ‘target’ should be creating errors with compilation, but I can paste my Stan code if it is helpful.

In R, I see:

rstan (Version 2.19.3, GitRev: 2e1f913d3ca3)
R version 3.4.4
Platform: x86_64-pc-linux-gnu (64-bit)

.libPaths()
[1] “/home/eliaw/R/x86_64-pc-linux-gnu-library/3.4”
[2] “/usr/local/lib/R/site-library”
[3] “/usr/lib/R/site-library”
[4] “/usr/lib/R/library”
packageVersion(“BH”)
[1] ‘1.72.0.3’
writeLines(readLines(file.path(Sys.getenv(“HOME”), “.R/Makevars”)))
CXX14FLAGS += -DSTAN_THREADS
devtools::session_info(“rstan”)
Error in loadNamespace(name) : there is no package called ‘devtools’

And here is the error message:

Error in compileCode(f, code, language = language, verbose = verbose) :
  Compilation ERROR, function(s)/method(s) not created! filea67a67f77f5.cpp:6:36: warning: ISO C++11 requires whitespace after the macro name
 #define STAN__SERVICES__COMMAND_HPP#include <boost/integer/integer_log2.hpp>
                                    ^
In file included from /home/eliaw/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/smart_ptr/shared_ptr.hpp:28:0,
                 from /home/eliaw/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/shared_ptr.hpp:17,
                 from /home/eliaw/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/date_time/time_clock.hpp:17,
                 from /home/eliaw/R/x86_64-pc-linux-gnu-library/3.4/BH/include/boost/date_time/posix_time/posix_time_types.hpp:10,
                 from /home/eliaw/R/x86_64-pc-linux-gnu-library/3.4/rstan/include/rstan/stan_fit.hpp:13,
                 from /home/eliaw/R/x86_64-pc-linux-gnu-library/3.4/rstan/include/rstan/rstaninc.hpp:3,
                 from filea67a67f77f5.cpp:
In addition: Warning message:
running command '/usr/lib/R/bin/R CMD SHLIB filea67a67f77f5.cpp 2> filea67a67f77f5.cpp.err.txt' had status 1
Error in sink(type = "output") : invalid connection

Thanks in advance!

2

Need to call stan or sampling with verbose = TRUE to see the actual error message.

3

Ah, you are right. I now see the message:

filebdbae32f2bc.cpp:240:25: error: no matching function for call to ‘integrate_ode_rk45’

which is something I can fix.

4

Suppose I have the following user defined functions:
function_for_map
wrapper_function
function_for_solver

In my model I call map_rect(function_for_map, ...)
function_for_map calls wrapper_function on a shard.
Inside wrapper_function, I call integrate_ode_rk45(function_for_solver, ...).

Is this legal? Or would this cause the “no matching function for call to ‘integrate_ode_rk45’” error I am still encountering?
Before I rewrote my model block inside function_for_map, the ODE solver and function_to_integrate were performing correctly.

5

If you can switch to cmdstan then maybe try reduce sum in 2.23

6

I was able to get multithreading to work by 1) fixing some indexing typos when I was unpacking the x_r argument to map_rect, 2) not calling the wrapper function, 3) and not creating local variables in the function_for_map function body to contain unpacked values of x_r and x_i, which were then passed to integrate_ode_rk45. If I did create a local variable real my_times[nTimes]; my_times = x_r[1:nTimes]; and passed it to integrate_ode_rk45, then the compiler would complain that only values from data could be used. Instead I had to use pass x_r[1:nTimes] exactly.

7

EDIT:
I got RStan to compile my model and begin sampling. I’m not sure which thing I changed fixed the problem, but the last thing I did was to make my .R/Makevars contain the following flags:

CXX14 = g++ -std=c++1y
CXX14FLAGS=-O3 -march=native -mtune=native -fPIC -Wno-unused-variable -Wno-unused-function
CXX14FLAGS += -DSTAN_THREADS -pthread

I also had reinstalled Rcpp, RcppEigen, StanHeaders, and rstan through R’s install.packages() after I did module add gcc/9.2.0 while in the cluster’s login node and with a Makevars that contained

CXX14 = g++ -std=c++1y
CXX14STD='-std=c++11'
CXX14FLAGS=-O3 -march=native -mtune=native -fPIC -Wno-unused-variable -Wno-unused-function
CXX14FLAGS += -DSTAN_THREADS

I got a lot more compiler warnings (but no errors) when re-installing that time around, so something had changed. I now suspect that the CXX14STD flag was preventing compilation. The default compiler before module add gcc/9.2.0 was 4.8.5.

Now I am trying to move my model to a remote cluster that uses Slurm.
Unfortunately, I get a segfault now. This is not due just to map_rect, as I running example(stan_model, run.dontrun = TRUE) gives me a segfault as well.
I tried re-installing Rcpp, RcppEigen, StanHeaders, and rstan as done here: https://github.com/stan-dev/rstan/issues/333, but same segfault message appears.
Not sure if this is the same issue: https://github.com/stan-dev/rstan/issues/444
Any hints?

operating system: CentOS Linux 7

the submission script:

#!/bin/bash
#SBATCH --job-name=200521_1a
#SBATCH --output=slurm.200521_1a.out
#SBATCH --error=slurm.200521_1a.out
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task 4
#SBATCH --mem-per-cpu=20G
#SBATCH --time=00:05:00
#SBATCH --qos=debug
module load R
module add gcc/9.2.0
which R
cd 200521_1a
time Rscript test.R

Inside test.R:

library(rstan)
library(Rcpp)
example(stan_model, run.dontrun = TRUE)

Inside ~/.R/Makevars:

CXX14 = g++ -std=c++1y
CXX14FLAGS += -fPIC -Wno-unused-variable -Wno-unused-function
CXX14FLAGS += -DSTAN_THREADS

The stdout messages I get:

TRANSLATING MODEL '16a540c6086086816528e4524def24d9' FROM Stan CODE TO C++ CODE NOW.
successful in parsing the Stan model '16a540c6086086816528e4524def24d9'.
OS: x86_64, linux-gnu; rstan: 2.19.3; Rcpp: 1.0.4.6; inline: 0.3.15 
 >> setting environment variables: 
PKG_LIBS =  -L'/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/StanHeaders/lib/' -lStanHeaders
PKG_CPPFLAGS =   -I"/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/Rcpp/include/"  -I"/central/home/eliaw/R/x86_64-pc-linux-gnu-li
brary/3.6/RcppEigen/include/"  -I"/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/RcppEigen/include/unsupported"  -I"/central/home/
eliaw/R/x86_64-pc-linux-gnu-library/3.6/BH/include" -I"/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/StanHeaders/include/src/"  -
I"/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/StanHeaders/include/"  -I"/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/r
stan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Ei
gen.hpp 
 >> Program source :

   1 : 
   2 : // includes from the plugin
   3 : // [[Rcpp::plugins(cpp14)]]
   4 : 
   5 : // user includes
   6 : #define STAN__SERVICES__COMMAND_HPP#include <boost/integer/integer_log2.hpp>
   7 : #include <rstan/rstaninc.hpp>
   8 : // Code generated by Stan version 2.19.1
   9 : 
  10 : #include <stan/model/model_header.hpp>
  11 : 
  12 : namespace model104f87d714c42_16a540c6086086816528e4524def24d9_namespace {
  13 :

…

 349 : // declarations
 350 : extern "C" {
 351 : SEXP file104f821e84681( ) ;
 352 : }
 353 : 
 354 : // definition
 355 : 
 356 : SEXP file104f821e84681(  ){
 357 :  return Rcpp::wrap("16a540c6086086816528e4524def24d9");
 358 : }
 359 : 
 360 : 
Compilation argument:
 /central/software/R/3.6.1/lib64/R/bin/R CMD SHLIB file104f821e84681.cpp 2> file104f821e84681.cpp.err.txt 
g++ -std=c++1y  -I"/central/software/R/3.6.1/lib64/R/include" -DNDEBUG   -I"/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/Rcpp/in
clude/"  -I"/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/RcppEigen/include/"  -I"/central/home/eliaw/R/x86_64-pc-linux-gnu-libra
ry/3.6/RcppEigen/include/unsupported"  -I"/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/BH/include" -I"/central/home/eliaw/R/x86_
64-pc-linux-gnu-library/3.6/StanHeaders/include/src/"  -I"/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/StanHeaders/include/"  -I
"/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PEN
DING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -I/usr/local/include     -fPIC -Wno-unused-variable -Wno-unused-function -DSTAN_THREADS -c file104f821e84681.cpp -o file104f821e84681.o
g++ -std=c++1y -shared -L/central/software/R/3.6.1/lib64/R/lib -L/usr/local/lib64 -o file104f821e84681.so file104f821e84681.o -L/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/StanHeaders/lib/ -lStanHeaders -L/central/software/R/3.6.1/lib64/R/lib -lR
file104f821e84681.cpp:6:9: warning: ISO C++11 requires whitespace after the macro name
    6 | #define STAN__SERVICES__COMMAND_HPP#include <boost/integer/integer_log2.hpp>
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/RcppEigen/include/Eigen/Core:392,
                 from /central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/RcppEigen/include/Eigen/Dense:1,
                 from /central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:4,
                 from <command-line>:
/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/RcppEigen/include/Eigen/src/Core/arch/SSE/PacketMath.h:60:39: warning: ignoring attributes on template argument ‘__m128’ {aka ‘__vector(4) float’} [-Wignored-attributes]
   60 | template<> struct is_arithmetic<__m128>  { enum { value = true }; };
      |                                       ^

… many warnings, then

/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/RcppEigen/include/Eigen/src/Core/ProductEvaluators.h:35:90: required from ‘Eigen::internal::evaluator<Eigen::Product<Lhs, Rhs, Option> >::evaluator(const XprType&) [with Lhs = Eigen::Product<Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<double>, const Eigen::Matrix<double, 1, -1> >, const Eigen::Transpose<Eigen::Matrix<double, -1, 1> > >, Eigen::Matrix<double, -1, -1>, 0>; Rhs = Eigen::Matrix<double, -1, 1>; int Options = 0; Eigen::internal::evaluator<Eigen::Product<Lhs, Rhs, Option> >::XprType = Eigen::Product<Eigen::Product<Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<double>, const Eigen::Matrix<double, 1, -1> >, const Eigen::Transpose<Eigen::Matrix<double, -1, 1> > >, Eigen::Matrix<double, -1, -1>, 0>, Eigen::Matrix<double, -1, 1>, 0>]’
/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/RcppEigen/include/Eigen/src/Core/Product.h:132:22: required from ‘Eigen::internal::dense_product_base<Lhs, Rhs, Option, 6>::operator const Scalar() const [with Lhs = Eigen::Product<Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op<double>, const Eigen::Matrix<double, 1, -1> >, const Eigen::Transpose<Eigen::Matrix<double, -1, 1> > >, Eigen::Matrix<double, -1, -1>, 0>; Rhs = Eigen::Matrix<double, -1, 1>; int Option = 0; Eigen::internal::dense_product_base<Lhs, Rhs, Option, 6>::Scalar = double]’
/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/StanHeaders/include/src/stan/mcmc/hmc/hamiltonians/dense_e_metric.hpp:24:60: required from ‘double stan::mcmc::dense_e_metric<Model, BaseRNG>::T(stan::mcmc::dense_e_point&) [with Model = model104f87d714c42_16a540c6086086816528e4524def24d9_namespace::model104f87d714c42_16a540c6086086816528e4524def24d9; BaseRNG = boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> >]’
/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/StanHeaders/include/src/stan/mcmc/hmc/hamiltonians/dense_e_metric.hpp:23:14: required from here
/central/home/eliaw/R/x86_64-pc-linux-gnu-library/3.6/RcppEigen/include/Eigen/src/Core/DenseCoeffsBase.h:55:30: warning: ignoring attributes on template argument ‘Eigen::internal::packet_traits<double>::type’ {aka ‘__vector(2) double’} [-Wignored-attributes]

stn_md> fit <- sampling(mod, data = list(y_mean = 0))

 *** caught segfault ***
address 0x20, cause 'memory not mapped'

Traceback:
 1: Module(module, mustStart = TRUE)
 2: .getModulePointer(x)
 3: new("Module", .xData = <environment>)$stan_fit4model104f87d714c42_16a540c6086086816528e4524def24d9
 4: new("Module", .xData = <environment>)$stan_fit4model104f87d714c42_16a540c6086086816528e4524def24d9
 5: eval(call("$", mod, paste("stan_fit4", model_cppname, sep = "")))
 6: eval(call("$", mod, paste("stan_fit4", model_cppname, sep = "")))
 7: object@mk_cppmodule(object)
 8: .local(object, ...)
 9: sampling(mod, data = list(y_mean = 0))
10: sampling(mod, data = list(y_mean = 0))
11: eval(ei, envir)
12: eval(ei, envir)
13: withVisible(eval(ei, envir))
14: source(tf, local, echo = echo, prompt.echo = paste0(prompt.prefix,     getOption("prompt")), continue.echo = paste0(prompt.prefix,     getOption("continue")), verbose = verbose, max.deparse.length = Inf,     encoding = "UTF-8", skip.echo = skips, keep.source = TRUE)
15: example(stan_model, run.dontrun = TRUE)
An irrecoverable exception occurred. R is aborting now ...
/var/spool/slurmd/job9926063/slurm_script: line 24: 66808 Segmentation fault      Rscript test.R