Compiler problems for Rstan 2.21 inside Docker container

Christopher-Peterson · March 12, 2020, 9:05am

I’m trying to get a docker image I’ve built with rstan 2.21 (following instructions from here) to run on an HPC system.

Due to the system’s policies, I’m building the container image on my local system with Docker, then running it on the cluster with Singularity. While it seems to work without trouble on my laptop, I’m having difficulties compiling any stan models when running the container on the cluster. Error messages and other details are below.

Does anyone have any advice:

Docker Image: crpeters/docker-stan:apt-0.1
Dockerfile

8schools R script I’m trying to run:

suppressPackageStartupMessages({
  library(rstan)
  library(rlang)
  library(magrittr)
})

args = commandArgs(TRUE)

# define default variables
iter         = args[1] %>% as.integer() %|% 2000L
chains      = args[2] %>% as.integer() %|% 4L
adapt_delta = args[3] %>% as.double()  %|% 0.95


dat <- list(J = 8,
                y = c(28,  8, -3,  7, -1,  1, 18, 12),
                sigma = c(15, 10, 16, 11,  9, 11, 10, 18))

mdl = "
// saved as 8schools.stan
data {
  int<lower=0> J;         // number of schools 
  real y[J];              // estimated treatment effects
  real<lower=0> sigma[J]; // standard error of effect estimates 
}
parameters {
  real mu;                // population treatment effect
  real<lower=0> tau;      // standard deviation in treatment effects
  vector[J] eta;          // unscaled deviation from mu by school
}
transformed parameters {
  vector[J] theta = mu + tau * eta;        // school treatment effects
}
model {
  target += normal_lpdf(eta | 0, 1);       // prior log-density
  target += normal_lpdf(y | theta, sigma); // log-likelihood
}"

fit = stan(model_code = mdl, data = dat, 
           iter = iter,
           control = list(adapt_delta = adapt_delta),
           chains = chains, cores = chains)

Error Message from stan() function:

Error in compileCode(f, code, language = language, verbose = verbose) : 
  Compilation ERROR, function(s)/method(s) not created! In file included from /usr/lib/R/site-library/RcppEigen/include/Eigen/Core:383:0,
from /usr/lib/R/site-library/RcppEigen/include/Eigen/Dense:1,
from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
from <command-line>:0:
  /usr/lib/R/site-library/RcppEigen/include/Eigen/src/Core/arch/SSE/PacketMath.h:60:39: warning: ignoring attributes on template argument ‘__m128 {aka __vector(4) float}’ [-Wignored-attributes]
template<> struct is_arithmetic<__m128>  { enum { value = true }; };
^
  /usr/lib/R/site-library/RcppEigen/include/Eigen/src/Core/arch/SSE/PacketMath.h:61:40: warning: ignoring attributes on template argument ‘__m128i {aka __vector(2) long long int}’ [-Wignored-attributes]
template<> struct is_arithmetic<__m128i> { enum { value = true }; };
^
  /u
In addition: Warning message:
  In system(cmd, intern = !verbose) :
  running command '/usr/lib/R/bin/R CMD SHLIB file5b57419673dc.cpp 2> file5b57419673dc.cpp.err.txt' had status 1
Error in sink(type = "output") : invalid connection

Makevars file:

CXXFLAGS=-O3 
CXXSTD=CXX14
CXX=g++
CXX14=g++
CXX14FLAGS=-O3 -mtune=native
CXX14FLAGS+= -fPIC
CXX14FLAGS+= -march=native

session info

─ Session info ───────────────────────────────────────────────────────────────
setting  value                       
version  R version 3.6.3 (2020-02-29)
os       Ubuntu 18.04.4 LTS          
system   x86_64, linux-gnu           
ui       X11                         
language (EN)                        
collate  en_US.UTF-8                 
ctype    en_US.UTF-8                 
tz       US/Central                  
date     2020-03-12                  

─ Packages ───────────────────────────────────────────────────────────────────
package      * version   date       lib source                         
assertthat     0.2.1     2019-03-21 [2] CRAN (R 3.5.3)                 
backports      1.1.5     2019-10-02 [2] CRAN (R 3.6.1)                 
BH             1.72.0-3  2020-01-08 [2] CRAN (R 3.6.2)                 
callr          3.4.2     2020-02-12 [2] CRAN (R 3.6.2)                 
checkmate      2.0.0     2020-02-06 [2] CRAN (R 3.6.2)                 
cli            2.0.2     2020-02-28 [2] CRAN (R 3.6.3)                 
colorspace     1.4-1     2019-03-18 [2] CRAN (R 3.5.3)                 
crayon         1.3.4     2017-09-16 [2] CRAN (R 3.5.0)                 
curl           4.2       2019-09-24 [2] CRAN (R 3.6.1)                 
desc           1.2.0     2018-05-01 [2] CRAN (R 3.5.0)                 
digest         0.6.25    2020-02-23 [2] CRAN (R 3.6.3)                 
ellipsis       0.3.0     2019-09-20 [2] CRAN (R 3.6.1)                 
evaluate       0.14      2019-05-28 [2] CRAN (R 3.6.0)                 
fansi          0.4.1     2020-01-08 [2] CRAN (R 3.6.2)                 
farver         2.0.1     2019-11-13 [2] CRAN (R 3.6.1)                 
ggplot2        3.3.0     2020-03-05 [2] CRAN (R 3.6.3)                 
glue           1.3.1     2019-03-12 [2] CRAN (R 3.5.3)                 
gridExtra      2.3       2017-09-09 [2] CRAN (R 3.5.0)                 
gtable         0.3.0     2019-03-25 [2] CRAN (R 3.5.3)                 
inline         0.3.15    2018-05-18 [2] CRAN (R 3.5.0)                 
isoband        0.2.0     2019-04-06 [2] CRAN (R 3.6.0)                 
jsonlite       1.6.1     2020-02-02 [2] CRAN (R 3.6.2)                 
labeling       0.3       2014-08-23 [2] CRAN (R 3.5.0)                 
lattice        0.20-40   2020-02-19 [3] CRAN (R 3.6.2)                 
lifecycle      0.2.0     2020-03-06 [2] CRAN (R 3.6.3)                 
loo            2.2.0     2019-12-19 [2] CRAN (R 3.6.2)                 
magrittr       1.5       2014-11-22 [2] CRAN (R 3.5.0)                 
MASS           7.3-51.5  2019-12-20 [3] CRAN (R 3.6.2)                 
Matrix         1.2-18    2019-11-27 [3] CRAN (R 3.6.1)                 
matrixStats    0.55.0    2019-09-07 [2] CRAN (R 3.6.1)                 
mgcv           1.8-31    2019-11-09 [3] CRAN (R 3.6.1)                 
munsell        0.5.0     2018-06-12 [2] CRAN (R 3.5.0)                 
nlme           3.1-144   2020-02-06 [3] CRAN (R 3.6.2)                 
pillar         1.4.3     2019-12-20 [2] CRAN (R 3.6.2)                 
pkgbuild       1.0.6     2019-10-09 [2] CRAN (R 3.6.1)                 
pkgconfig      2.0.3     2019-09-22 [2] CRAN (R 3.6.1)                 
pkgload        1.0.2     2018-10-29 [2] CRAN (R 3.5.1)                 
praise         1.0.0     2015-08-11 [2] CRAN (R 3.5.0)                 
prettyunits    1.1.1     2020-01-24 [2] CRAN (R 3.6.2)                 
processx       3.4.1     2019-07-18 [2] CRAN (R 3.6.1)                 
ps             1.3.2     2020-02-13 [2] CRAN (R 3.6.2)                 
R6             2.4.1     2019-11-12 [2] CRAN (R 3.6.1)                 
RColorBrewer   1.1-2     2014-12-07 [2] CRAN (R 3.5.0)                 
Rcpp           1.0.3     2019-11-08 [2] CRAN (R 3.6.1)                 
RcppEigen      0.3.3.7.0 2019-11-16 [2] CRAN (R 3.6.1)                 
RcppParallel   4.4.4     2019-09-27 [2] CRAN (R 3.6.1)                 
rlang          0.4.5     2020-03-01 [2] CRAN (R 3.6.3)                 
rprojroot      1.2       2017-01-16 [2] CRAN (R 3.5.0)                 
rstan          2.21.1    2020-03-12 [1] Github (stan-dev/rstan@fef82e9)
rstudioapi     0.11      2020-02-07 [2] CRAN (R 3.6.2)                 
scales         1.1.0     2019-11-18 [2] CRAN (R 3.6.1)                 
StanHeaders    2.21.0-1  2020-01-19 [1] CRAN (R 3.6.3)                 
testthat       2.3.2     2020-03-02 [2] CRAN (R 3.6.3)                 
tibble         2.1.3     2019-06-06 [2] CRAN (R 3.6.0)                 
utf8           1.1.4     2018-05-24 [2] CRAN (R 3.5.0)                 
V8             3.0.1     2020-01-22 [1] CRAN (R 3.6.3)                 
vctrs          0.2.3     2020-02-20 [2] CRAN (R 3.6.2)                 
viridisLite    0.3.0     2018-02-01 [2] CRAN (R 3.5.0)                 
withr          2.1.2     2018-03-15 [2] CRAN (R 3.5.0)                 

[1] /usr/local/lib/R/site-library
[2] /usr/lib/R/site-library
[3] /usr/lib/R/library

bgoodri · March 12, 2020, 3:46pm

Can you post the output when you add verbose = TRUE to stan or sampling.

Christopher-Peterson · March 12, 2020, 5:28pm

I’ve attached the stderr and stdout from attempting to compile the model with verbose = TRUE.

stan_errors.txt (1.2 KB) stan_output.txt (1.2 MB)

bgoodri · March 12, 2020, 5:34pm

stan_output.txt says

Compilation argument:
/usr/lib/R/bin/R CMD SHLIB file43df04ad030d.cpp 2> file43df04ad030d.cpp.err.txt
g++ -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG -I"/usr/lib/R/site-library/Rcpp/include/" -I"/usr/lib/R/site-library/RcppEigen/include/" -I"/usr/lib/R/site-library/RcppEigen/include$
g++ -std=gnu++14 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o file43df04ad030d.so file43df04ad030d.o /usr/local/lib/R/site-library/rstan/lib//libStanServices.a -L/usr/$
XALT Error: unable to find ld

My guess is that ld is installed but it cannot be found in your Docker thing for some reason.

Christopher-Peterson · March 12, 2020, 7:55pm

Thanks; I’ve determined that the issue is with part of the HPC’s ld bleeding into the container; it uses a program called XALT, which wraps ld in a script that allows the administrators to better monitor resource usage. Somehow, the container’s version of R is calling this instead of the local ld. Hopefully the HPC support staff can figure out how to resolve this now the problem has been narrowed down.

Topic		Replies	Views
Stan installed but example code fails to run. Something about Rtools? RStan rstan	31	4282	March 9, 2020
Help Building Rstan into Docker Container General rstan	4	1236	March 20, 2021
Running rstan on AWS Fargate RStan	4	422	July 8, 2022
Compilation error, rstan on HPC brms rstan	2	924	March 5, 2022
Error installing rstanarm and compiling stan code on cluster with Intel compiler RStan	7	1588	December 30, 2018

Compiler problems for Rstan 2.21 inside Docker container

Related topics