Model Compilation error on Centos7 Cluster - cmdstanr

Interfaces CmdStan
1

I am using R on the Centos7 cluster, and successfully installed all the required packages including rstan and cmdstanr. First loaded the required modules:

module load gcc/8.3.0
module load R/4.0.3
module load gsl/2.6
R

and installed the required packages succesfully, and I tried the following models:

library(cmdstanr)
stan_program <- write_stan_tempfile("
parameters {
  real x;
}
model {
  real mu = -2;
  x ~ normal(fabs(mu), 1);
}
"
)
mod <- cmdstan_model(stan_program)
fit <- mod$sample()

and

library(rstan)
example(stan_model, package = "rstan", run.dontrun = TRUE)

both of them worked fine and didn’t get any error. After all, I am trying to submit the job with bash script, but getting this compilation error.

mod <- cmdstan_model('model3.stan')
Compiling Stan program...
In file included from /share/builds/spack-compilers/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/gcc-8.3.0-mnwujmnexs6zvenuhlu5m4io2om4sg55/include/c++/8.3.0/x86_64-pc-linux-gnu/bits/c++config.h:508,
                 from /share/builds/spack-compilers/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/gcc-8.3.0-mnwujmnexs6zvenuhlu5m4io2om4sg55/include/c++/8.3.0/new:39,
                 from stan/lib/stan_math/lib/eigen_3.3.9/Eigen/Core:82,
                 from stan/lib/stan_math/lib/eigen_3.3.9/Eigen/Dense:1,
                 from stan/lib/stan_math/stan/math/prim/fun/Eigen.hpp:22,
                 from stan/lib/stan_math/stan/math/rev.hpp:4,
                 from stan/lib/stan_math/stan/math.hpp:19,
                 from stan/src/stan/model/model_header.hpp:4,
                 from /tmp/RtmpDzWg2n/model-5a62383c457c.hpp:3:
/share/builds/spack-compilers/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/gcc-8.3.0-mnwujmnexs6zvenuhlu5m4io2om4sg55/include/c++/8.3.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: fatal error: features.h: No such file or directory
 #include <features.h>
          ^~~~~~~~~~~~
compilation terminated.
make: *** [/tmp/RtmpDzWg2n/model-5a62383c457c] Error 1
Error: An error occured during compilation! See the message above for more information.
Execution halted

If anybody can help with this, I would really appreciate it.

**I do not have admin rights.

1 Like
2

Hi,
so one possible reason is that in the job, you have a differently set environment (path or other environmental variables), could you check if the env variables are the same?

3

Hi @martinmodrak ,

Thank you for your response. HPC that I’ve been using primarily running two types of jobs: batch and interactive. Stan model compiling on interactive queue but when I submit it as a batch job having the same error. Could you please let me know, how can I check the environmental variables? Thanks!

4

Hi @martinmodrak ,

I tried to compile the same model with rstan and got this error:

Error in compileCode(f, code, language = language, verbose = verbose) : 
                   from <command-line>:/share/builds/spack-compilers/opt/spack/linux-centos7-sandybridge/gcc-8.3.1/gcc-8.3.0-mnwujmnexs6zvenuhlu5m4io2om4sg55/include/c++/8.3.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: fatal error: features.h: No such file or directory #include <features.h>          ^~~~~~~~~~~~compilation terminated.make: *** [/nethome/mxk841/anaconda3/envs/r4_Mrt/lib/R/etc/Makeconf:177: file8c0218b5be9.o] Error 1
Calls: stan ... cxxfunctionplus -> <Anonymous> -> cxxfunction -> compileCode
Error in sink(type = "output") : invalid connection
Calls: stan -> stan_model -> cxxfunctionplus -> sink
Execution halted
5

Hey,

is not a supported compiler on Linux according to 1 CmdStan Installation | CmdStan User’s Guide. I got cmdstanr to work on a Centos 7 cluster easily by following the conda installation in that link, installing CmdStan and the C++ compiler with

conda create -n stan-env -c conda-forge cmdstan

and then before submitting a batch script activate the environment with

conda activate stan-env

Then the cmdstanr startup message says
- CmdStan path set to: ~/.conda/envs/stan-env/bin/cmdstan

1 Like
6

Hi @jtimonen ,

First of all, thank you so much for the help. I was using the conda env also, and I tried your suggestions. Usually at the beginning I was loading modules like gsl and gcc. This time I did not load gcc/8.3.1 module:

(r4_Mrt) [mxk841@login4 testsecurity]$ conda create -n stan-env -c conda-forge cmdstan

It forged well and installed the required packages but I did not load R and install them again. Then I activated the stan environment:

(r4_Mrt) [mxk841@login4 testsecurity]$ conda activate stan-env

However , after the activation I did not get the startup message with the path like you had. Then submit the bash script :

(stan-env) [mxk841@login4 testsecurity]$ bsub < dglnrt_re_conda.sh

Here is the bash script I have:

(stan-env) [mxk841@login4 testsecurity]$ cat  dglnrt_re_conda.sh
#!/bin/bash
#BSUB -J cmd_stan
#BSUB -P testsecurity
#BSUB -R "span[hosts=1]"
#BSUB -q bigmem
#BSUB -n 3
#BSUB -B
#BSUB -u iverskyliners@hotmail.com
#BSUB -N
#BSUB -o cmd_stan.%J.out
#BSUB -e cmd_stan.%J.err
#BSUB -R "rusage[mem=4000]"

. "/nethome/mxk841/anaconda3/etc/profile.d/conda.sh"
conda activate r4_Mrt
cd /scratch/projects/testsecurity/dglnrt_re_conda.R

R CMD BATCH dglnrt_re_conda.R

(Also tried to activate stan env inside the bash script but got the same error)

After submitting the bash script, it provided following error:

mod <- cmdstanr::cmdstan_model('dglnrt_re_conda.stan')
g++: error: unrecognized command line option ‘-std=c++17’
g++: error: unrecognized command line option ‘-fno-plt’
make: *** [make/program:58: /tmp/RtmpcNfKSF/model-6f24490dc687] Error 1
Error: An error occured during compilation! See the message above for more information.
Execution halted

Not#1: When I load older gcc modules (4.7.3 or 5.5.0), I am having the same compilation error.
Not#2: When I load gcc/8.3.1, I am getting the features.h error.

7

It is not invoked by activation but library(cmdstanr) in your R script and will go to output (cmd_stan.%J.out).

You are activating a different conda environment that stan-env here. I don’t have any conda commands
in the batch script, only module load R/ver where ver is the R version to which I have installed cmdstanr.

8

I just updated the bash script, but still same error:

(stan-env) [mxk841@login4 testsecurity]$ cat  dglnrt_re_conda.sh
#!/bin/bash
#BSUB -J cmd_stan
#BSUB -P testsecurity
#BSUB -R "span[hosts=1]"
#BSUB -q bigmem
#BSUB -n 3
#BSUB -B
#BSUB -u iverskyliners@hotmail.com
#BSUB -N
#BSUB -o cmd_stan.%J.out
#BSUB -e cmd_stan.%J.err
#BSUB -R "rusage[mem=4000]"

module load R/4.1.0

R CMD BATCH dglnrt_re_conda.R

Same error is produced:

mod <- cmdstanr::cmdstan_model('dglnrt_re_conda.stan')
g++: error: unrecognized command line option ‘-std=c++17’
g++: error: unrecognized command line option ‘-fno-plt’
make: *** [make/program:58: /tmp/RtmpcNfKSF/model-6f24490dc687] Error 1
Error: An error occured during compilation! See the message above for more information.
Execution halted
1 Like
9

I guess the docs are written in a weird way here. Cmdstan supports g++ 4.9.3 or newer.

1 Like
10

Hi @rok_cesnovar @martinmodrak @jtimonen ,

Do you have any further suggestions? Thanks.