Unable to use expose_functions method with CmdStanR on Linux HPC cluster

Interfaces CmdStan
1

Summary

I am unable to use the expose_functions method of a CmdStanModel, using the CmdStanR interface on a Linux HPC cluster. This issue came up in a number of recent topics:
https://discourse.mc-stan.org/t/tbb-interface-new-flag-is-not-working/35664
https://discourse.mc-stan.org/t/segfault-when-using-brms-cmdstanr-compile-model-methods-true/33771
https://discourse.mc-stan.org/t/brms-on-linux-cluster-file-permissions-for-expose-functions/34242
The solution given in those topics was to add the flag USE_STAN_THREADS = TRUE to the cmdstan make/local file (and then rebuild cmdstan). I’ve done that, but the issue is still there.

To be clear, apart from this issue involving the expose_functions method, everything else involving cmdstan seems to run perfectly fine. I can compile models, sample from them, inspect the results, etc.

Details

  • Operating system: RHEL 8.8
  • cmdstan version: 2.35.0
  • R version: 4.1.2
  • cmdstanr version: 0.8.1.9000

Example script and output:

$ salloc -t 60 -n 1 --mem=10gb -p dev_single
[redacted]
$ module purge
$ module load math/R/4.1.2
Loading module dependency 'compiler/intel/2021.4.0'.
Loading module dependency 'numlib/mkl/2021.4.0'.
$ module load lib/icu4c/74.1_intel2021.4.0
$ module load devel/tbb/2021.4.0
$ export TBB_LIB=$TBB_LIB_DIR
$ export TBB_INC=$TBB_INC_DIR
$ export TBB_INTERFACE_NEW='true'
$ R -q

library(cmdstanr)

This is cmdstanr version 0.8.1.9000
- CmdStanR documentation and vignettes: mc-stan.org/cmdstanr
- Use set_cmdstan_path() to set the path to CmdStan
- Use install_cmdstan() to install CmdStan

set_cmdstan_path("~/opt/cmdstan")

CmdStan path set to: /home/[redacted]/opt/cmdstan

check_cmdstan_toolchain()

The C++ toolchain required for CmdStan is setup properly!

mod <- cmdstan_model(stan_file = "test_expose.stan")
mod$check_syntax(pedantic = TRUE)

Stan program is syntactically correct

mod$print()

functions {
  real a_plus_b(real a, real b) {
    return a + b;
  }
}

mod$expose_functions(global = TRUE)

This returns a huge amount of messages to the R console, ending with:

compilation aborted for fileb247e961c461.cpp (code 4)
make: *** [/opt/bwhpc/common/math/R/4.1.2-mkl-2021.4.0-intel-2021.4.0_O3_pragma_noopt_el8_javafix_c++11/lib64/R/etc/Makeconf:177: fileb247e961c461.o] Error 4
Error in Rcpp::sourceCpp(code = code, env = env, verbose = verbose, ...) :
  Error 1 occurred building shared library.

More details

My Makevars file looks like this (following instructions from the wiki of my HPC cluster):

$ cat ~/.R/Makevars
CXX14=icpc
CXX17=icpc
CXX14FLAGS=-03 -fPIC -std=c++14 -wd308 -axCORE-AVX512,CORE-AVX2,AVX -xSSE4.2 -fp-model strict -qopenmp
CXX17FLAGS=-03 -fPIC -std=c++17 -wd308 -axCORE-AVX512,CORE-AVX2,AVX -xSSE4.2 -fp-model strict -qopenmp
CXXFLAGS += -wd308
PKG_CXXFLAGS += -std=c++14 -wd308

My cmdstan make/local file looks like this:

$ cat ~/opt/cmdstan/make/local
STAN_THREADS=true
STAN_HAS_CXX17=true
CPPFLAGS_SUNDIALS += -fPIC

Lastly, not sure if this is relevant, but I was told by the administrators of my HPC cluster to change one line in one of the make files for cmdstan, due to an “unsupported compiler option”. Specifically, in cmdstan/stan/lib/stan_math/make/compiler_flags, I changed this line:

CXX_VERSION :=  $(shell $(CXX) -dumpfullversion -dumpversion 2>&1)

into the following (i.e., removing -dumpfullversion):

CXX_VERSION :=  $(shell $(CXX) -dumpversion 2>&1)

My R session info looks like this:

Click to show my R session info 
R version 4.1.2 (2021-11-01)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux 8.8 (Ootpa)

Matrix products: default
BLAS/LAPACK: /pfs/data5/software_uc2/all/toolkit/Intel_OneAPI/mkl/2021.4.0/lib/intel64/libmkl_intel_lp64.so.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] cmdstanr_0.8.1.9000

loaded via a namespace (and not attached):
 [1] ps_1.7.7             fansi_1.0.6          utf8_1.2.4
 [4] distributional_0.4.0 R6_2.5.1             jsonlite_1.8.8
 [7] lifecycle_1.0.4      backports_1.5.0      magrittr_2.0.3
[10] posterior_1.6.0      pillar_1.9.0         rlang_1.1.4
[13] cli_3.6.3            checkmate_2.3.2      vctrs_0.6.5
[16] generics_0.1.3       tools_4.1.2          glue_1.7.0
[19] processx_3.8.4       xfun_0.41            abind_1.4-5
[22] compiler_4.1.2       pkgconfig_2.0.3      knitr_1.45
[25] tibble_3.2.1         tensorA_0.36.2.1

Many thanks in advance for your help!

2

Sorry to bump this thread, but I’d really appreciate any advice on this issue. I might be missing something obvious.
Perhaps @andrjohns you have any insights, given your solutions in other related threads?
Thanks again.

3

Sorry you’ve had to ping this again. I don’t know the answer, but…

@Jonah is the other person who was heavily involved in coding cmdstanr. Presumably the cmdstanr approach is based on the implementation in RStan, which I believe was coded by @bgoodri.

1 Like
4

The compilation when exposing functions is done using Rcpp not CmdStan, so it seems like there’s an issue affecting only Rcpp compilation since you can compile models fine using CmdStan. Can you successfully compile anything with Rcpp on the cluster you’re using? I’m curious if it’s only a problem with using Rcpp to expose Stan functions or if this is a more general issue involving using Rcpp on the cluster.

@andrjohns is definitely the expert on this, so hopefully he can take a look when he has some time if we’re not able to sort this out.

1 Like
5

Thanks Jonah (and Bob)!

I just checked and yes, it looks like I can use Rcpp to compile a function and use it in R. Specifically, I ran the following example from slide 7 of an “Introduction to Rcpp” presentation by Dirk Eddelbuettel:

Rcpp::cppFunction("
bool isOdd_cpp(int num = 10) {
  bool result = (num % 2 == 1);
  return result;
}")

This ran just fine on my system (without any error or warning messages), and worked as expected:

isOdd_cpp(42L)

[1] FALSE

isOdd_cpp(3L)

[1] TRUE

So at first glance, it doesn’t look like there’s anything wrong with Rcpp on my system.

6

Yeah Rcpp seems to be working ok. I’m wondering if there’s some issue caused by using the Intel compiler as opposed to GCC, but I’m not sure (are you allowed to use GCC on the cluster?). I’m guessing you had to change cmdstan/stan/lib/stan_math/make/compiler_flags because the Intel compiler doesn’t support -dumpfullversion and that makes me suspicious that either this is needed or that there’s some other incompatibility issue with the Intel compiler. But I could be completely wrong about that! Hopefully @andrjohns can take a peak. I’m guessing he would be able to spot the issue pretty quickly.

1 Like
7

You were right Jonah, it looks like the issue was caused by me using the Intel compiler. With the help of the support team of my HPC cluster, I’m now using a more up-to-date set-up, including R 4.4.1, GCC 13.3, and MKL 2022.2.

As far as I can tell, this has solved the issue - I can now run the example script that I provided in the original post without issues. I also no longer needed to manually tweak the CmdStan compiler flags.

Thanks again for thinking along.

8

That’s great, glad it’s working now. And thanks for following up to let us know.