BRMS on linux cluster file permissions for "expose_functions"

Interfaces brms
1

I would like to run a series of evaluations of a simple-ish regression model with a custom family on a Linux cluster. BRMS and cmdstanr are both installed.

  • cmdstanr version 0.7.1
  • CmdStan version: 2.34.1
  • brms version 2.20.4
  • os RHEL fedora 8.2

In cmdstanr, I can compile a model and then call expose_functions.

mod <- cmdstanr::cmdstan_model("prev_3preds.stan")
Compiling Stan program...
> mod$expose_functions()
Compiling standalone functions...

In BRMS, I get a file permissions error. I was able to compile the file in the current directory, but the expose_functions still got a segfault, below:

This is cmdstanr version 0.7.1
- CmdStanR documentation and vignettes: mc-stan.org/cmdstanr
- CmdStan path: /burg/home/mm2368/.cmdstan/cmdstan-2.34.1
- CmdStan version: 2.34.1
Loading required package: Rcpp
Loading 'brms' package (version 2.20.4). Useful instructions
can be found by typing help('brms'). A more detailed introduction
to the package is available through vignette('brms_overview').

.... 

 *** caught segfault ***
address 0x1555370d9008, cause 'invalid permissions'

Traceback:
 1: dyn.load("/tmp/RtmpNpVa45/sourceCpp-x86_64-pc-linux-gnu-1.0.12/sourcecpp_38983b5b3630e9/sourceCpp_2.so")
 2: eval(ei, envir)
 3: eval(ei, envir)
 4: withVisible(eval(ei, envir))
 5: source(scriptPath, local = env)
 6: Rcpp::sourceCpp(code = code, env = env, verbose = verbose)
 7: force(code)
 8: force(code)
 9: with_envvar(c(R_MAKEVARS_USER = makevars_file), {    set_makevars(new, path, makevars_file, assignment = assignment)    force(code)})
10: withr::with_makevars(c(USE_CXX14 = 1, PKG_CPPFLAGS = ifelse(cmdstan_version() <=     "2.30.1", "-DCMDSTAN_JSON", ""), PKG_CXXFLAGS = paste0(cxxflags,     cmdstanr_includes, collapse = " "), PKG_LIBS = libs), Rcpp::sourceCpp(code = code,     env = env, verbose = verbose))
11: force(code)
12: withr::with_path(paste0(cmdstan_path(), lib_paths), withr::with_makevars(c(USE_CXX14 = 1,     PKG_CPPFLAGS = ifelse(cmdstan_version() <= "2.30.1", "-DCMDSTAN_JSON",         ""), PKG_CXXFLAGS = paste0(cxxflags, cmdstanr_includes,         collapse = " "), PKG_LIBS = libs), Rcpp::sourceCpp(code = code,     env = env, verbose = verbose)))
13: rcpp_source_stan(mod_stan_funs, env, verbose)
14: compile_functions(function_env, verbose, global)
15: expose_stan_functions(self$functions, global, verbose)
16: stanmodel$expose_functions()
17: withCallingHandlers(expr, message = function(c) if (inherits(c,     classes)) tryInvokeRestart("muffleMessage"))
18: suppressMessages(stanmodel$expose_functions())
19: .expose_functions_cmdstanr(stanmodel, vectorize = vectorize,     env = env, ...)
20: expose_functions.brmsfit(gen_fit, vectorize = TRUE)

what are the set of BRMS options that will work here? it seems like BRMS is trying to use Rstan’s, expose_functions, not cmdstanr?

@rok_cesnovar ?

2 Likes
2

I don’t know why there’s a file permission error, but I think you’re right that brms is trying to use rstan to expose the functions. That’s probably because cmdstanr didn’t used to have this functionality, so brms would always have to call rstan::expose_stan_functions to expose functions even if the user wanted to fit the model using cmdstanr. Maybe brms hasn’t been updated since cmdstanr added this functionality?

Are you able to use rstan::expose_stan_functions outside of brms? I wonder if you’d get the same file permissions error. If you want to test it you could try the simple example in the rstan doc: Expose user-defined Stan functions to R for testing and simulation — expose_stan_functions • rstan

3

other posts and issues on GitHub suggest that this can be done - Spooky segfault from rstan + cmdstanr:::expose_functions. and exposing functions from models with backend cmdstanr · Issue #1176 · paul-buerkner/brms · GitHub

4

You need to add STAN_THREADS=true to the cmdstan make/local file, to ensure that both the cmdstan executable and cmdstanr expose functions routines are compiled with threading enabled.

The issue occurs because Stan uses a global static pointer for the AD stack. An AD stack pointer is initialised when rstan is loaded (happens automatically when brms is loaded), which causes an error when the cmdstanr-exposed functions are loaded and another AD stack pointer is attempted to be initialised.

By compiling with STAN_THREADS, the AD stack pointers are thread-local and do not interfere with each other

2 Likes