Error 126 preventing brms backend = cmdstan in HPC

Developers
, , ,
1

Hello thank you for the help

I am trying to run R code on a High Performance Compute Cluster at my university and I can’t seem to get beyond a certain point getting brms to use backend=cmdstan
I have gotten other brms jobs to work, but I fear the compilation step is not being done properly.
Currently, I attempt to import the cmdstan amd64 tar file using install_cmdstan and I get stuck at

--- Translating Stan model to C++ code ---
bin/stanc --o=examples/bernoulli/bernoulli.hpp examples/bernoulli/bernoulli.stan
bash: bin/stanc: cannot execute binary file: Exec format error
make: *** [make/program:50: examples/bernoulli/bernoulli.hpp] Error 126
Warning message:
There was a problem during installation. See the error message(s) above.
[1] "running scripts"

Here is the following relevant R code

install.packages("packages/loo_2.7.0.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/Matrix_1.6-5.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/inline_0.3.19.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/QuickJSR_1.1.3.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/RcppEigen_0.3.4.0.0.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/RcppParallel_5.1.7.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/BH_1.84.0-0.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/StanHeaders_2.32.6.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/colourpicker_1.3.0.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/threejs_0.3.3.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/dygraphs_1.1.1.6.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/shinythemes_1.2.0.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/Brobdingnag_1.2-9.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/rlang_1.1.3.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/scales_1.3.0.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/vctrs_0.6.5.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/lifecycle_1.0.4.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/ggplot2_3.5.0.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/rstan_2.32.5.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/rstantools_2.4.0.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/posterior_1.5.0.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/cmdstanr_0.7.1.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/bayesplot_1.11.1.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/shinystan_2.6.0.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/bridgesampling_1.1-2.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages("packages/nleqslv_3.3.5.tar.gz", lib = ".", repos = NULL, type = "source")
install.packages('packages/brms_2.20.4.tar.gz', lib = ".", repos = NULL, type = "source")
library("loo", lib.loc = "./")
library("Matrix", lib.loc = "./")
library("inline", lib.loc = "./")
library("QuickJSR", lib.loc = "./")
library("RcppEigen", lib.loc = "./")
library("RcppParallel", lib.loc = "./")
library("BH", lib.loc = "./")
library("StanHeaders", lib.loc = "./")
library("colourpicker", lib.loc = "./")
library("threejs", lib.loc = "./")
library("dygraphs", lib.loc = "./")
library("shinythemes", lib.loc = "./")
library("Brobdingnag", lib.loc = "./")
library("rlang", lib.loc = "./")
library("scales", lib.loc = "./")
library("vctrs", lib.loc = "./")
library("lifecycle", lib.loc = "./")
library("ggplot2", lib.loc = "./")
library("rstan", lib.loc = "./")
library("rstantools", lib.loc = "./")
library("posterior", lib.loc = "./")
library("cmdstanr", lib.loc = "./")
library("bayesplot", lib.loc = "./")
library("brms", lib.loc = "./")
library("pracma")

rstan_options(auto_write = TRUE)
options(mc.cores = parallel::detectCores())
install_cmdstan(
  dir = "./",
  cores = getOption("mc.cores", mc.cores),
  quiet = FALSE,
  overwrite = FALSE,
  timeout = 1200,
  version = NULL,
  release_url = NULL,
  release_file = "./packages/cmdstan-2.34.1.tar.gz",
  cpp_options = list(),
  check_toolchain = TRUE,
  wsl = FALSE
)

data = readRDS('./data50.brm.RDS')
print("running scripts")
run_model_prior <- function(md, data, seed = 1, iter_prior = 100, warmup_prior = 16, chains = 1, cores = 16, backend = "cmdstanr"){
  command_prior = sprintf('tomodpriors = brm(%s, data = data, backend = backend,chains = chains, iter = iter_prior, warmup = warmup_prior, seed = seed, cores = cores)', md)   
...some code here
}
md = "graymatter~ scale(  Total_cog  ) + scale(Age_yrs) + Gender + scale(BMI) + scale(walk_pace) + scale(Education_yrs) + scale(EstimatedTotalIntraCranialVol) + (1 + scale(  Total_cog  ) + scale(Age_yrs) + Gender + scale(BMI)   + scale(walk_pace) + scale(Education_yrs) + scale(EstimatedTotalIntraCranialVol) | regionlvl1b) + (1 | ID)"
print("running run_model_prior")
set_cmdstan_options(error_on_unknown_arg = TRUE)
temp = run_model_prior(md, data = data)

Edited by @jsocolar for code formatting

2

The R code you have provided doesn’t run install_cmdstan.

Can you show us a condensed, minimal R script that reproduces the error you’re trying to troubleshoot? It would help if you can avoid showing us any code that isn’t strictly necessary to reproduce the error.

Also, welcome to the Stan forums! You’re in the right place to figure this out :)

3

Thanks for the welcome!

I have updated my code to include the install_cmdstan step.
I believe my HPC cluster has it available as a module, but I ran into some problems trying to use it. I just don’t know how to prep it properly as backend=cmdstan, and in that iteration of the code I did not install_cmdstan at all.

Warning: The parameter sigma has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
Warning: The parameter sd_2 has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
Warning: The parameter sd_1 has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
Warning: The parameter b has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
Warning: The parameter L_2 has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
Warning: The parameter Intercept has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.

Start sampling
Chain 1 [compute-a8-22:21193:0:21193] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))

Chain 1 ==== backtrace (tid:  21193) ====
....
more chain errors here

The code is identical to the one provided above but with the following section commented out

#rstan_options(auto_write = TRUE)
#options(mc.cores = parallel::detectCores())
#install_cmdstan(
#  dir = "./",
#  cores = getOption("mc.cores", mc.cores),
#  quiet = FALSE,
#  overwrite = FALSE,
#  timeout = 1200,
#  version = NULL,
#  release_url = NULL,
#  release_file = "./packages/cmdstan-2.34.1.tar.gz",
#  cpp_options = list(),
#  check_toolchain = TRUE,
#  wsl = FALSE
#)
4

What happens if you do whatever you need to do to install brms, cmdstanr and cmdstan (and nothing else), and then do

brm(Sepal.Length ~ Sepal.Width, data = iris, backend = "cmdstanr")
5

I see a similar error.
Weird thing is, there are less warnings about priors.
Not sure what to make of it

Warning: The parameter sigma has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
Warning: The parameter b has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
Warning: The parameter Intercept has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.

Start sampling
Chain 1 [compute-b5-54:478543:0:478543] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x19)
Chain 1 ==== backtrace (tid: 478543) ====
6

The original had this amount of parameter warnings:

Warning: The parameter sigma has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
Warning: The parameter sd_2 has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
Warning: The parameter sd_1 has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
Warning: The parameter b has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
Warning: The parameter L_2 has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
Warning: The parameter Intercept has no priors. This means either no prior is
    provided, or the prior(s) depend on data variables. In the later case,
    this may be a false positive.
7

This looks like an issue with your cluster, not an issue with Stan per se. Try increasing the memory or unlimiting the stacksize. Ask your university’s HPC if they agree with these potential fixes, and if so how to proceed.