Facing error while compiling brms code

Developers
1

Hi I am compiling brms code as below:

I have picked the data from BRMS reference page

model_poisson ← brm(
y ~ 1 + x1 + x2 + (1 | g),
data = fake,
family = poisson(),
iter = 100, # short sampling to speedup example
prior = prior(normal(0,1), class = b) +
prior(constant(1), class = sd, group = g)
#,backend = “cmdstanr”,
#threads = threading(2)
)

When I compile this, I get the following error:

Compiling Stan program…
make cmd is
make -f ‘/usr/lib/R/etc/Makeconf’ -f ‘/usr/share/R/share/make/shlib.mk’ CXX=’(CXX14) (CXX14STD)’ CXXFLAGS=’(CXX14FLAGS)' CXXPICFLAGS='(CXX14PICFLAGS)’ SHLIB_LDFLAGS=’(SHLIB_CXX14LDFLAGS)' SHLIB_LD='(SHLIB_CXX14LD)’ SHLIB=‘file19a475629b5.so’ OBJECTS=‘file19a475629b5.o’

make would use
if test “zfile19a475629b5.o” != “z”; then
echo g++ -std=gnu++14 -shared -L"/usr/lib/R/lib" -Wl,-Bsymbolic-functions -Wl,-z,relro -o file19a475629b5.so file19a475629b5.o ‘/usr/local/lib/R/site-library/rstan/lib//libStanServices.a’ -L’/home/usmta123/R/x86_64-pc-linux-gnu-library/3.6/StanHeaders/lib/’ -lStanHeaders -L’/usr/local/lib/R/site-library/RcppParallel/lib/’ -ltbb -L"/usr/lib/R/lib" -lR;
g++ -std=gnu++14 -shared -L"/usr/lib/R/lib" -Wl,-Bsymbolic-functions -Wl,-z,relro -o file19a475629b5.so file19a475629b5.o ‘/usr/local/lib/R/site-library/rstan/lib//libStanServices.a’ -L’/home/usmta123/R/x86_64-pc-linux-gnu-library/3.6/StanHeaders/lib/’ -lStanHeaders -L’/usr/local/lib/R/site-library/RcppParallel/lib/’ -ltbb -L"/usr/lib/R/lib" -lR;
fi
Error in compileCode(f, code, language = language, verbose = verbose) :
^~~/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/products/GeneralMatrixVector.h:385:62: warning: ignoring attributes on template argument ‘Eigen::internal::packet_traits::type {aka __vector(2) double}’ [-Wignored-attributes]/usr/bin/ld: /usr/local/lib/R/site-library/rstan/lib//libStanServices.a(stan_fit.o): relocation R_X86_64_PC32 against undefined hidden symbol `_ZTCN5boost10wrapexceptISt14overflow_errorEE0_NS_16exception_detail10clone_implINS3_19error_info_injectorIS1_EEEE’ can not be used when making a shared object/usr/bin/ld: final link failed: Bad valuecollect2: error: ld returned 1 exit statusmake: *** [file19a475629b5.so] Error 1
Error in sink(type = “output”) : invalid connection

Can anyone help here ?

2

Can you try restarting R and then reinstalling RStan and StanHeaders from source:

# Compile packages using all cores
Sys.setenv(MAKEFLAGS = paste0("-j",parallel::detectCores()))

install.packages(c("StanHeaders","rstan"),type="source")
3

I am using this code on Linux. Can you help me understand where should I run the first line ?

4

Run all lines in R

5

Thanks a lot. This works

6

I am use this approach on my actual dataset which is approximately 20 million rows and 16 columns, I found that BRMS is very slow. Here is my approach below:

  1. I have standardized the data
  2. Priors for some of the coefficients are Normally distributed with mean =0 and SD = 1
  3. While for the other coefficients, as I am unable to get positive coefficients with normal distribution, I use beta distributed priors. This throws me some warnings saying I need to put lower and upper bound. With lb and ub specified, I need to enclose this in a wrapper function with nl=TRUE set something like below:

model ← brm_multiple(bf(y_variable ~ Int
A+ B+ C+ D
,nl=TRUE)+
lf(Int ~ 1)+
lf(A~ 0+d_A)+
lf (B~ 0+d_B)+
lf(C ~ 0+d_C)+
lf(D ~ 0+d_D),
data = df_split,
family = bernoulli(“logit”),
prior = my_prior,warmup = 1000,chains = 4,cores=24,seed=12345,iter=2000,silent = FALSE,
thin=1)

This is taking too long to execute so I just ran the code on 5% of data which took 7 hrs to complete. I am wondering if there is a way to speed up this. Also, as I increase beta priors with lb and ub defined, it increases it’s completion time.

7

With 20 million rows of data, with estimation repeated across multiple datasets, the model is always going to be fairly slow.

There are two things you can do to speed things up here. Both of these will require you to install the cmdstanr package from here, and running your brms models with the backend="cmdstanr".

Firstly, even though you have specified cores=24, brms will only use 4 cores (one per chain). To use multiple cores per chain, you need to use the threading option. There’s more information and examples in the vignette.

Next, you should run the models with the option normalize=FALSE, which will skip the computation of normalising constants in the model likelihood. However, this is only appropriate if you are not computing Bayes factors.

8

Thank you for this information. Can you help me understand if we can pass ‘normalize=F’ in BRM function ? It throws me error while I specify this.

9

Have you installed cmdstanr and are using the backend="cmdstanr" option?

10

Yes, I am using backend = “cmdstanr”, threads = threading(2)

11

After double-checking, it looks like you’ll need the github version of brms for the normalize option as well. Can you try installing that via:

remotes::install_github("paul-buerkner/brms")

and then running your model again?

12

Initially, I was unable to install cmdstanR with CRAN version of BRMS but when I had installed from github, it worked. So I believe, the BRMS package I am using is from github only.

13

Can you post your full brms command and the full error?

Also, what do you get from: packageVersion("brms")?

14

BRMS package version → 2.14.4

Code :

my_prior ← c(prior(normal(0,1),class=‘b’,nlpar=‘Int’)+
prior(beta(3,12),class=‘b’,nlpar=‘A’,lb=0,ub=1)+
prior(beta(0.5,6),class=‘b’,nlpar=‘B’,lb=0,ub=1)+
prior(beta(4,13),class=‘b’,nlpar=‘C’,lb=0,ub=1)+
prior(beta(8,14),class=‘b’,nlpar=‘D’,lb=0,ub=1))

start.time ← Sys.time()
model ← brm_multiple(bf(y_variable ~ Int
+A + B+ C+ D
,nl=TRUE)+
lf(Int ~ 1)+
lf(A ~ 0+d_A)+
lf(B ~ 0+d_B)+
#lf(C ~ 0+d_C)+
lf(D ~ 0+d_D),
data = df_split,
family = bernoulli(“logit”),
backend = “cmdstanr”,
threads = threading(2),normalize=FALSE,
prior = my_prior,warmup = 1000,chains = 4,cores=24,seed=12345,iter=2000,silent = FALSE,
thin=1)
total_time ← Sys.time()-start.time
total_time

15

Ah, there’s the issue

BRMS package version → 2.14.4

2.14.4 is the current CRAN version, if you install from GitHub you’ll have version 2.14.11. Can you try the github install?

16

Thanks Andrew. I re-installed BRMS from GitHub. Now, I do not get any error.

But, can you help me understand one thing. Once the compilation gets over, I see below statements

Compiling the C++ model
Running MCMC with 1 chain, with 2 thread(s) per chain…

Chain 1 num_threads = 2
Chain 1 WARNING: No variance estimation is
Chain 1 performed for num_warmup < 20
Chain 1 Iteration: 1 / 2 [ 50%] (Warmup)
Chain 1 Iteration: 2 / 2 [100%] (Sampling)
Chain 1 finished in 0.4 seconds.

Does this mean that the chain has really got finished in 0.4 seconds ? or does that mean something else ?

17

That output indicates that only 1 chain with 2 iterations were requested, can you double-check your input? 4 chains with 2000 iterations is the default, so you also don’t need to specify the chains and iter/warmup if that’s what you want

18

I have just specified 4 chains and 2000 iterations as shared with you. Not sure, how is it taking 1 chain and 2 iterations.

19

Is it different if you don’t specify the number of chains or iterations?

20

Actually, I’ve just reproduced this locally. Let me dig into this and see where the error is