Ubuntu kills brms loo process

Interfaces brms
1

Background
I am using a virtual machine (e2-standard-16) on the Google Compute Engine to run some brms models. The models are 2PL item response theory models for a test with 30 items administered to 1050 participants. General syntax for the models looks like this (+ my fixed and random effects):

brm(bf(Resp ~  exp(logalpha) * eta,
                      eta ~ 1 + (1 |i| Item) + (1 | ID),
                      logalpha ~ 1 + (1 |i| Item),
                      nl = TRUE),
                   data = df, family = bernoulli("logit"),
                   prior = TwoPL_priors, seed = SEED, iter = 3000, warmup = 1000,
                   sample_prior = "yes")

, with the following priors:

TwoPL_priors <- 
  prior("normal(0, 1)", class = "b", nlpar = "eta") +
  prior("normal(0, 1)", class = "b", nlpar = "logalpha") +
  prior("constant(1)" , class = "sd", group = "ID", nlpar = "eta") +
  prior("normal(0, 2)", class = "sd", group = "Item", nlpar = "eta") +
  prior("normal(0, 1)", class = "sd", group = "Item", nlpar = "logalpha")

I’ve run many different models now all without any significant issues (e.g., no divergent transitions, good chain mixing). I obviously want to be able to compare various models, so I have been using the add_criterion(..., criterion = "loo") function. In addition to loo comparisons, I was also hoping to use model_weights(..., weights = "pseudobma"). Due to some missing data, I have 5 imputed datasets via the mice package that are being fit with the brm_multiple() function. When I attempt to add the loo criterion (or even the Bayesian R2 for that matter) for these models, my R session is terminated and I am returned to the console with the line “KILLED”. Trying them on my own PC, I get the classic Windows error of “unable to allocate a vector of size 9.4GB”.

Technical Specs
On the cloud, I am running a machine with Ubuntu 20.10 and R 4.0.2. Aside from R, the only thing I have installed on the machine is screen (and then the various dependencies for stan and future). Within R, the only packages I have installed are rstan, brms, and future (plus their dependencies). I assume the issue is the machine running out of memory (16 virtual cores, 64GB memory), though I thought that running Ubuntu in R (and not RStudio) would avoid memory issues.

Questions

  1. Is there something that I’m missing other than running out of memory that would cause R to be terminated like this? I’m not too familiar with the future package or running plan(multicore), so I’m not sure whether maybe there’s a memory leak in there. Similarly, I’m used to doing everything in Windows, so all problems in Linux are new problems to me.

  2. If this is just an issue with running out of memory, is it worth it to create a more memory intensive machine, or are there maybe other ways of doing the model comparisons? I’ve used loo_compare() and model_weights() to compare simpler versions of the models, but I’m open to doing something else – I just want a consistent method used for all of the models.

1 Like
2

As you mentioned, this is most likely caused by running out of memory. A couple of things to investigate here.

After running your brm model on your local Windows machine, how much RAM is used up by the R/RStudio process and how much RAM does your computer have in total?

Some things to try are to run add_criterion(..., criterion = "loo",pointwise=TRUE), as it uses less memory. You can also try making sure that the loo calculations are only using a single core, as using multi-core processing can require more memory for loo. You can do this by specifying options(mc.cores=1) prior to the add_criterion call. I’m not sure how the future package interacts with add_criterion, so if you’ve also setup a multicore session then also run plan(sequential) prior to the call

1 Like
3

I’m giving add_criterion(..., criterion = "loo", pointwise = TRUE) a shot right now. Looks like, per Task Manager, RStudio is taking up about 2500 MB at the moment. On my PC, I’ve got 32 GB of RAM (31.7 useable).

In the past, I’ve had to extend the memory available to R with memory.limit(size = ...) because Windows sets a hard limit by default, and I find that usually RStudio still kills the process well before my actual RAM is used up (even if I set the memory limit to something like 25 GB). My hope in setting up a Linux virtual machine running R and not RStudio was to get around some of these memory usage limits, but maybe that’s not the case.

4

The addition of pointwise = TRUE to the add_criterion() function does let me get a loo estimate for the large models. It takes a good length of time, but it works.

Any recommendations/workarounds for getting model_weights() to work with weights = "pseudobma"? I can get weights with weights = "loo", but my preference is with pseudo-Bayesian Model Averaging. I’ll give it another shot on a high memory virtual machine without calling future to see if that fixes the Ubuntu console kill, but just from a practical sense, wondering whether there’s any memory saving tricks like pointwise = TRUE but for model_weights().

5

I’m not familiar with pseudo-bma weights, so I can’t speak to that unfortunately.

However if the pointwise loo is working, then it is likely a memory issue. I’d try using the regular loo & pseudobma calls, but ensuring no parallelism (i.e.,options(mc.cores=1), plan(sequential)) to see how it compares

6

I actually found that on a cluster its easier to max out the cores to get the job done faster. And since the clusters usually have a fixed amount of memory per slot, this also maxes the RAM available (for me 24*15G=815G). Thanks for the speedy command though @andrjohns .

options(mc.cores = 24)
brms::add_criterion(brmsfitobject, criterion = "loo", pointwise=TRUE)