Double free or corruption (out) depending on Linux version

Interfaces CmdStan
, ,
1

I am running a model on my university cluster. It’s my first time running my Stan code anywhere except my laptop. The computing center support team has been able to help me narrow the issue to a difference between the interactive interface using RStudio and terminal. In response to my initial question they stated:

It looks like this is a side effect of using the rocker containers to provide Rstudio. They are built on Ubuntu 20.04, which has a slightly newer glibc/libstdc++ version than CentOS 8 which is used for the cluster OS. The immediate workaround is to uninstall the rstan package and reinstall it using the terminal via the R/4.1 module. This will use the older glibc/libstdc++ from CentOS 8. Since those are forward compatible, it should work in the RStudio environment as well.

I reinstalled my packages through the terminal so they are compiled against CentOS 8. When I compile my model through RStudio, it works fine. However, it’s a fairly large dataset and the model takes a long time to run, so I need to run it through the terminal to make full use of the cluster (additional CPUs are available with this option). If I compile the model in the terminal (on CentOS 8), it gives me:

Chain 1 double free or corruption (out)
Warning: Chain 1 finished unexpectedly!

Any ideas I could try or bring back to the support team to get this working?

2

We’ve worked out a short-term solution to run the model on an Ubuntu virtual environment. It’s limited to 20 cores, which should be sufficient for me. I’d still like to get the initial problem sorted out though.

3

Are you setting any custom compile flags?

4

No. Is that something I should be doing?

5

No, I’ve encountered “double-free or corruption” errors on ubuntu with “-march=native” in my make/local and wondered if you were hitting that too

1 Like
6

Are you parallelizing across multiple compute nodes or are you just using multiple CPUs in a single machine? If the latter (and probably the former as well, just would require more effort), why not just write an R script and call it from the command line?

7

Are you using rstan or cmdstanr? If you aren’t already, I’d recommend trying your model with cmdstanr, since it tends to be more resistant to crashes/undefined behaviours than rstan

There’s more background on getting things up and running in this article: Getting started with CmdStanR • cmdstanr

8

Thanks for the feedback. I should perhaps provide some more details. I am using cmdstanr. The model runs on Ubuntu (i.e., it compiles and generates samples). If I install cmdstanr on Centos, it will compile the model but gives the error when I call model$sample(). If I compile the model on Centos and run it on the Ubuntu interactive environment so I can use RStudio, it gives the same error. I’m setting up a Ubuntu environment on the HPC, which they have the option for with some limitations. I was hoping to test it yesterday, but it ended up taking longer than expected and I should really be working on a proposal today…

@Richard_Border Yes, that’s exactly what I’m doing. I can either run an R script from the terminal (Linux Centos 8) or interactively in RStudio (Linux Ubuntu 20). Neither option works if I compile the model from the terminal (using Centos 8).

9

What compiler version is installed on the Centos instance (e.g., what’s the output from g++ --version?)

10

@andrjohns the compiler version is 10.3.0

11

Alright, so that rules out the usual suspects! Let’s ignore anything ubuntu/interactive for now, and just consider the centos instance.

The first step is to test whether it’s cmdstan in general, or if your model in particular has found a bug.

Can you run and post all output from the following code:

library(cmdstanr)

test_mod <- " data { real ymean; } parameters { real y; } model { y ~ normal(ymean, 1); }"
model <- cmdstan_model(write_stan_file(test_mod), quiet = FALSE)
sampled <- model$sample(data = list(ymean = 0), refresh = 0)
12

I guess the problem was that I was mixing too many different OS and Stan implementations. I setup a basic model using brms, then used cmdstanr whereas I used rstan on past projects.

The key points:

  1. Needed to ensure I compiled the model on Centos and not Ubuntu.
  2. Needed to ensure I ran install_cmdstan() and not just install.packages(“cmdstanr”). I ran it when I initially tested things on the Rstudio/Ubuntu environment, but I guess I missed something on the Centos install.

Thanks for the help, everyone!

1 Like