Makefile to make cmdstan use les ram during compilation

Please provide this additional information in addition to your question:

  • Operating System: Ubuntu 18.04.2 LTS
  • CmdStan Version: 2.14
  • Compiler/Toolkit: gcc

Hi,

I have a linux box with 32 cores that I use for running models (and has only 16GB of ram). I often try to run multiple models in parallel using cmdstan (usually up to 7 models with 4 chain each). It used to work fine until I updated Ubuntu and I guess it updated gcc with it. Now gcc is using huge amounts of ram during compilation time, overflowing my ram and leading to use SWAP, which is grinding everything to a halt.

Make version: GNU Make 4.1
GCC version: 7.4.0

I read that I could force gcc to use less ram by using the following flags:
CFLAGS="$CFLAGS --param ggc-min-expand=0 --param ggc-min-heapsize=524288"
such that it uses no more than 500MB per compile for each model (I don’t really mind if it slows down compilation, as long as it doesn’t overflow the ram. I’m still getting enough speedup by running multiple models in parallel vs. sequentially).

However I’m not sure as to where I should put these arguments ? Or if they would indeed help at all.
Would these flags need to be in the makefile present in the cmdstan directory?

I hope this is the right place to ask for this kind of help,
Thank you so much in advance,
Best,
F.

Make the file “make/local” in your cmdstan folder and add whatever you want in there.

For instance I have a make/local in a custom cmdstan where I’m using with a bunch of stuff in it like:

STANCFLAGS=--allow_undefined --name=rus
USER_HEADER=src/modal_cpp/stan_mech.hpp
CXXFLAGS=-fopenmp -I src/modal_cpp/spectra/include/Spectra
LDFLAGS=-fopenmp
LDLIBS=-llapack

Thanks so much @bbbales2 for your help with this.

Unfortunately I wasn’t able to get it working.
I created local in the make folder of cmdstan, and tried:

CFLAGS= --param ggc-min-expand=0 --param ggc-min-heapsize=524288

but wouldn’t compile. However, the following command allowed compilation to go through:

CXXFLAGS= --param ggc-min-expand=0 --param ggc-min-heapsize=524288

Unfortunately no matter what arguments I passed to ggc-min-heapsize, the compilation still used huge amounts of ram. So it seems like setting these flags didn’t seem to help.

As it currently stands compiling a single model takes approx 1400M of Virt. Mem and 1300M Res. Mem, when I run ‘htop’. I’m not sure whether this is supposed to be ‘a lot’ for a rather basic logistic model?

I think I’ll run the compilation sequentially for all models, and then run the sampling of all models in parallel. Given that sampling is the process that takes a lot more time, I’ll still be able to see substantial speedup.

It’s not that surprising, unfortunately. Stan uses lots of templated C++ and file that eventually gets compiled is massive.

You can swap out your compiler by specifying CXX=whatever I think, so if you can figure out your old gcc you can try installing a version and going back to it.

Thanks for your help @bbbales2, I’ll try to re-install an older version of gcc and see if this helps (If I can’t get it to work that way I’ll just pre-compile serially and run models in parallel after compilation).

1 Like