Rstan- Error: vector memory exhausted (limit reached?)

Interfaces RStan
1

Hi,

When I run the “8schools” example found at https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started I get the following error: “Error: vector memory exhausted (limit reached?).”

I have reinstalled R and added “R_MAX_VSIZE=100Gb” (I have used R_MAX_VSIZE ranging from 10 to 100Gb) to the .Renviron, yet I still get the above error.

Thank you!

  • Operating System
    macOS Mojave 10.14.3

  • RStan Version
    2.18.2

  • Output of writeLines(readLines(file.path(Sys.getenv("HOME"), ".R/Makevars")))
    CC=/usr/local/Cellar/gcc/8.2.0/bin/gcc-8
    CXX=/usr/local/Cellar/gcc/8.2.0/bin/g+±8
    CXX11=/usr/local/Cellar/gcc/8.2.0/bin/g+±8
    CXX14=/usr/local/Cellar/gcc/8.2.0/bin/g+±8
    cxx17=/usr/local/cellar/gcc/8.2.0/bin/g+±8
    cxx1X=/usr/local/cellar/gcc/8.2.0/bin/g+±8
    LDFLAGS=-L/usr/local/Cellar/gcc/8.2.0/lib
    CXX14FLAGS=-O3 -march=native -mtune=native
    CXX14FLAGS += -arch x86_64 -ftemplate-depth-256

  • Output of devtools::session_info("rstan")
    setting value
    version R version 3.5.2 (2018-12-20)
    os macOS Mojave 10.14.3
    system x86_64, darwin15.6.0
    ui RStudio
    language (EN)
    collate en_US.UTF-8
    ctype en_US.UTF-8

2

Do you run out of RAM as it is compiling or as it is sampling?

3

I do not believe that it is running out of RAM during the compiling of the package.

install.packages(“rstan”, repos = “https://cloud.r-project.org/”, dependencies = TRUE)
trying URL ‘https://cloud.r-project.org/bin/macosx/el-capitan/contrib/3.5/rstan_2.18.2.tgz’
Content type ‘application/x-gzip’ length 19107931 bytes (18.2 MB)
==================================================
downloaded 18.2 MB
The downloaded binary packages are in
/var/folders/5n/s7_r8kvs68l1ktbm3vs1nf7c0000gn/T//RtmpI5iYRD/downloaded_packages

pkgbuild::has_build_tools(debug = TRUE)
[1] TRUE

I would guess it is during the sampling process-
This is the line that produces the error:
fit ← stan(file = ‘~/Desktop/8schools.stan’, data = schools_dat)

4

The stan call combines compilation and sampling. What happens if you do

sm <- stan_model("~/Desktop/8schools.stan")

to only compile it?

5

sm <- stan_model("~/Desktop/8schools.stan") does not produce any errors and creates a “Formal class stanmodel.”

6

And then

post <- sampling(sm, data = schools_dat)

runs out of RAM?

7

Yes, that line produces “Error: vector memory exhausted (limit reached?).”

8

OK, the 8 schools example is tiny, so my guess is that the compiler build something with a memory leak somehow. Can you try building it with clang++ rather than g++-8 in your ~/.R/Makevars file?

10

Thank you for your help!

Unfortunately, no change with using a clang compiler. I have completed a fresh install of Mojave, command line tools, R, and RStudio.

Rstan from Source:

> install.packages(“rstan”, type = “source”)

trying URL ‘https://cran.rstudio.com/src/contrib/rstan_2.18.2.tar.gz’

Content type ‘application/x-gzip’ length 838052 bytes (818 KB)

==================================================

downloaded 818 KB

  • installing source package ‘rstan’ …

** package ‘rstan’ successfully unpacked and MD5 sums checked

** libs

…

17 warnings generated.

clang++ -std=gnu++14 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o rstan.so chains.o init.o lang__ast_def.o lang__grammars__bare_type_grammar_inst.o lang__grammars__expression07_grammar_inst.o lang__grammars__expression_grammar_inst.o lang__grammars__functions_grammar_inst.o lang__grammars__indexes_grammar_inst.o lang__grammars__program_grammar_inst.o lang__grammars__semantic_actions.o lang__grammars__statement_2_grammar_inst.o lang__grammars__statement_grammar_inst.o lang__grammars__term_grammar_inst.o lang__grammars__var_deccls_grammar_inst.o lang__grammars__whitespace_grammar_inst.o misc.o pointer-tools.o sparse_extractors.o stanc.o -F/Library/Frameworks/R.framework/… -framework R -Wl,-framework -Wl,CoreFoundation

installing to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/rstan/libs

** R

** inst

** byte-compile and prepare package for lazy loading

** help

*** installing help indices

*** copying figures

** building package indices

** installing vignettes

** testing if installed package can be loaded

  • DONE (rstan)

The downloaded source packages are in

‘/private/var/folders/w8/r8s5wmy50wgcpm81sbx8v_gc0000gn/T/RtmpK5C6oz/downloaded_packages’

> knitr::opts_chunk$set(echo = TRUE)

> library(“rstan”)

Loading required package: ggplot2

Loading required package: StanHeaders

rstan (Version 2.18.2, GitRev: 2e1f913d3ca3)

For execution on a local, multicore CPU with excess RAM we recommend calling

options(mc.cores = parallel::detectCores()).

To avoid recompilation of unchanged Stan programs, we recommend calling

rstan_options(auto_write = TRUE)

However, I am now receiving a different error when sampling the 8 school data:

> sm <- stan_model("~/Desktop/8schools.stan")

> post <- sampling(sm, data = schools_dat)

Error in dyn.load(libLFile) :

unable to load shared object ‘/var/folders/w8/r8s5wmy50wgcpm81sbx8v_gc0000gn/T//RtmpK5C6oz/file6ab1127e83b.so’:

dlopen(/var/folders/w8/r8s5wmy50wgcpm81sbx8v_gc0000gn/T//RtmpK5C6oz/file6ab1127e83b.so, 6): Symbol not found: __ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev

Referenced from: /var/folders/w8/r8s5wmy50wgcpm81sbx8v_gc0000gn/T//RtmpK5C6oz/file6ab1127e83b.so

Expected in: /usr/lib/libstdc++.6.dylib

in /var/folders/w8/r8s5wmy50wgcpm81sbx8v_gc0000gn/T//RtmpK5C6oz/file6ab1127e83b.so

Please let me know if this should be a separate post.

11

Do you still have LDFLAGS=-L/usr/local/Cellar/gcc/8.2.0/lib in ~/.R/Makevars? It is running into a linker error now.

12

Hi, I’m resurrecting this thread because I ran into the same error.

I have tried all the suggested steps above, still run into the same error:

> fit <- stan(file = '8schools.stan', data = schools_dat)
Error: vector memory exhausted (limit reached?)

My session info:

> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rstan_2.19.2          ggplot2_3.2.1         StanHeaders_2.18.1-10

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.2         rstudioapi_0.10    magrittr_1.5       tidyselect_0.2.5  
 [5] munsell_0.5.0      colorspace_1.4-1   R6_2.4.0           rlang_0.4.0       
 [9] dplyr_0.8.3        tools_3.6.1        parallel_3.6.1     pkgbuild_1.0.4    
[13] grid_3.6.1         gtable_0.3.0       loo_2.1.0          cli_1.1.0         
[17] withr_2.1.2        matrixStats_0.54.0 lazyeval_0.2.2     assertthat_0.2.1  
[21] tibble_2.1.3       crayon_1.3.4       processx_3.4.1     gridExtra_2.3     
[25] purrr_0.3.2        callr_3.3.1        ps_1.3.0           inline_0.3.15     
[29] glue_1.3.1.9000    compiler_3.6.1     pillar_1.4.2       prettyunits_1.0.2 
[33] scales_1.0.0       stats4_3.6.1       pkgconfig_2.0.2

My ~/.R/Makevars file

CC=/usr/local/Cellar/gcc/8.2.0/bin/gcc-8
CXX=/usr/local/Cellar/gcc/8.2.0/bin/clang++
CXX11=/usr/local/Cellar/gcc/8.2.0/bin/clang++
CXX14=/usr/local/Cellar/gcc/8.2.0/bin/clang++
cxx18=/usr/local/cellar/gcc/8.2.0/bin/clang++
cxx1X=/usr/local/cellar/gcc/8.2.0/bin/clang++
CXX14FLAGS=-O3 -march=native -mtune=native
CXX14FLAGS += -arch x86_64 -ftemplate-depth-256

(after changing g++-8 to clang++)

1 Like
13

Same issue. Did you resolve it?

14

@Lorenzo_Braschi + @bgoodri – Any chance you guys figured this out? I’m stumped…

15

As luck would have it, I encountered this issue a week ago, but no, I did not figure out what was causing it. Maybe take -march=native -mtune=native out of CXX14FLAGS in ~/.R/Makevars .

17

It didn’t help :/

Any ideas re: what to do? I’m unable to run Stan at all on this computer :(

18

The next thing to try is to make sure that Rcpp, RcppEigen, and rstan are all installed wit the same settings by compiling them from source with

install.packages(c("Rcpp", "RcppEigen", "rstan"), type = "source")
19

Okay, I think I fixed it somehow. Here is what I think did it:
– I tried running the line you suggested to install rccp/eigen w/ rstan from source. I hit an error about the fortran libraries.
I found a credible seeming discussion of this error here but didn’t want to follow the proposed solution for fear that other things installed on my computer are relying on the gcc paths.

– Instead, I followed along with roughly this. After that, removing rstan, restarting and trying to install everything from source again, it seemed to have fixed the problem.

For reference, this is my current Makevars:

VER=-8
CC=gcc$(VER)
CXX=g++$(VER)
FLIBS=-L/usr/local/Cellar/gcc/8.3.0_2/lib/gcc/8

CXX14FLAGS=-O3
CXX14FLAGS += -arch x86_64 -ftemplate-depth-256

CXX14FLAGS=-O3 -march=native -mtune=native
CXX14FLAGS += -arch x86_64 -ftemplate-depth-256
1 Like
20

I spoke too soon… tried sampling from a simple model with generated quantities and got: Error in Module(module, mustStart = TRUE) : vector is too large :(

21

Ok, I then tried following this, to change makevars to:
CC=gcc CXX=g++ CXX11=g++ CXX14=g++ cxx18=g++ cxx1X=g++ LDFLAGS=-L/usr/lib
and it seems to have fixed it… [fingers crossed]