Rstan- Error: vector memory exhausted (limit reached?)

#1

Hi,

When I run the “8schools” example found at https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started I get the following error: “Error: vector memory exhausted (limit reached?).”

I have reinstalled R and added “R_MAX_VSIZE=100Gb” (I have used R_MAX_VSIZE ranging from 10 to 100Gb) to the .Renviron, yet I still get the above error.

Thank you!

  • Operating System
    macOS Mojave 10.14.3

  • RStan Version
    2.18.2

  • Output of writeLines(readLines(file.path(Sys.getenv("HOME"), ".R/Makevars")))
    CC=/usr/local/Cellar/gcc/8.2.0/bin/gcc-8
    CXX=/usr/local/Cellar/gcc/8.2.0/bin/g+±8
    CXX11=/usr/local/Cellar/gcc/8.2.0/bin/g+±8
    CXX14=/usr/local/Cellar/gcc/8.2.0/bin/g+±8
    cxx17=/usr/local/cellar/gcc/8.2.0/bin/g+±8
    cxx1X=/usr/local/cellar/gcc/8.2.0/bin/g+±8
    LDFLAGS=-L/usr/local/Cellar/gcc/8.2.0/lib
    CXX14FLAGS=-O3 -march=native -mtune=native
    CXX14FLAGS += -arch x86_64 -ftemplate-depth-256

  • Output of devtools::session_info("rstan")
    setting value
    version R version 3.5.2 (2018-12-20)
    os macOS Mojave 10.14.3
    system x86_64, darwin15.6.0
    ui RStudio
    language (EN)
    collate en_US.UTF-8
    ctype en_US.UTF-8

0 Likes

#2

Do you run out of RAM as it is compiling or as it is sampling?

0 Likes

#3

I do not believe that it is running out of RAM during the compiling of the package.


install.packages(“rstan”, repos = “https://cloud.r-project.org/”, dependencies = TRUE)
trying URL ‘https://cloud.r-project.org/bin/macosx/el-capitan/contrib/3.5/rstan_2.18.2.tgz
Content type ‘application/x-gzip’ length 19107931 bytes (18.2 MB)
==================================================
downloaded 18.2 MB
The downloaded binary packages are in
/var/folders/5n/s7_r8kvs68l1ktbm3vs1nf7c0000gn/T//RtmpI5iYRD/downloaded_packages

pkgbuild::has_build_tools(debug = TRUE)
[1] TRUE


I would guess it is during the sampling process-
This is the line that produces the error:
fit <- stan(file = ‘~/Desktop/8schools.stan’, data = schools_dat)

0 Likes

#4

The stan call combines compilation and sampling. What happens if you do

sm <- stan_model("~/Desktop/8schools.stan")

to only compile it?

0 Likes

#5

sm <- stan_model("~/Desktop/8schools.stan") does not produce any errors and creates a “Formal class stanmodel.”

0 Likes

#6

And then

post <- sampling(sm, data = schools_dat)

runs out of RAM?

0 Likes

#7

Yes, that line produces “Error: vector memory exhausted (limit reached?).”

0 Likes

#8

OK, the 8 schools example is tiny, so my guess is that the compiler build something with a memory leak somehow. Can you try building it with clang++ rather than g++-8 in your ~/.R/Makevars file?

0 Likes

#10

Thank you for your help!

Unfortunately, no change with using a clang compiler. I have completed a fresh install of Mojave, command line tools, R, and RStudio.

Rstan from Source:

> install.packages(“rstan”, type = “source”)

trying URL ‘https://cran.rstudio.com/src/contrib/rstan_2.18.2.tar.gz

Content type ‘application/x-gzip’ length 838052 bytes (818 KB)

==================================================

downloaded 818 KB

  • installing source package ‘rstan’ …

** package ‘rstan’ successfully unpacked and MD5 sums checked

** libs

17 warnings generated.

clang++ -std=gnu++14 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o rstan.so chains.o init.o lang__ast_def.o lang__grammars__bare_type_grammar_inst.o lang__grammars__expression07_grammar_inst.o lang__grammars__expression_grammar_inst.o lang__grammars__functions_grammar_inst.o lang__grammars__indexes_grammar_inst.o lang__grammars__program_grammar_inst.o lang__grammars__semantic_actions.o lang__grammars__statement_2_grammar_inst.o lang__grammars__statement_grammar_inst.o lang__grammars__term_grammar_inst.o lang__grammars__var_deccls_grammar_inst.o lang__grammars__whitespace_grammar_inst.o misc.o pointer-tools.o sparse_extractors.o stanc.o -F/Library/Frameworks/R.framework/… -framework R -Wl,-framework -Wl,CoreFoundation

installing to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/rstan/libs

** R

** inst

** byte-compile and prepare package for lazy loading

** help

*** installing help indices

*** copying figures

** building package indices

** installing vignettes

** testing if installed package can be loaded

  • DONE (rstan)

The downloaded source packages are in

‘/private/var/folders/w8/r8s5wmy50wgcpm81sbx8v_gc0000gn/T/RtmpK5C6oz/downloaded_packages’

> knitr::opts_chunk$set(echo = TRUE)

> library(“rstan”)

Loading required package: ggplot2

Loading required package: StanHeaders

rstan (Version 2.18.2, GitRev: 2e1f913d3ca3)

For execution on a local, multicore CPU with excess RAM we recommend calling

options(mc.cores = parallel::detectCores()).

To avoid recompilation of unchanged Stan programs, we recommend calling

rstan_options(auto_write = TRUE)

However, I am now receiving a different error when sampling the 8 school data:

> sm <- stan_model("~/Desktop/8schools.stan")

> post <- sampling(sm, data = schools_dat)

Error in dyn.load(libLFile) :

unable to load shared object ‘/var/folders/w8/r8s5wmy50wgcpm81sbx8v_gc0000gn/T//RtmpK5C6oz/file6ab1127e83b.so’:

dlopen(/var/folders/w8/r8s5wmy50wgcpm81sbx8v_gc0000gn/T//RtmpK5C6oz/file6ab1127e83b.so, 6): Symbol not found: __ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev

Referenced from: /var/folders/w8/r8s5wmy50wgcpm81sbx8v_gc0000gn/T//RtmpK5C6oz/file6ab1127e83b.so

Expected in: /usr/lib/libstdc++.6.dylib

in /var/folders/w8/r8s5wmy50wgcpm81sbx8v_gc0000gn/T//RtmpK5C6oz/file6ab1127e83b.so

Please let me know if this should be a separate post.

0 Likes

#11

Do you still have LDFLAGS=-L/usr/local/Cellar/gcc/8.2.0/lib in ~/.R/Makevars? It is running into a linker error now.

0 Likes