Cannot resolve the "cannot allocate vector of size … " error

Interfaces RStan
41

Hi I have a similar problem but with my developing package, now I can not install it from source, I got this error re oxygenizing

roxygen2::roxygenize(clean = TRUE)
Loading varstan
Error in Rcpp::loadModule(module = “stan_fit4Bekk_mod”, what = TRUE, env = ns, :
Unable to load module “stan_fit4Bekk_mod”: cannot allocate a vector of size 15599.7 Gb

I updated my R and Rtools and my make vars file have:

CXX14FLAGS=-O3 -march=native -mtune=native
CXX11FLAGS=-O3 -march=corei7 -mtune=corei7
CXX14FLAGS += -mtune=native -march=native -Wno-ignored-attributes -Wno-deprecated-declarations
Can somebody help me please?:)

42

It seems like the best thing to do is put the full path the new C++ compiler in ~/.R/Makevars like

CXX14 = C:/rtools40/mingw64/bin/g++

or wherever it is installed.

43

Thanks @bgoodri it works :)

44

Here is my model that continues to run on R3.6 but not R4.0 (Windows 10):

J <- 21
n <- c(5,98,238,266,25,10,5,5,5,5,5,18,5,5,5,33,23,383,61,15,56)
y <- c(0,0,165,246,15,0,0,0,0,5,0,18,5,0,5,0,23,345,61,15,56)

ratio <- y/n
plot(y/n,ratio)
plot(n,ratio)

library(rstan)
rstan_options(auto_write = TRUE)
Sys.setenv(LOCAL_CPPFLAGS = '-march=native')
options(mc.cores = parallel::detectCores())
datafitX <- stan(file="model.stan",data=c("J","y","n"), iter = 10000, chains = 4,control=list(adapt_delta=0.8, max_treedepth =10))

model.stan as follows:

data {
  int<lower=0> J; 
  int<lower=0> y[J]; 
  int<lower=0> n[J]; 
}
parameters {
  real<lower=0,upper=1> theta[J]; 
  real<lower=0,upper=1> lambda; 
  real<lower=0.1> kappa; 
}
transformed parameters {
  real<lower=0> alpha;
  real<lower=0> beta; 
  alpha = lambda* kappa;  
  beta =   (1 - lambda)* kappa; 
}
model {
  lambda ~ uniform(0,1); 
  kappa ~ uniform(0.1,5); 
  theta ~ beta(alpha,beta); 
  y ~ binomial(n,theta); 
  
}
generated quantities {
  real<lower=0,upper=1> avg; 
  int<lower=0,upper=1> above_avg[J]; 
  int<lower=1,upper=J> rnk[J]; 
  int<lower=0,upper=1> highest[J]; 
  avg = mean(theta);
    for (j in 1:J)
    above_avg[j] = (theta[j] > avg);
    for (j in 1:J) {
    rnk[j] = rank(theta,j) + 1;
    highest[j] = rnk[j] == 1;
  }
}
45

Can somebody please help me. I am continuing to get the error “cannot callocate vector of size …” when running the above model using R4 and Rtools40 on Windows 10.

46

This model runs for me on Windows with RTools40. When I compile it with verbose = TRUE, it compiles like

Compilation argument:
C:/PROGRA~1/R/R-40~1.0/bin/x64/R CMD SHLIB file40e424ba7b98.cpp 2> file40e424ba7b98.cpp.err.txt
C:/RBuildTools/4.0/mingw64/bin/g++ -m64 -std=gnu++14 -I"C:/PROGRA~1/R/R-40~1.0/include" -DNDEBUG -I"C:/Users/Stan/Documents/R/win-library/4.0/Rcpp/include/" -I"C:/Users/Stan/Documents/R/win-library/4.0/RcppEigen/include/" -I"C:/Users/Stan/Documents/R/win-library/4.0/RcppEigen/include/unsupported" -I"C:/Users/Stan/Documents/R/win-library/4.0/BH/include" -I"C:/Users/Stan/Documents/R/win-library/4.0/StanHeaders/include/src/" -I"C:/Users/Stan/Documents/R/win-library/4.0/StanHeaders/include/" -I"C:/Users/Stan/Documents/R/win-library/4.0/rstan/include" -DEIGEN_NO_DEBUG -D_REENTRANT -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp -std=c++1y -march=native -O3 -march=native -mtune=native -Wno-ignored-attributes -Wno-deprecated-declarations -c file40e424ba7b98.cpp -o file40e424ba7b98.o

How is your Compilation argument: line different that that?

47

But do you have both version of R and Rtools installed at the same time? Maybe is that!

48

I took down previous versions of Rtools and only have Rtools40 installed.
How would I get the compilation argument?
Console shows this:

TRANSLATING MODEL ‘flights_model’ FROM Stan CODE TO C++ CODE NOW.
successful in parsing the Stan model ‘flights_model’.

CHECKING DATA AND PREPROCESSING FOR MODEL ‘flights_model’ NOW.

COMPILING MODEL ‘flights_model’ NOW.
Error: cannot allocate vector of size 15482.7 Gb

49

You have to call stan or stan_model with the verbose = TRUE argument.

50

Ok I did and above is what I got.

51

You have the same Compilation argument as I do?

52

But I am still getting the error. We have not solved the issue. Any recommendations?

53

What is your Complilation argument line?

54

Where is the compilation argument line? Console? Terminal?

55

Not sure if it helps. I unistalled RStudio, all versions of R, and rtools, then installed everything and followed your Github instructions Home · stan-dev/rstan Wiki · GitHub.

  1. pkgbuild::has_build_tools(debug = TRUE) not possible to install rtools via Rstudio. Was only possible manually.
  2. created pers. Makevars, no problem
  3. removed rstan packages, no problem
  4. set cores, no problem
  5. neither installation from source worked. here are error msgs:

Sys.setenv(MAKEFLAGS = “-j4”) # four cores used
install.packages(“rstan”, type = “source”)
Installing package into ‘C:/Users/tschwoerer/Documents/R/win-library/4.0’
(as ‘lib’ is unspecified)
trying URL ‘https://cran.rstudio.com/src/contrib/rstan_2.19.3.tar.gz’
Content type ‘application/x-gzip’ length 922337 bytes (900 KB)
downloaded 900 KB

  • installing source package ‘rstan’ …
    ** package ‘rstan’ successfully unpacked and MD5 sums checked
    ** using staged installation
    ** libs

*** arch - i386

C:/Rtools/mingw_32/bin/g++ -m32 -std=c++1y -std=gnu++14 -I"C:/PROGRA~1/R/R-40~1.0/include" -DNDEBUG -I"…/inst/include" -I"." -I"C:/Users/tschwoerer/Documents/R/win-library/4.0/StanHeaders/include/src" -DBOOST_DISABLE_ASSERTS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION -I’C:/Users/tschwoerer/Documents/R/win-library/4.0/Rcpp/include’ -I’C:/Users/tschwoerer/Documents/R/win-library/4.0/RcppEigen/include’ -I’C:/Users/tschwoerer/Documents/R/win-library/4.0/BH/include’ -I’C:/Users/tschwoerer/Documents/R/win-library/4.0/StanHeaders/include’ -O3 -march=corei7 -mtune=corei7 -c sparse_extractors.cpp -o sparse_extractors.o
sh: C:/Rtools/mingw_32/bin/g++: No such file or directory

make: *** [C:/PROGRA~1/R/R-40~1.0/etc/i386/Makeconf:229: sparse_extractors.o] Error 127
ERROR: compilation failed for package ‘rstan’

  • removing ‘C:/Users/tschwoerer/Documents/R/win-library/4.0/rstan’
    Warning in install.packages :
    installation of package ‘rstan’ had non-zero exit status

The downloaded source packages are in
‘C:\Users\tschwoerer\AppData\Local\Temp\RtmpakBgyJ\downloaded_packages’

remotes::install_github(“stan-dev/rstan”, ref = “develop”, subdir = “rstan/rstan”, build_opts = “”)
Downloading GitHub repo stan-dev/rstan@develop
stan-dev-rstan-5869a5e/StanHeaders/inst/include/libsundials: Can’t create ‘\\?\C:\Users\TSCHWO~1\AppData\Local\Temp\RtmpakBgyJ\remotes40886e1c1fe4\stan-dev-rstan-5869a5e\StanHeaders\inst\include\libsundials’
stan-dev-rstan-5869a5e/StanHeaders/inst/include/src: Can’t create ‘\\?\C:\Users\TSCHWO~1\AppData\Local\Temp\RtmpakBgyJ\remotes40886e1c1fe4\stan-dev-rstan-5869a5e\StanHeaders\inst\include\src’
stan-dev-rstan-5869a5e/StanHeaders/inst/include/stan: Can’t create ‘\\?\C:\Users\TSCHWO~1\AppData\Local\Temp\RtmpakBgyJ\remotes40886e1c1fe4\stan-dev-rstan-5869a5e\StanHeaders\inst\include\stan’
tar.exe: Error exit delayed from previous errors.
“C:\PROGRA~1\Git\cmd\git.exe” clone --depth 1 --no-hardlinks --recurse-submodules --branch develop GitHub - stan-dev/stan: Stan development repository. The master branch contains the current release. The develop branch contains the latest stable development. See the Developer Process Wiki for details. C:\Users\TSCHWO~1\AppData\Local\Temp\RtmpakBgyJ\remotes40886e1c1fe4/stan-dev-rstan-5869a5e/rstan/rstan/…/…/StanHeaders/inst/include/upstream
Error: Failed to install ‘rstan’ from GitHub:
Command failed (128)
In addition: Warning messages:
1: In utils::untar(tarfile, …) :
‘tar.exe -xf “C:\Users\TSCHWO~1\AppData\Local\Temp\RtmpakBgyJ\file40883eb67ae.tar.gz” -C “C:/Users/TSCHWO~1/AppData/Local/Temp/RtmpakBgyJ/remotes40886e1c1fe4”’ returned error code 1
2: In system(full, intern = TRUE, ignore.stderr = quiet) :
running command ‘“C:\PROGRA~1\Git\cmd\git.exe” clone --depth 1 --no-hardlinks --recurse-submodules --branch develop GitHub - stan-dev/stan: Stan development repository. The master branch contains the current release. The develop branch contains the latest stable development. See the Developer Process Wiki for details. C:\Users\TSCHWO~1\AppData\Local\Temp\RtmpakBgyJ\remotes40886e1c1fe4/stan-dev-rstan-5869a5e/rstan/rstan/…/…/StanHeaders/inst/include/upstream’ had status 128

56

Following https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started:
6. install.packages(“rstan”, repos = “https://cloud.r-project.org/”, dependencies = TRUE), installed rstan no problem
7.pkgbuild::has_build_tools(debug = TRUE), error message in RStudio: “Building R package from source requires installation of additional build tools. Do you want to install the additionalk tools now?”
Confused as Rtools 4.0 is clearly installed. I checked the Windows apps.
7a: click yes: FALSE
7b: click no: FALSE

57

In what directory is Rtools installed?

58

Honestly, I think this might be due to some unnecessary Makevars.win configuration. The wiki is outdated.

Uninstall Rtools (all versions). Install Rtools 40. Rename your Makevars.win to something else, just to remove it from the picture for now. And try again. You may also need to rename .Renviron to something else as well.

First, a default configuration for Rtools 40 should not be C:/Rtools at all. It should be C:/rtools40.

Second, ensure after running Rtools40, that you open R and run: writeLines('PATH="${RTOOLS40_HOME}\\usr\\bin;${PATH}"', con = "~/.Renviron")

Third, if that still does not work, my guess is you have some lingering PATH problems, and a lingering Rtools directory.

You do not need a Makevars.win file with R 4.0 and Rtools40 to run Stan; it’s mainly useful for adding compiler optimizations, which should be done after ensuring that it works. All you should need, is R 4.0, Rtools40 (with defaults selected), and to run the writeLines('PATH="${RTOOLS40_HOME}\\usr\\bin;${PATH}"', con = "~/.Renviron") command in R. Then restart R, and it should compile the model.

59

Hi,
had the same problem as you since upgarding to R 4.0 and followed the wiki to install Rstan, and amend the Makevars.win files.
Still, I got error messages “cannot allocate…”
Turned out my code was also using the recommendation to avoid recompilation of unchanged Stan programs by calling
rstan::rstan_options(auto_write = TRUE)

However my stanmodel.rds were outdated. Once these were removed and models were recompiled, everything was fine.

Could that be part of your problem?

Cheers

1 Like
60

I uninstalled everything completely. Then re-installed just R4.0 and rtools40.
Followed your instructions renaming Makevars.win and .Renviron to something else. Same error: “cannot allocate vector of size …” Then did your recommended steps:

  1. rtools installed correctly in C:/rtools40

  2. Console output:
    TRANSLATING MODEL ‘flights_model’ FROM Stan CODE TO C++ CODE NOW.
    successful in parsing the Stan model ‘flights_model’.
    CHECKING DATA AND PREPROCESSING FOR MODEL ‘flights_model’ NOW.
    COMPILING MODEL ‘flights_model’ NOW.
    Error in get(“storage”, envir = as.environment(x)) :
    object ‘storage’ not found

  3. If I do have lingering path problems, how can I eliminate those?

  4. Followed Mauthier’s suggestions
    4.1. took out rstan_options(autho_write = TRUE), still “cannot allocate vector…”
    4.2. removed stanmodel.rds
    RStudio error message as before during install: "Building R package from source requires installation of additional build tools. Do you want to install the additional tools now?”
    RStudio asked me this twice, I clicked yes, twice:
    Compilation argument:

C:/PROGRA~1/R/R-40~1.0/bin/x64/R CMD SHLIB file23ac3b7144af.cpp 2> file23ac3b7144af.cpp.err.txt 
Warning message:
In system(cmd) : 'make' not found
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
  cannot open file 'file23ac3b7144af.cpp.err.txt': No such file or directory```