Error: cannot allocate vector of size 703998.9 Gb

gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11)

That compiler isn’t one that we have much experience with, but it is supposed to work. My guess is that Rcpp was installed with a different compiler or settings, so I would try reinstalling Rcpp with what you have in Makevars.

1 Like

That’s a good suggestion. I’ll try it and report back.

By golly, reinstalling Rcpp worked! Thanks, Ben!

2 Likes

I encounter the same problem and I already removed and reinstalled the Rcpp package. However I did not specify anything according to what is written in the Makevar file.
I got the following options on what I read before to make it work with Rtools 4:
CXX14=$(BINPREF)g++ -O2 -march=native -mtune=native CXX14FLAGS=-O3 -march=native -mtune=native CXX11FLAGS=-O3 -march=native -mtune=native

How should I go forward to properly “reinstalling Rcpp with what I [you] have in Makevars”?
I also had problems before with Rtools 4.0 not being connected correctly and not being able to compile, but I fixed it.
pkgbuild::check_build_tools() Your system is ready to build packages!

There are also two more packages that I see in the win-library folder called RcppEigen and RcppParallel. Does it have something to do with that?
Or do I need to remove the Rcpp package by hand and not with the remove.packages("Rcpp") command?

Have you tried the instructions over in this thread: R 4.0, rstan, and you? They should help

1 Like

Yes I did. Already followed them twice (install-uninstall-reinstall) and it didn’t solve the problem.

Does the example model compile for you?

example(stan_model,run.dontrun = TRUE,verbose=TRUE)

If not, can you post the outputs from:

Sys.which("make")
Sys.getenv("BINPREF")
Sys.getenv("PATH")
readLines("~/.R/Makevars.win")

Just to check for any common sources of configuration issues

1 Like

I already tried the example yesterday because I read it here (Error: cannot allocate vector of size 17348.7 Gb - #10 by bgoodri) but RStudio always freezes and doesn’t react anymore after around 2 minutes.

Those are the outputs:
Sys.which("make")
make
"C:\\rtools40\\usr\\bin\\make.exe"

Sys.getenv("BINPREF")
""
Sys.getenv("PATH")

“C:\rtools40\usr\bin;C:\Program Files\R\R-4.0.0\bin\x64;C:\Rtools\bin;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\IBM\SPSS\Statistics\24\JRE\bin;C:\Program Files\Java\jdk1.8.0_111\bin;C:\Program Files\apache-maven-3.5.0\bin;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Sawtooth Software\CBCHB;C:\Program Files\PsychoPy3;C:\Program Files\PsychoPy3\DLLs;C:\Program Files\MATLAB\R2019b\bin;C:\Program Files\SlikSvn\bin;C:\Users\carin\AppData\Local\Microsoft\WindowsApps;”

readLines("~/.R/Makevars.win")

[1] “CXX14=$(BINPREF)g++ -O2 -march=native -mtune=native”
[2] “CXX14FLAGS=-O3 -mtune=native -march=native -Wno-unused-variable -Wno-unused-function”
[3] “CXX11FLAGS=-O3 -march=native -mtune=native”

I guess the problem is the empty BINPREF. I am sure that I set it manually yesterday but now it’s empty again. Also there seems to be an old PATH variable from previous version of Rtools that I set manually back than to make it work, so maybe that why it wasn’t removed when de-installing them. I tried to figure out how to remove them but my ‘.Rprofil’ file where it should be is empty. So, it seems it’s all not how it should be…

The empty BINPREF actually isn’t an issue, since RTools4 will be automatically taking care of it. It’s usually an issue when it’s not empty, and it’s set to the old rtools directories.

There are two things that I would try here, first thing is to try removing the CXX14=$(BINPREF)g++ -O2 -march=native -mtune=native line from your Makevars.win file, since that’s not needed anymore.

The next thing to try is removing the PATH entry for the old RTools. You can do this by opening the Control Panel and going to the System menu, on the bottom left there will be the option ‘Advanced system settings’. Go to the ‘Advanced’ tab and and click on the ‘Environment Variables’ button, and from there you can edit your PATH variable

1 Like

Thanks for the explanation how to remove the path variable. I followed your instruction and after opening a new session I can confirm that it has been removed.

I could run the line example(stan_model,run.dontrun = TRUE,verbose=TRUE) now and it does some sampling. However at the end there is a warning saying:
Warning: In find.package(package, lib.loc, verbose = verbose) : package ‘base’ found more than once, using the first from “C:/PROGRA~1/R/R-40~1.0/library/base”, “C:/Program Files/R/R-4.0.0/library/base”

Also when I try to run my simple model (which worked before with the exact same code with old Rtools) I still get the vector allocation error.

Those warnings are normal and safe to ignore. For your models that aren’t working, can you make sure that you’ve deleted any .rds files for your models and set rstan_options(auto_write = FALSE) (just for the testing), to make sure that the models get fully recompiled?

2 Likes

Thanks!! I thought I’d already deleted them (because I also read that it worked for other people) but now I saw that I deleted them in the wrong project folder - trying to solve that problems for days definitely had an effect on my concentration. However, it works now.
Thank you so much for your fast answers and helping me!

2 Likes

Hi I have tried to reinstall rcpp by first removing it from Rstudio and then manually deleting the folder from the library as well. After reinstalling and running the code I getfollowing error after file has compiled:

Error in unserialize(socklist[[n]]) : error reading from connection
In addition: Warning message:
In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
‘C:/RBUILD~1/4.0/usr/mingw_/bin/g++’ not found
Error in serialize(data, node$con, xdr = FALSE) :
error writing to connection

This usually means that RTools40 was not found or perhaps was not before RTools35 on the PATH.

I uninstalled everything and did a fresh install of R, Rstudio and stan. But now am back at the first error. When i run more than one chain I get cannot allocate vector of 2.2 GB.

My makvars file looks like so:
CXX14FLAGS=-O3 -march=native -mtune=native
CXX11FLAGS=-O3 -march=corei7 -mtune=corei7
CXX14=C:/RBuildTools/4.0/mingw64/bin/g++ -m$(WIN) -v

Please help!

This is my ouput for

Sys.which(“make”)

“C:\RBUILD~1\4.0\usr\bin\make.exe”

Sys.getenv(“BINPREF”)
" "

Sys.getenv(“PATH”)
“C:\RBuildTools\4.0\usr\bin;C:\Program Files\R\R-4.0.2\bin\x64;C:\Program Files\R\R-4.0.2\bin\x64;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\SASHome\SASFoundation\9.4\core\sasexe;C:\Program Files\SASHome\SASFoundation\9.4\ets\sasexe;C:\Program Files\SASHome\Secure\ccme4;c:Rtoolsbin;c:Rtoolsperlbin;c:RtoolsMinGWbin;c:Rbin;C:\RBuildTools\4.0\usr\bin;”

Hi AAkash,

Some of those Makevars flags will cause issues with RTools 4.0. The first step is to reset those flags to only those that you need. You can do this from R via:

cat("CXX14FLAGS += -O3 -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -Wno-ignored-attributes",file = "~/.R/Makevars.win", sep = "\n", append = FALSE)

After that, can you try running the RStan example model:

library(rstan)
example(stan_model,run.dontrun=TRUE,verbose=TRUE)

And post any lines that start with error:?

I ran the example model you have suggested. This is what i got some warnings:

Warning messages:
1: In find.package(package, lib.loc, verbose = verbose) :
package ‘base’ found more than once, using the first from
“C:/R/R-40~1.2/library/base”,
“C:/R/R-4.0.2/library/base”
2: In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
‘C:/RBUILD~1/4.0/usr/mingw_/bin/g++’ not found

Are these serious or can be ignored?

This might be stupid but could this error be because of large dataset. In the past I have analysed big data with stan without issues so am not sure

Those warnings are safe to ignore

If the example models work but your own model gives the vector size error, then there’s a good chance that its the model and/or data. Try fitting the model to a subset of your data and see of you still get the error. Also, if you post your model and some information about your data, I can make some suggestions on how to cut down the memory usage.