Error: cannot allocate vector of size 703998.9 Gb

Modeling
1

I’m getting this insane error.

Environment details:

I’m running RStudio Server on Ubuntu:

Operating System: Ubuntu 16.04.6 LTS
Kernel: Linux 4.4.0-157-generic
Architecture: x86-64

Here is the R version stuff:

platform       x86_64-pc-linux-gnu         
arch           x86_64                      
os             linux-gnu                   
system         x86_64, linux-gnu           
status                                     
major          3                           
minor          6.1                         
year           2019                        
month          07                          
day            05                          
svn rev        76782                       
language       R                           
version.string R version 3.6.1 (2019-07-05)
nickname       Action of the Toes

Session info:

R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.6 LTS

Matrix products: default
BLAS:   /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] bayestestR_0.2.2      shinystan_2.5.0       shiny_1.3.2          
[4] rstan_2.19.2          ggplot2_3.2.0         StanHeaders_2.18.1-10

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.1         lattice_0.20-38    prettyunits_1.0.2  ps_1.3.0          
 [5] zoo_1.8-6          gtools_3.8.1       assertthat_0.2.1   digest_0.6.19     
 [9] mime_0.7           R6_2.4.0           plyr_1.8.4         ggridges_0.5.1    
[13] stats4_3.6.1       colourpicker_1.0   pillar_1.4.1       rlang_0.3.4       
[17] lazyeval_0.2.2     rstudioapi_0.10    miniUI_0.1.1.1     callr_3.2.0       
[21] DT_0.7             shinythemes_1.1.2  shinyjs_1.0        stringr_1.4.0     
[25] htmlwidgets_1.3    loo_2.1.0          igraph_1.2.4.1     munsell_0.5.0     
[29] compiler_3.6.1     httpuv_1.5.1       xfun_0.7           pkgconfig_2.0.2   
[33] base64enc_0.1-3    pkgbuild_1.0.3     htmltools_0.3.6    insight_0.3.0     
[37] tidyselect_0.2.5   tibble_2.1.3       gridExtra_2.3      threejs_0.3.1     
[41] matrixStats_0.54.0 crayon_1.3.4       dplyr_0.8.1        withr_2.1.2       
[45] later_0.8.0        grid_3.6.1         xtable_1.8-4       gtable_0.3.0      
[49] magrittr_1.5       scales_1.0.0       stringi_1.4.3      cli_1.1.0         
[53] reshape2_1.4.3     promises_1.0.1     dygraphs_1.1.1.6   xts_0.11-2        
[57] tools_3.6.1        glue_1.3.1         markdown_1.0       purrr_0.3.2       
[61] crosstalk_1.0.0    rsconnect_0.8.13   processx_3.3.1     parallel_3.6.1    
[65] inline_0.3.15      colorspace_1.4-1   bayesplot_1.7.0    knitr_1.23

Here is the very simple model I’m trying to fit:

N1 <- 18  # Define the sample size
y1 <- c(rep(1, 12), rep(0, 6))  # 12 successes and 6 failures
stan_data <- list(N = N1, y = y1)
stan_data

As I’m in an R Notebook, my Stan code chunk looks like this. (The chunk option specifies output.var = "bin_unif", cache = TRUE)

data {
    int<lower = 0> N;
    int<lower = 0, upper = 1> y[N];
}
parameters {
    real<lower = 0, upper = 1> theta;
}
model {
    theta ~ uniform(0, 1);  // prior
    y ~ bernoulli(theta);   // likelihood
}

The code giving the error is:

set.seed(54321)
fit_bin_unif <- sampling(bin_unif, data = stan_data)

Traceback:

10. Module(module, mustStart = TRUE)
9. .getModulePointer(x)
8. new("Module", .xData = <environment>)$stan_fit4modeld911db93a46_stan_d91119d4951
7. new("Module", .xData = <environment>)$stan_fit4modeld911db93a46_stan_d91119d4951
6. eval(call("$", mod, paste("stan_fit4", model_cppname, sep = "")))
5. eval(call("$", mod, paste("stan_fit4", model_cppname, sep = "")))
4. object@mk_cppmodule(object)
3. .local(object, ...)
2. sampling(bin_unif, data = stan_data)
1. sampling(bin_unif, data = stan_data)

Any help would be much appreciated. I teach Bayesian data analysis in a college classroom, so RStudio Server has to work in order for my students to learn Stan and complete the assignments.

1 Like
2

That model + this code worked for me:

library(rstan)

N1 <- 18  # Define the sample size
y1 <- c(rep(1, 12), rep(0, 6))  # 12 successes and 6 failures
stan_data <- list(N = N1, y = y1)
stan_data

set.seed(54321)
bin_unif = stan_model("model.stan")
fit = sampling(bin_unif, data = stan_data)

Edit: Ubuntu 18.04, Rstudio server 1.2.1335

1 Like
3

It works for me too on my Windows machine…it works everywhere except on the server.

4

I would guess compiler flags are the culprit, or perhaps a compiler bug. What do you have in ~/.R/Makevars and what compiler and version are you using?

5

Yeah, the Makevars thing has been problematic. Honestly, I can’t even get rstan to install properly using most suggestions out there on the web about what should go into Makevars. I finally found one magic incantation in some forum somewhere that would even allow rstan to install:

CXX14 = g++ -std=c++1y -Wno-unused-variable -Wno-unused-function -fPIC

(I freely admit I have no idea what any of these flags do.)

6

OK, that is similar to the one on the RStan wiki

But what version of g++ is that?

7

gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11)

8

That compiler isn’t one that we have much experience with, but it is supposed to work. My guess is that Rcpp was installed with a different compiler or settings, so I would try reinstalling Rcpp with what you have in Makevars.

1 Like
9

That’s a good suggestion. I’ll try it and report back.

10

By golly, reinstalling Rcpp worked! Thanks, Ben!

2 Likes
11

I encounter the same problem and I already removed and reinstalled the Rcpp package. However I did not specify anything according to what is written in the Makevar file.
I got the following options on what I read before to make it work with Rtools 4:
CXX14=$(BINPREF)g++ -O2 -march=native -mtune=native CXX14FLAGS=-O3 -march=native -mtune=native CXX11FLAGS=-O3 -march=native -mtune=native

How should I go forward to properly “reinstalling Rcpp with what I [you] have in Makevars”?
I also had problems before with Rtools 4.0 not being connected correctly and not being able to compile, but I fixed it.
pkgbuild::check_build_tools() Your system is ready to build packages!

There are also two more packages that I see in the win-library folder called RcppEigen and RcppParallel. Does it have something to do with that?
Or do I need to remove the Rcpp package by hand and not with the remove.packages("Rcpp") command?

12

Have you tried the instructions over in this thread: R 4.0, rstan, and you? They should help

1 Like
13

Yes I did. Already followed them twice (install-uninstall-reinstall) and it didn’t solve the problem.

14

Does the example model compile for you?

example(stan_model,run.dontrun = TRUE,verbose=TRUE)

If not, can you post the outputs from:

Sys.which("make")
Sys.getenv("BINPREF")
Sys.getenv("PATH")
readLines("~/.R/Makevars.win")

Just to check for any common sources of configuration issues

1 Like
15

I already tried the example yesterday because I read it here (Error: cannot allocate vector of size 17348.7 Gb - #10 by bgoodri) but RStudio always freezes and doesn’t react anymore after around 2 minutes.

Those are the outputs:
Sys.which("make")
make
"C:\\rtools40\\usr\\bin\\make.exe"

Sys.getenv("BINPREF")
""
Sys.getenv("PATH")

“C:\rtools40\usr\bin;C:\Program Files\R\R-4.0.0\bin\x64;C:\Rtools\bin;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\IBM\SPSS\Statistics\24\JRE\bin;C:\Program Files\Java\jdk1.8.0_111\bin;C:\Program Files\apache-maven-3.5.0\bin;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Sawtooth Software\CBCHB;C:\Program Files\PsychoPy3;C:\Program Files\PsychoPy3\DLLs;C:\Program Files\MATLAB\R2019b\bin;C:\Program Files\SlikSvn\bin;C:\Users\carin\AppData\Local\Microsoft\WindowsApps;”

readLines("~/.R/Makevars.win")

[1] “CXX14=$(BINPREF)g++ -O2 -march=native -mtune=native”
[2] “CXX14FLAGS=-O3 -mtune=native -march=native -Wno-unused-variable -Wno-unused-function”
[3] “CXX11FLAGS=-O3 -march=native -mtune=native”

I guess the problem is the empty BINPREF. I am sure that I set it manually yesterday but now it’s empty again. Also there seems to be an old PATH variable from previous version of Rtools that I set manually back than to make it work, so maybe that why it wasn’t removed when de-installing them. I tried to figure out how to remove them but my ‘.Rprofil’ file where it should be is empty. So, it seems it’s all not how it should be…

16

The empty BINPREF actually isn’t an issue, since RTools4 will be automatically taking care of it. It’s usually an issue when it’s not empty, and it’s set to the old rtools directories.

There are two things that I would try here, first thing is to try removing the CXX14=$(BINPREF)g++ -O2 -march=native -mtune=native line from your Makevars.win file, since that’s not needed anymore.

The next thing to try is removing the PATH entry for the old RTools. You can do this by opening the Control Panel and going to the System menu, on the bottom left there will be the option ‘Advanced system settings’. Go to the ‘Advanced’ tab and and click on the ‘Environment Variables’ button, and from there you can edit your PATH variable

1 Like
17

Thanks for the explanation how to remove the path variable. I followed your instruction and after opening a new session I can confirm that it has been removed.

I could run the line example(stan_model,run.dontrun = TRUE,verbose=TRUE) now and it does some sampling. However at the end there is a warning saying:
Warning: In find.package(package, lib.loc, verbose = verbose) : package ‘base’ found more than once, using the first from “C:/PROGRA~1/R/R-40~1.0/library/base”, “C:/Program Files/R/R-4.0.0/library/base”

Also when I try to run my simple model (which worked before with the exact same code with old Rtools) I still get the vector allocation error.

18

Those warnings are normal and safe to ignore. For your models that aren’t working, can you make sure that you’ve deleted any .rds files for your models and set rstan_options(auto_write = FALSE) (just for the testing), to make sure that the models get fully recompiled?

2 Likes
19

Thanks!! I thought I’d already deleted them (because I also read that it worked for other people) but now I saw that I deleted them in the wrong project folder - trying to solve that problems for days definitely had an effect on my concentration. However, it works now.
Thank you so much for your fast answers and helping me!

2 Likes
20

Hi I have tried to reinstall rcpp by first removing it from Rstudio and then manually deleting the folder from the library as well. After reinstalling and running the code I getfollowing error after file has compiled:

Error in unserialize(socklist[[n]]) : error reading from connection
In addition: Warning message:
In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
‘C:/RBUILD~1/4.0/usr/mingw_/bin/g++’ not found
Error in serialize(data, node$con, xdr = FALSE) :
error writing to connection