Cannot resolve the "cannot allocate vector of size … " error

toschwo · May 18, 2020, 10:50pm

Where is the compilation argument line? Console? Terminal?

toschwo · May 18, 2020, 11:39pm

Not sure if it helps. I unistalled RStudio, all versions of R, and rtools, then installed everything and followed your Github instructions https://github.com/stan-dev/rstan/wiki/Installing-RStan-from-source-on-Windows#download-and-install-rtools.

pkgbuild::has_build_tools(debug = TRUE) not possible to install rtools via Rstudio. Was only possible manually.
created pers. Makevars, no problem
removed rstan packages, no problem
set cores, no problem
neither installation from source worked. here are error msgs:

Sys.setenv(MAKEFLAGS = “-j4”) # four cores used
install.packages(“rstan”, type = “source”)
Installing package into ‘C:/Users/tschwoerer/Documents/R/win-library/4.0’
(as ‘lib’ is unspecified)
trying URL ‘https://cran.rstudio.com/src/contrib/rstan_2.19.3.tar.gz’
Content type ‘application/x-gzip’ length 922337 bytes (900 KB)
downloaded 900 KB

installing source package ‘rstan’ …
** package ‘rstan’ successfully unpacked and MD5 sums checked
** using staged installation
** libs

*** arch - i386

C:/Rtools/mingw_32/bin/g++ -m32 -std=c++1y -std=gnu++14 -I"C:/PROGRA~1/R/R-40~1.0/include" -DNDEBUG -I"…/inst/include" -I"." -I"C:/Users/tschwoerer/Documents/R/win-library/4.0/StanHeaders/include/src" -DBOOST_DISABLE_ASSERTS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION -I’C:/Users/tschwoerer/Documents/R/win-library/4.0/Rcpp/include’ -I’C:/Users/tschwoerer/Documents/R/win-library/4.0/RcppEigen/include’ -I’C:/Users/tschwoerer/Documents/R/win-library/4.0/BH/include’ -I’C:/Users/tschwoerer/Documents/R/win-library/4.0/StanHeaders/include’ -O3 -march=corei7 -mtune=corei7 -c sparse_extractors.cpp -o sparse_extractors.o
sh: C:/Rtools/mingw_32/bin/g++: No such file or directory

make: *** [C:/PROGRA~1/R/R-40~1.0/etc/i386/Makeconf:229: sparse_extractors.o] Error 127
ERROR: compilation failed for package ‘rstan’

removing ‘C:/Users/tschwoerer/Documents/R/win-library/4.0/rstan’
Warning in install.packages :
installation of package ‘rstan’ had non-zero exit status

The downloaded source packages are in
‘C:\Users\tschwoerer\AppData\Local\Temp\RtmpakBgyJ\downloaded_packages’

remotes::install_github(“stan-dev/rstan”, ref = “develop”, subdir = “rstan/rstan”, build_opts = “”)
Downloading GitHub repo stan-dev/rstan@develop
stan-dev-rstan-5869a5e/StanHeaders/inst/include/libsundials: Can’t create ‘\\?\C:\Users\TSCHWO~1\AppData\Local\Temp\RtmpakBgyJ\remotes40886e1c1fe4\stan-dev-rstan-5869a5e\StanHeaders\inst\include\libsundials’
stan-dev-rstan-5869a5e/StanHeaders/inst/include/src: Can’t create ‘\\?\C:\Users\TSCHWO~1\AppData\Local\Temp\RtmpakBgyJ\remotes40886e1c1fe4\stan-dev-rstan-5869a5e\StanHeaders\inst\include\src’
stan-dev-rstan-5869a5e/StanHeaders/inst/include/stan: Can’t create ‘\\?\C:\Users\TSCHWO~1\AppData\Local\Temp\RtmpakBgyJ\remotes40886e1c1fe4\stan-dev-rstan-5869a5e\StanHeaders\inst\include\stan’
tar.exe: Error exit delayed from previous errors.
“C:\PROGRA~1\Git\cmd\git.exe” clone --depth 1 --no-hardlinks --recurse-submodules --branch develop https://github.com/stan-dev/stan.git C:\Users\TSCHWO~1\AppData\Local\Temp\RtmpakBgyJ\remotes40886e1c1fe4/stan-dev-rstan-5869a5e/rstan/rstan/…/…/StanHeaders/inst/include/upstream
Error: Failed to install ‘rstan’ from GitHub:
Command failed (128)
In addition: Warning messages:
1: In utils::untar(tarfile, …) :
‘tar.exe -xf “C:\Users\TSCHWO~1\AppData\Local\Temp\RtmpakBgyJ\file40883eb67ae.tar.gz” -C “C:/Users/TSCHWO~1/AppData/Local/Temp/RtmpakBgyJ/remotes40886e1c1fe4”’ returned error code 1
2: In system(full, intern = TRUE, ignore.stderr = quiet) :
running command ‘“C:\PROGRA~1\Git\cmd\git.exe” clone --depth 1 --no-hardlinks --recurse-submodules --branch develop https://github.com/stan-dev/stan.git C:\Users\TSCHWO~1\AppData\Local\Temp\RtmpakBgyJ\remotes40886e1c1fe4/stan-dev-rstan-5869a5e/rstan/rstan/…/…/StanHeaders/inst/include/upstream’ had status 128

toschwo · May 18, 2020, 11:53pm

Following https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started:
6. install.packages(“rstan”, repos = “https://cloud.r-project.org/”, dependencies = TRUE), installed rstan no problem
7.pkgbuild::has_build_tools(debug = TRUE), error message in RStudio: “Building R package from source requires installation of additional build tools. Do you want to install the additionalk tools now?”
Confused as Rtools 4.0 is clearly installed. I checked the Windows apps.
7a: click yes: FALSE
7b: click no: FALSE

bgoodri · May 19, 2020, 6:36am

In what directory is Rtools installed?

Stephen_Martin · May 19, 2020, 7:06am

Honestly, I think this might be due to some unnecessary Makevars.win configuration. The wiki is outdated.

Uninstall Rtools (all versions). Install Rtools 40. Rename your Makevars.win to something else, just to remove it from the picture for now. And try again. You may also need to rename .Renviron to something else as well.

First, a default configuration for Rtools 40 should not be C:/Rtools at all. It should be C:/rtools40.

Second, ensure after running Rtools40, that you open R and run: writeLines('PATH="${RTOOLS40_HOME}\\usr\\bin;${PATH}"', con = "~/.Renviron")

Third, if that still does not work, my guess is you have some lingering PATH problems, and a lingering Rtools directory.

You do not need a Makevars.win file with R 4.0 and Rtools40 to run Stan; it’s mainly useful for adding compiler optimizations, which should be done after ensuring that it works. All you should need, is R 4.0, Rtools40 (with defaults selected), and to run the writeLines('PATH="${RTOOLS40_HOME}\\usr\\bin;${PATH}"', con = "~/.Renviron") command in R. Then restart R, and it should compile the model.

mauthier · May 19, 2020, 8:14am

Hi,
had the same problem as you since upgarding to R 4.0 and followed the wiki to install Rstan, and amend the Makevars.win files.
Still, I got error messages “cannot allocate…”
Turned out my code was also using the recommendation to avoid recompilation of unchanged Stan programs by calling
rstan::rstan_options(auto_write = TRUE)

However my stanmodel.rds were outdated. Once these were removed and models were recompiled, everything was fine.

Could that be part of your problem?

Cheers

toschwo · May 19, 2020, 3:17pm

I uninstalled everything completely. Then re-installed just R4.0 and rtools40.
Followed your instructions renaming Makevars.win and .Renviron to something else. Same error: “cannot allocate vector of size …” Then did your recommended steps:

rtools installed correctly in C:/rtools40
Console output:
TRANSLATING MODEL ‘flights_model’ FROM Stan CODE TO C++ CODE NOW.
successful in parsing the Stan model ‘flights_model’.
CHECKING DATA AND PREPROCESSING FOR MODEL ‘flights_model’ NOW.
COMPILING MODEL ‘flights_model’ NOW.
Error in get(“storage”, envir = as.environment(x)) :
object ‘storage’ not found
If I do have lingering path problems, how can I eliminate those?
Followed Mauthier’s suggestions
4.1. took out rstan_options(autho_write = TRUE), still “cannot allocate vector…”
4.2. removed stanmodel.rds
RStudio error message as before during install: "Building R package from source requires installation of additional build tools. Do you want to install the additional tools now?”
RStudio asked me this twice, I clicked yes, twice:
Compilation argument:

C:/PROGRA~1/R/R-40~1.0/bin/x64/R CMD SHLIB file23ac3b7144af.cpp 2> file23ac3b7144af.cpp.err.txt 
Warning message:
In system(cmd) : 'make' not found
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
  cannot open file 'file23ac3b7144af.cpp.err.txt': No such file or directory```

mauthier · May 19, 2020, 5:25pm

Hi,
just to be sure, have you also done the new steps required with Rtools 4.0 to put Rtools on your path? It’s described in section " Putting Rtools on the PATH" there:
https://cran.r-project.org/bin/windows/Rtools/

I also had the error about ‘make’ not found because I first did not see that there was this extra step now.
Cheers

Stephen_Martin · May 19, 2020, 6:13pm

Could you try it outside of RStudio? Just open R (x64), and run an RStan example.

Is it loading a saved workspace from before? After opening R, run rm(list=ls()) and try again.

And what is ‘storage’?

toschwo · May 19, 2020, 6:18pm

mauthier, yes, I did, I followed the steps but the last step–installing from source–threw an error.
C:/Rtools/mingw_32/bin/gcc -I"C:/PROGRA~1/R/R-40~1.0/include" -DNDEBUG -Iyajl/api -D__USE_MINGW_ANSI_STDIO -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c base64.c -o base64.o
sh: C:/Rtools/mingw_32/bin/gcc: No such file or directory
make: *** [C:/PROGRA~1/R/R-40~1.0/etc/i386/Makeconf:222: base64.o] Error 127
ERROR: compilation failed for package ‘jsonlite’

removing ‘C:/Users/tschwoerer/Documents/R/win-library/4.0/jsonlite’
restoring previous ‘C:/Users/tschwoerer/Documents/R/win-library/4.0/jsonlite’
Warning in install.packages :
installation of package ‘jsonlite’ had non-zero exit status

toschwo · May 19, 2020, 6:35pm

Ran model from within R (x64) pointing to proper working directory and got the following:
Compilation argument:
C:/PROGRA~1/R/R-40~1.0/bin/x64/R CMD SHLIB file3f383f61ee.cpp 2> file3f383f61ee.cpp.err.txt
C:/Rtools/mingw_64/bin/g++ -m64 -std=c++1y -std=gnu++14 -I"C:/PROGRA~1/R/R-40~1.0/include" -DNDEBUG -I"C:/Users/tschwoerer/Documents/R/win-library/4.0/Rcpp/include/" -I"C:/Users/tschwoerer/Documents/R/win-library/4.0/RcppEigen/include/" -I"C:/Users/tschwoerer/Documents/R/win-library/4.0/RcppEigen/include/unsupported" -I"C:/Users/tschwoerer/Documents/R/win-library/4.0/BH/include" -I"C:/Users/tschwoerer/Documents/R/win-library/4.0/StanHeaders/include/src/" -I"C:/Users/tschwoerer/Documents/R/win-library/4.0/StanHeaders/include/" -I"C:/Users/tschwoerer/Documents/R/win-library/4.0/rstan/include" -DEIGEN_NO_DEBUG -D_REENTRANT -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp -std=c++1y -march=native -O3 -march=native -mtune=native -c file3f383f61ee.cpp -o file3f383f61ee.o
sh: C:/Rtools/mingw_64/bin/g++: No such file or directory
make: *** [C:/PROGRA~1/R/R-40~1.0/etc/x64/Makeconf:229: file3f383f61ee.o] Error 127
Error in file(con, “r”) : cannot open the connection
In addition: Warning message:
In file(con, “r”) :
cannot open file ‘file3f383f61ee.cpp.err.txt’: No such file or directory

Same compilation error after rm(list=ls())

What do you mean by "What is ‘storage’?

toschwo · May 19, 2020, 6:52pm

Seems like it is still looking for Rtools when it should be looking for rtools40 …

Stephen_Martin · May 19, 2020, 8:04pm

Are you absolutely, positively, 100% certain you have renamed or deleted your Makevars.win file, in Documents/.R/Makevars.win ? Because yes, it is looking for a compiler in C:/Rtools, which doesn’t exist.

And are you 100% certain you only have PATH="${RTOOLS40_HOME}\usr\bin;${PATH}" in Documents/.Renviron ?

And are you 100% certain you are starting with a clean workspace?

If so, then click start, and search for ‘environment variables’, then click ‘edit system environment variables’. Then click environment variables. Go to system variables, and edit Path. Remove only the line that has Rtools in it.

Then log out, log back in, and try Rstan again.

Stephen_Martin · May 19, 2020, 8:07pm

Could you also open R, and type Sys.getenv() and paste the output here?

toschwo · May 19, 2020, 8:57pm

no, Makevars.win is in its location in C:\Users\tschwoerer\Documents.R but in Windows explorer the Type for that file is listed as Adobe CMS Extension
yes
yes
Did not go this far as answer to 1. was “no”
Then followed your next comment below

AGSDESKTOPJAVA C:\Program Files (x86)\ArcGIS\Desktop10.6
ALLUSERSPROFILE C:\ProgramData
APPDATA C:\Users\tschwoerer\AppData\Roaming
BINPREF C:/Rtools/mingw_(WIN)/bin/ ChocolateyInstall C:\ProgramData\chocolatey ChocolateyLastPathUpdate Wed Sep 26 17:18:32 2018 CommonProgramFiles C:\Program Files\Common Files CommonProgramFiles(x86) C:\Program Files (x86)\Common Files CommonProgramW6432 C:\Program Files\Common Files COMPUTERNAME ANC-MB0910A18 ComSpec C:\WINDOWS\system32\cmd.exe DriverData C:\Windows\System32\Drivers\DriverData GFORTRAN_STDERR_UNIT -1 GFORTRAN_STDOUT_UNIT -1 GOPATH C:\Users\tschwoerer\go GOROOT C:\Go\ HOME C:\Users\tschwoerer\Documents HOMEDRIVE P: HOMEPATH \ HOMESHARE \\anc-filesrv01.ua.ad.alaska.edu\employeeHome\30458916
LOCALAPPDATA C:\Users\tschwoerer\AppData\Local
LOGONSERVER \ANC-ADUA01
MSYS2_ENV_CONV_EXCL R_ARCH
NUMBER_OF_PROCESSORS 8
OneDrive C:\Users\tschwoerer\OneDrive
OS Windows_NT
PATH C:\rtools40\usr\bin;C:\Rtools\bin;C:\Program
Files (x86)\Common
Files\Oracle\Java\javapath;C:\Rtools\bin;c:\Rtools\bin;c:\Rtools\mingw_32\bin;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\WINDOWS\System32\OpenSSH;C:\ProgramData\chocolatey\bin;C:\Go\bin;C:\Program
Files\Git\cmd;C:\Hugo\bin;C:\Users\tschwoerer\AppData\Local\GitHubDesktop\bin;C:\Users\tschwoerer\AppData\Local\Programs\Python\Python37;C:\Users\tschwoerer\AppData\Local\Programs\MiKTeX
2.9\miktex\bin\x64;%USERPROFILE%\AppData\Local\Microsoft\WindowsApps;
PATHEXT .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE
AMD64
PROCESSOR_IDENTIFIER Intel64 Family 6 Model 58 Stepping 9,
GenuineIntel
PROCESSOR_LEVEL 6
PROCESSOR_REVISION 3a09
ProgramData C:\ProgramData
ProgramFiles C:\Program Files
ProgramFiles(x86) C:\Program Files (x86)
ProgramW6432 C:\Program Files
PSModulePath C:\Program
Files\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules
PUBLIC C:\Users\Public
R_ARCH /x64
R_COMPILED_BY gcc 8.3.0
R_HOME C:/PROGRA~1/R/R-40~1.0
R_LIBS_USER C:\Users\tschwoerer\Documents/R/win-library/4.0
R_USER C:\Users\tschwoerer\Documents
RTOOLS40_HOME C:\rtools40
SESSIONNAME Console
SystemDrive C:
SystemRoot C:\WINDOWS
TEMP C:\Users\TSCHWO~1\AppData\Local\Temp
TMP C:\Users\TSCHWO~1\AppData\Local\Temp
UATDATA C:\WINDOWS\CCM\UATData\D9F8C395-CAB8-491d-B8AC-179A1FE1BE77
USERDNSDOMAIN UA.AD.ALASKA.EDU
USERDOMAIN UA
USERDOMAIN_ROAMINGPROFILE
UA
USERNAME tschwoerer
USERPROFILE C:\Users\tschwoerer
windir C:\WINDOWS

toschwo · May 19, 2020, 9:03pm

Also, here is my system variable window

Stephen_Martin · May 19, 2020, 10:02pm

Ok, again - You should not have anything defined in your Makevars.win right now. Move, delete, or rename it to something else. BINPREF is pointing to C:/Rtools, and it shouldn’t be.

Do you have a .Rprofile somewhere defining BINPREF?

It seems like there is some configuration, somewhere on your system (Makevars.win, .Renviron, .Rprofile) that is telling R that the compiler should be in BINPREF, defined somewhere as being C:/Rtools.

Edit: As for Windows saying that Makevars.win is an adobe CMS type, ignore it. Windows has a really rudimentary, stupid way of finding filetypes. It thinks Makevars.win is really a file named Makevars with extension win, which it apparently thinks is adobe. It is in reality a plain text file that defines compiler options on windows - hence why I said it should be removed from the equation for now.

toschwo · May 19, 2020, 10:30pm

.Rprofile contains:
Sys.setenv(BINPREF = "C:/Rtools/mingw_$(WIN)/bin/")

.Renviron contains:
PATH="${RTOOLS40_HOME}\usr\bin;${PATH}"

Stephen_Martin · May 20, 2020, 12:19am

I can’t see what .Renviron contains, but I assume it is fine.

Remove Sys.setenv(BINPREF = “C:/Rtools/mingw_$(WIN)/bin/”) from your .Rprofile.

Also… move or rename Documents/.R/Makevars.win.

toschwo · May 20, 2020, 4:17am

Ok see .Renviron above.

Model seems to compile properly now.

Is there anything else I should know to not run into the same issue again when updating to new versions or R? What about the edits to .Rprofile and the renaming of Makevars.win?