Problems getting Stan to run after upgrading to R 4.0.2

Chris_Dalton · September 7, 2020, 1:03pm

2020-09-06T14:00:00Z
Hello
This seems to be a common problem but the threads are long and difficult to follow. Can’t get Stan to run now after multiple re-installs of R and R studio.
Tried fiddling with the makevars.win file; this is what I have in it now in the makevars file:

CXX14=$(BINPREF)g++ -O2 -march=native -mtune=native”

and I’ve tried

Sys.setenv(BINPREF = "C:/RBuildTools/4.0/mingw64/bin/g++.exe")

as my Rtools directory downloaded as RBuildTools. I keep getting this error message:

Error in compileCode(f, code, language = language, verbose = verbose) : 
  sh: C:/Rtools/mingw_64/bin/g++: No such file or directorymake: *** [C:/PROGRA~1/R/R-40~1.2/etc/x64/Makeconf:229: file123061c3410a.o] Error 127
In addition: Warning message:
In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  '-E' not found
Error in sink(type = "output") : invalid connection

Thanks for any advice.
regards
Chris

Windows 10
RStan 2.21.2
R 4.0.2
RStudio 1.3.1073

andrjohns · September 7, 2020, 1:41pm

Hi Chris,

With RTools 4, you don’t want to be setting the CXX14 variable or the BINPREF variable, you also don’t want to use -march=native, since that can cause crashes.

The first step is to reset your Makevars to some ‘safe’ but optimised settings:

cat("CXX14FLAGS += -O3 -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2",file = "~/.R/Makevars.win", sep = "\n", append = FALSE)

Can you run that and then run the example model:

library(rstan)
example(stan_model,run.dontrun=TRUE,verbose=TRUE)

And post any line(s) that start with error:?

andrjohns · September 7, 2020, 1:45pm

Also, small world! I finished my Biostats masters through Sydney uni last year

Chris_Dalton · September 8, 2020, 4:09am

Thanks very much for your reply.
The error message I got now is

Error in compileCode(f, code, language = language, verbose = verbose) : 
  sh: C:/Rtools/mingw_64/bin/g++: No such file or directorymake: *** [C:/PROGRA~1/R/R-40~1.2/etc/x64/Makeconf:229: file1fc42e38522.o] Error 127
In addition: Warning messages:
1: In find.package(package, lib.loc, verbose = verbose) :
  package ‘base’ found more than once, using the first from
  “C:/PROGRA~1/R/R-40~1.2/library/base”,
  “C:/Program Files/R/R-4.0.2/library/base”
2: In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  'C:/RBUILD~1/4.0/usr/mingw_/bin/g++' not found

I finished the MBiostats last semester at Sydney Uni.
Are you Sydney based (if it’s not an inappropriate question on a public forum)?
Regards
Chris

andrjohns · September 8, 2020, 5:43am

No, I’m over in Perth. But always good to see more Australians on the forums!

This line:

sh: C:/Rtools/mingw_64/bin/g++

Indicates that R is looking for the old Rtools installation. What output do you get from the following:

Sys.getenv("PATH")
Sys.getenv("BINPREF")
readLines("~/.R/Makevars.win")
readLines("~/.Renviron")

Chris_Dalton · September 8, 2020, 6:47am

Thanks
I get this with that request

 Sys.getenv("PATH")
[1] "C:\\RBuildTools\\4.0\\usr\\bin;C:\\Program Files\\R\\R-4.0.2\\bin\\x64;C:\\Program Files (x86)\\Common Files\\Oracle\\Java\\javapath_target_289689671;C:\\Perl64\\site\\bin;C:\\Perl64\\bin;c:\\Rtools\\bin;c:\\Rtools\\mingw_32\\bin;C:\\Windows\\System32;C:\\Windows;C:\\Windows\\System32\\wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0;C:\\Windows\\System32\\OpenSSH;C:\\Program Files\\Intel\\WiFi\\bin;C:\\Program Files\\Common Files\\Intel\\WirelessCommon;C:\\Program Files (x86)\\Intel\\Intel(R) Management Engine Components\\DAL;C:\\Program Files\\Intel\\Intel(R) Management Engine Components\\DAL;C:\\Users\\dalto\\AppData\\Local\\Microsoft\\WindowsApps"
> Sys.getenv("BINPREF")
[1] "C:/Rtools/mingw_$(WIN)/bin/"
> readLines("~/.R/Makevars.win")
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
  cannot open file 'C:/Users/dalto/Documents/.R/Makevars.win': No such file or directory
> readLines("~/.Renviron")
[1] "PATH=\"${RTOOLS40_HOME}\\usr\\bin;${PATH}\""

Something to do with RTools and RBuildTools?
Regards
Chris

andrjohns · September 8, 2020, 6:56am

It looks like the BINPREF variable is being set to the old RTools installation, which tells R where to look for the compilers. I think you probably have an .Rprofile file in your home directory that’s setting the BINPREF variable on startup.

If you don’t need any other settings in the file, you can just delete it via:

unlist("~/.Rprofile")

Otherwise just open the file in a text editor and remove the line mentioning BINPREF.

After deleting/editing the .Rprofile file, restart R and try your model again.

Additionally, you’re missing out on some compiler optimisations by not setting up a Makevars file. You can run the following to setup a Makevars:

cat("CXX14FLAGS += -O3 -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2",file = "~/.R/Makevars.win", sep = "\n", append = FALSE)

Chris_Dalton · September 8, 2020, 7:24am

Now I seem to have some other problems!
1.RStudio keeps aborting the session when I try to open a file.
2.When using

cat("CXX14FLAGS += -O3 -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2",file = "~/.R/Makevars.win", sep = "\n", append = FALSE)

I get

Error in file(file, ifelse(append, "a", "w")) : 
  cannot open the connection
In addition: Warning message:
In file(file, ifelse(append, "a", "w")) :
  cannot open file 'C:/Users/dalto/Documents/.R/Makevars.win': No such file or directory

My makevars file is here

"C:\Users\dalto\Documents\R\win-library\4.0\rstantools\include\sys\Makevars.win"

and contains

cat("CXX14FLAGS += -O3 -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2",file = "~/.R/Makevars.win", sep = "\n", append = FALSE)

3.When running a stan program I get

Warning message:
In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  'C:/RBUILD~1/4.0/usr/mingw_/bin/g++' not found

4.Running

library(rstan)
example(stan_model,run.dontrun=TRUE,verbose=TRUE)

I get

Warning messages:
1: In find.package(package, lib.loc, verbose = verbose) :
  package ‘base’ found more than once, using the first from
  “C:/PROGRA~1/R/R-40~1.2/library/base”,
  “C:/Program Files/R/R-4.0.2/library/base”
2: In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  'C:/RBUILD~1/4.0/usr/mingw_/bin/g++' not found

Sorry about this! Stan used to work well before I upgraded R and RStudio!
Regards
Chris

andrjohns · September 8, 2020, 7:35am

For the makevars file, you need to have it in your Documents folder, in a folder named ‘.R’

So you need to create a folder:

C:/Users/dalto/Documents/.R/

And then inside that folder create a file called Makevars.win which contains:

CXX14FLAGS += -O3 -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2

The cat(...) command was just to automatically create that file and it’s contents (not sure why it didn’t work), you don’t want to have the actual cat(...) command in there.

As for the models, those warnings are all safe to ignore (the g++ one will be fixed in the next release). Do the models all compile and sample without issue?

andrjohns · September 8, 2020, 7:38am

As another check, would you be able to post the output from:

devtools::session_info("rstan")

Chris_Dalton · September 8, 2020, 8:38am

This is the output from

devtools::session_info("rstan")

Session info -----------------------------------------------------------------------------------------------------------------
 setting  value                       
 version  R version 4.0.2 (2020-06-22)
 os       Windows 10 x64              
 system   x86_64, mingw32             
 ui       RStudio                     
 language (EN)                        
 collate  English_Australia.1252      
 ctype    English_Australia.1252      
 tz       Australia/Sydney            
 date     2020-09-08                  

- Packages ---------------------------------------------------------------------------------------------------------------------
 ! package      * version   date       lib source                         
   assertthat     0.2.1     2019-03-21 [1] CRAN (R 4.0.2)                 
   backports      1.1.9     2020-08-24 [1] CRAN (R 4.0.2)                 
   BH             1.72.0-3  2020-01-08 [1] CRAN (R 4.0.0)                 
   callr          3.4.3     2020-03-28 [1] CRAN (R 4.0.2)                 
   checkmate      2.0.0     2020-02-06 [1] CRAN (R 4.0.2)                 
   cli            2.0.2     2020-02-28 [1] CRAN (R 4.0.2)                 
   colorspace     1.4-1     2019-03-18 [1] CRAN (R 4.0.2)                 
   crayon         1.3.4     2017-09-16 [1] CRAN (R 4.0.2)                 
   curl           4.3       2019-12-02 [1] CRAN (R 4.0.2)                 
   desc           1.2.0     2018-05-01 [1] CRAN (R 4.0.2)                 
   digest         0.6.25    2020-02-23 [1] CRAN (R 4.0.2)                 
   ellipsis       0.3.1     2020-05-15 [1] CRAN (R 4.0.2)                 
   evaluate       0.14      2019-05-28 [1] CRAN (R 4.0.2)                 
   fansi          0.4.1     2020-01-08 [1] CRAN (R 4.0.2)                 
   farver         2.0.3     2020-01-16 [1] CRAN (R 4.0.2)                 
   ggplot2      * 3.3.2     2020-06-19 [1] CRAN (R 4.0.2)                 
   glue           1.4.2     2020-08-27 [1] CRAN (R 4.0.2)                 
   gridExtra      2.3       2017-09-09 [1] CRAN (R 4.0.2)                 
   gtable         0.3.0     2019-03-25 [1] CRAN (R 4.0.2)                 
   inline         0.3.15    2020-09-07 [1] Github (bgoodri/inline@21105c7)
   isoband        0.2.2     2020-06-20 [1] CRAN (R 4.0.2)                 
   jsonlite       1.7.0     2020-06-25 [1] CRAN (R 4.0.2)                 
   labeling       0.3       2014-08-23 [1] CRAN (R 4.0.0)                 
   lattice        0.20-41   2020-04-02 [2] CRAN (R 4.0.2)                 
   lifecycle      0.2.0     2020-03-06 [1] CRAN (R 4.0.2)                 
   loo            2.3.1     2020-07-14 [1] CRAN (R 4.0.2)                 
   magrittr       1.5       2014-11-22 [1] CRAN (R 4.0.2)                 
   MASS           7.3-51.6  2020-04-26 [2] CRAN (R 4.0.2)                 
   Matrix         1.2-18    2019-11-27 [2] CRAN (R 4.0.2)                 
   matrixStats    0.56.0    2020-03-13 [1] CRAN (R 4.0.2)                 
   mgcv           1.8-31    2019-11-09 [2] CRAN (R 4.0.2)                 
   munsell        0.5.0     2018-06-12 [1] CRAN (R 4.0.2)                 
   nlme           3.1-148   2020-05-24 [2] CRAN (R 4.0.2)                 
   pillar         1.4.6     2020-07-10 [1] CRAN (R 4.0.2)                 
   pkgbuild       1.1.0     2020-07-13 [1] CRAN (R 4.0.2)                 
   pkgconfig      2.0.3     2019-09-22 [1] CRAN (R 4.0.2)                 
   pkgload        1.1.0     2020-05-29 [1] CRAN (R 4.0.2)                 
   praise         1.0.0     2015-08-11 [1] CRAN (R 4.0.2)                 
   prettyunits    1.1.1     2020-01-24 [1] CRAN (R 4.0.2)                 
   processx       3.4.4     2020-09-03 [1] CRAN (R 4.0.2)                 
   ps             1.3.4     2020-08-11 [1] CRAN (R 4.0.2)                 
   R6             2.4.1     2019-11-12 [1] CRAN (R 4.0.2)                 
   RColorBrewer   1.1-2     2014-12-07 [1] CRAN (R 4.0.0)                 
   Rcpp           1.0.5     2020-07-06 [1] CRAN (R 4.0.2)                 
   RcppEigen      0.3.3.7.0 2019-11-16 [1] CRAN (R 4.0.2)                 
 D RcppParallel   5.0.2     2020-06-24 [1] CRAN (R 4.0.2)                 
   rlang          0.4.7     2020-07-09 [1] CRAN (R 4.0.2)                 
   rprojroot      1.3-2     2018-01-03 [1] CRAN (R 4.0.2)                 
   rstan        * 2.21.2    2020-07-27 [1] CRAN (R 4.0.2)                 
   rstudioapi     0.11      2020-02-07 [1] CRAN (R 4.0.2)                 
   scales         1.1.1     2020-05-11 [1] CRAN (R 4.0.2)                 
   StanHeaders  * 2.21.0-6  2020-08-16 [1] CRAN (R 4.0.2)                 
   testthat       2.3.2     2020-03-02 [1] CRAN (R 4.0.2)                 
   tibble         3.0.3     2020-07-10 [1] CRAN (R 4.0.2)                 
   utf8           1.1.4     2018-05-24 [1] CRAN (R 4.0.2)                 
   V8             3.2.0     2020-06-19 [1] CRAN (R 4.0.2)                 
   vctrs          0.3.4     2020-08-29 [1] CRAN (R 4.0.2)                 
   viridisLite    0.3.0     2018-02-01 [1] CRAN (R 4.0.2)                 
   withr          2.2.0     2020-04-20 [1] CRAN (R 4.0.2)                 

[1] C:/Users/dalto/Documents/R/win-library/4.0
[2] C:/Program Files/R/R-4.0.2/library

 D -- DLL MD5 mismatch, broken installation.

and running a stan model I still get

Warning messages:
1: In find.package(package, lib.loc, verbose = verbose) :
  package ‘base’ found more than once, using the first from
  “C:/PROGRA~1/R/R-40~1.2/library/base”,
  “C:/Program Files/R/R-4.0.2/library/base”
2: In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  'C:/RBUILD~1/4.0/usr/mingw_/bin/g++' not found

Thanks again for your help!

Chris_Dalton · September 8, 2020, 8:42am

But the models do compile and sample now with only this message!

Warning message:
In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  'C:/RBUILD~1/4.0/usr/mingw_/bin/g++' not found

Thanks!

andrjohns · September 8, 2020, 9:03am

Great, all of those warnings are safe to ignore, so you’re good to go

royworth · September 8, 2020, 5:20pm

For posterity, I thought I’d mention within this thread that I had every issue exactly as listed and tested by Chris above. Unfortunately, after following all of the suggestions, I still couldn’t get the example model to compile.

However, after going through these instructions and then also adding to my makevars file the line:

CXX14 = C:/Rtools40/mingw64/bin/g++

…everything worked out.

My makevars.win file now reads:

CXX14FLAGS += -O3 -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2
CXX14 = C:/Rtools40/mingw64/bin/g++

I took this advice from an older comment by @bgoodri here: G++ Compile Error With R Package

andrjohns · September 9, 2020, 12:11am

Hi Roy,

Just to check some reasons for why it wasn’t working initially (in case it’s a new issue I need to be aware of), can you post the outputs from:

Sys.getenv("BINPREF")
Sys.getenv("PATH")
Sys.which("make")
readLines("~/.Renviron")
readLines("~/.Rprofile")

royworth · September 9, 2020, 8:42am

Hi Andrew,

Thanks for the follow up. I admit I have little idea what is going on here and I was a little afraid I might unintentionally lead someone astray. Here is what I get running those commands:

> Sys.getenv("BINPREF")
[1] "C:/Rtools/mingw_$(WIN)/bin/"

> Sys.getenv("PATH")
[1] "C:\\rtools40\\usr\\bin;C:\\Program Files\\R\\R-4.0.2\\bin\\x64;C:\\Program Files (x86)\\Common Files\\Oracle\\Java\\javapath_target_325312;C:\\Windows\\System32;C:\\Windows;C:\\Windows\\System32\\wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0;C:\\Program Files (x86)\\Windows Live\\Shared;C:\\Program Files\\Git\\cmd;C:\\Windows\\System32\\OpenSSH;C:\\Program Files\\McAfee\\Solidcore\\Tools\\GatherInfo;C:\\Program Files\\McAfee\\Solidcore\\Tools\\ScAnalyzer;C:\\Program Files\\McAfee\\Solidcore;C:\\Program Files\\McAfee\\Solidcore\\Tools\\ScGetCerts;C:\\Users\\rmartin\\AppData\\Local\\Microsoft\\WindowsApps"

> Sys.which("make")
                              make 
"C:\\rtools40\\usr\\bin\\make.exe" 

> readLines("~/.Renviron")
[1] "PATH=\"${RTOOLS40_HOME}\\usr\\bin;${PATH}\""

> readLines("~/.Rprofile")
[1] "Sys.setenv(BINPREF = \"C:/Rtools/mingw_$(WIN)/bin/\")"

Also, not sure if relevant, but I run Stan on an intel Xeon workstation (2 processors), which is maybe different from the average user. In the past, I think I’ve needed to fiddle a bit with the makevars settings relative to the usual recommendations for those settings, but unfortunately I forget the details. For example, I’m not sure if that was because of the Xeon setup, or just the general ‘march=native’ issues.

Finally, in case it helps, here is the output of errors I get when I run

library(rstan)
example(stan_model,run.dontrun=TRUE,verbose=TRUE)

without adding the

CXX14 = C:/Rtools40/mingw64/bin/g++

to the makevars.win file.

Compilation ERROR, function(s)/method(s) not created!
Error in compileCode(f, code, language = language, verbose = verbose) : 
  sh: C:/Rtools/mingw_64/bin/g++: No such file or directorymake: *** [C:/PROGRA~1/R/R-40~1.2/etc/x64/Makeconf:229: file9ce4183c6852.o] Error 127
In addition: Warning messages:
1: In find.package(package, lib.loc, verbose = verbose) :
  package ‘base’ found more than once, using the first from
  “C:/PROGRA~1/R/R-40~1.2/library/base”,
  “C:/Program Files/R/R-4.0.2/library/base”
2: In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  'C:/rtools40/usr/mingw_/bin/g++' not found

Thanks again,

Roy

andrjohns · September 9, 2020, 9:02am

Hi Roy,

That’s an easy fix thankfully! You were getting those errors because you’ve set the BINPREF variable in your .Rprofile file.

If you follow the instructions I posted above to delete the file, you won’t need to set the CXX14 variable.

Cheers
Andrew

royworth · September 9, 2020, 9:18am

Ohhhh OK! I actually thought I did the delete the .Rprofile by running unlist("~/.Rprofile"), but I’ll fiddle around and try it again.Great to know what the underlying issue was, in any case!

-Roy

Namurava · September 25, 2020, 10:53pm

I had the same issue from above, and after switching withr to 2.2.0 and running:

example(stan_model,run.dontrun=TRUE,verbose=TRUE)

Warning messages:
1: In find.package(package, lib.loc, verbose = verbose) :
package ‘base’ found more than once, using the first from
“C:/Users/NMURAV~1/DOCUME~1/R/R-40~1.2/library/base”,
“C:/Users/nmuravskaya/Documents/R/R-4.0.2/library/base”
2: In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
‘C:/RBUILD~1/4.0/usr/mingw_/bin/g++’ not found

andrjohns · September 26, 2020, 2:21am

Hi Namurava,

None of those are errors that imply that the fitting failed. Were there any other errors or did the model finish running?