Can't get stan() to run with R 4.0.2 despite hours of messing with Makevars.win

JohnKruschke · October 8, 2020, 4:20pm

Greetings.

I’ve tried installing RStan etc. under R 4.0.2 and cannot get Stan to work.

I’ve updated the C++ toolchain as described at
https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started and
https://github.com/stan-dev/rstan/wiki/Installing-RStan-from-source-on-Windows#download-and-install-rtools.

I’ve read and tried the solutions in various threads about RStan with R 4.0 such as
Dealing with R 4.0,
Problems getting Stan to run after upgrading to R 4.0.2, and
New error: cleanup_makevar(old) : argument "RMU" is missing, with no default.

I’ve tried running with and without options:

options( mc.cores=parallel::detectCores() )
rstan_options( auto_write=TRUE )

But there’s always some (different) error and warnings after trying stan() regardless of what I try.

Here’s the most recent error message when I try running the simple schools model:

Restarting R session...

> library("rstan")
Loading required package: StanHeaders
Loading required package: ggplot2
rstan (Version 2.21.2, GitRev: 2e1f913d3ca3)
For execution on a local, multicore CPU with excess RAM we recommend calling
options(mc.cores = parallel::detectCores()).
To avoid recompilation of unchanged Stan programs, we recommend calling
rstan_options(auto_write = TRUE)
Do not specify '-march=native' in 'LOCAL_CPPFLAGS' or a Makevars file
> schools_dat <- list(J = 8, 
+                     y = c(28,  8, -3,  7, -1,  1, 18, 12),
+                     sigma = c(15, 10, 16, 11,  9, 11, 10, 18))
> fit <- stan(file = 'schools.stan', data = schools_dat)
make cmd is
  make -f "C:/PROGRA~1/R/R-40~1.2/etc/x64/Makeconf" -f "C:/PROGRA~1/R/R-40~1.2/share/make/winshlib.mk" CXX='$(CXX14) $(CXX14STD)' CXXFLAGS='$(CXX14FLAGS)' CXXPICFLAGS='$(CXX14PICFLAGS)' SHLIB_LDFLAGS='$(SHLIB_CXX14LDFLAGS)' SHLIB_LD='$(SHLIB_CXX14LD)' SHLIB="file4e3c432f2b87.dll" WIN=64 TCLBIN=64 OBJECTS="file4e3c432f2b87.o"

make would use
if test "zfile4e3c432f2b87.o" != "z"; then \
  if test -e "file4e3c432f2b87-win.def"; then \
    echo C:/Rtools/mingw_64/bin/g++  -shared -s -static-libgcc -o file4e3c432f2b87.dll file4e3c432f2b87-win.def file4e3c432f2b87.o   "C:/Users/John/Documents/R/win-library/4.0/rstan/lib/x64/libStanServices.a" -L"C:/Users/John/Documents/R/win-library/4.0/StanHeaders/libs/x64" -lStanHeaders -L"C:/Users/John/Documents/R/win-library/4.0/RcppParallel/lib/x64" -ltbb  -L"C:/PROGRA~1/R/R-40~1.2/bin/x64" -lR ; \
    C:/Rtools/mingw_64/bin/g++  -shared -s -static-libgcc -o file4e3c432f2b87.dll file4e3c432f2b87-win.def file4e3c432f2b87.o   "C:/Users/John/Documents/R/win-library/4.0/rstan/lib/x64/libStanServices.a" -L"C:/Users/John/Documents/R/win-library/4.0/StanHeaders/libs/x64" -lStanHeaders -L"C:/Users/John/Documents/R/win-library/4.0/RcppParallel/lib/x64" -ltbb  -L"C:/PROGRA~1/R/R-40~1.2/bin/x64" -lR ; \
  else \
    echo EXPORTS > tmp.def; \
    C:/Rtools/mingw_64/bin/nm file4e3c432f2b87.o | sed -n 's/^.* [BCDRT] / /p' | sed -e '/[.]refptr[.]/d' -e '/[.]weak[.]/d' | sed 's/[^ ][^ ]*/"&"/g'  >> tmp.def; \
    echo C:/Rtools/mingw_64/bin/g++  -shared -s -static-libgcc -o file4e3c432f2b87.dll tmp.def file4e3c432f2b87.o   "C:/Users/John/Documents/R/win-library/4.0/rstan/lib/x64/libStanServices.a" -L"C:/Users/John/Documents/R/win-library/4.0/StanHeaders/libs/x64" -lStanHeaders -L"C:/Users/John/Documents/R/win-library/4.0/RcppParallel/lib/x64" -ltbb  -L"C:/PROGRA~1/R/R-40~1.2/bin/x64" -lR ; \
    C:/Rtools/mingw_64/bin/g++  -shared -s -static-libgcc -o file4e3c432f2b87.dll tmp.def file4e3c432f2b87.o   "C:/Users/John/Documents/R/win-library/4.0/rstan/lib/x64/libStanServices.a" -L"C:/Users/John/Documents/R/win-library/4.0/StanHeaders/libs/x64" -lStanHeaders -L"C:/Users/John/Documents/R/win-library/4.0/RcppParallel/lib/x64" -ltbb  -L"C:/PROGRA~1/R/R-40~1.2/bin/x64" -lR ; \
    rm -f tmp.def; \
  fi \
fi
Error in compileCode(f, code, language = language, verbose = verbose) : 
  C:/Users/John/Documents/R/win-library/4.0/rstan/lib/x64/libStanServices.a(stan_fit.o):stan_fit.cpp:(.text$_ZN4stan8services4util20run_adaptive_samplerINS_4mcmc24adapt_dense_e_static_hmcINS_5model10model_baseEN5boost6random23additive_combine_engineINS8_26linear_congruential_engineIjLj40014ELj0ELj2147483563EEENSA_IjLj40692ELj0ELj2147483399EEEEEEES6_SD_EEvRT_RT0_RSt6vectorIdSaIdEEiiiibRT1_RNS_9callbacks9interruptERNSP_6loggerERNSP_6writerESV_[_ZN4stan8services4util20run_adaptive_samplerINS_4mcmc24adapt_dense_e_static_hmcINS_5model10model_baseEN5boost6random23additive_combine_engineINS8_26linear_congruential_engineIjLj40014ELj0ELj2147483563EEENSA_IjLj40692ELj0ELj2147483399EEEEEEES6_SD_EEvRT_RT0_RSt6vectorIdSaIdEEiiiibRT1_RNS_9callbacks9interruptERNSP_6loggerERNSP_6writerESV_]+0x55c): undefined reference to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_stringstream()'C:/Users/John/Documents/R/win-library/4.0/rstan/lib/x64/libStanSer
In addition: Warning message:
In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  'C:/rtools40/usr/mingw_/bin/g++' not found
Error in sink(type = "output") : invalid connection
>

Below is information about the configuration of my computer system:

Operating System

version  R version 4.0.2 (2020-06-22)
 os       Windows 10 x64              
 system   x86_64, mingw32             
 ui       RStudio

RStan Version

rstan        * 2.21.2    2020-07-27 [1] CRAN (R 4.0.2)

Output of writeLines(readLines(file.path(Sys.getenv("HOME"), ".R/Makevars")))

Makevars is not present because I removed it, based on advice from most recent post on this discussion forum (which I can’t repeat here because the interface will only let me post 5 links).

> writeLines(readLines(file.path(Sys.getenv("HOME"), ".R/Makevars")))
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
  cannot open file 'C:\Users\John\Documents/.R/Makevars': No such file or directory

Output of devtools::session_info("rstan")

> devtools::session_info("rstan")
- Session info -------------------------------------------------------------------------
 setting  value                       
 version  R version 4.0.2 (2020-06-22)
 os       Windows 10 x64              
 system   x86_64, mingw32             
 ui       RStudio                     
 language (EN)                        
 collate  English_United States.1252  
 ctype    English_United States.1252  
 tz       America/Indianapolis        
 date     2020-10-08                  

- Packages -----------------------------------------------------------------------------
 ! package      * version   date       lib source        
   assertthat     0.2.1     2019-03-21 [1] CRAN (R 4.0.0)
   backports      1.1.10    2020-09-15 [1] CRAN (R 4.0.2)
   BH             1.72.0-3  2020-01-08 [1] CRAN (R 4.0.0)
   callr          3.4.4     2020-09-07 [1] CRAN (R 4.0.2)
   checkmate      2.0.0     2020-02-06 [1] CRAN (R 4.0.2)
   cli            2.0.2     2020-02-28 [1] CRAN (R 4.0.0)
   colorspace     1.4-1     2019-03-18 [1] CRAN (R 4.0.0)
   crayon         1.3.4     2017-09-16 [1] CRAN (R 4.0.0)
   curl           4.3       2019-12-02 [1] CRAN (R 4.0.0)
   desc           1.2.0     2018-05-01 [1] CRAN (R 4.0.0)
   digest         0.6.25    2020-02-23 [1] CRAN (R 4.0.0)
   ellipsis       0.3.1     2020-05-15 [1] CRAN (R 4.0.0)
   evaluate       0.14      2019-05-28 [1] CRAN (R 4.0.0)
   fansi          0.4.1     2020-01-08 [1] CRAN (R 4.0.0)
   farver         2.0.3     2020-01-16 [1] CRAN (R 4.0.0)
   ggplot2      * 3.3.2     2020-06-19 [1] CRAN (R 4.0.2)
   glue           1.4.2     2020-08-27 [1] CRAN (R 4.0.2)
   gridExtra      2.3       2017-09-09 [1] CRAN (R 4.0.2)
   gtable         0.3.0     2019-03-25 [1] CRAN (R 4.0.0)
   inline         0.3.16    2020-09-06 [1] CRAN (R 4.0.2)
   isoband        0.2.2     2020-06-20 [1] CRAN (R 4.0.2)
   jsonlite       1.7.1     2020-09-07 [1] CRAN (R 4.0.2)
   labeling       0.3       2014-08-23 [1] CRAN (R 4.0.0)
   lattice        0.20-41   2020-04-02 [2] CRAN (R 4.0.2)
   lifecycle      0.2.0     2020-03-06 [1] CRAN (R 4.0.0)
   loo            2.3.1     2020-07-14 [1] CRAN (R 4.0.2)
   magrittr       1.5       2014-11-22 [1] CRAN (R 4.0.0)
   MASS           7.3-51.6  2020-04-26 [2] CRAN (R 4.0.2)
   Matrix         1.2-18    2019-11-27 [2] CRAN (R 4.0.2)
   matrixStats    0.57.0    2020-09-25 [1] CRAN (R 4.0.2)
   mgcv           1.8-31    2019-11-09 [2] CRAN (R 4.0.2)
   munsell        0.5.0     2018-06-12 [1] CRAN (R 4.0.0)
   nlme           3.1-148   2020-05-24 [2] CRAN (R 4.0.2)
   pillar         1.4.6     2020-07-10 [1] CRAN (R 4.0.2)
   pkgbuild       1.1.0     2020-07-13 [1] CRAN (R 4.0.2)
   pkgconfig      2.0.3     2019-09-22 [1] CRAN (R 4.0.0)
   pkgload        1.1.0     2020-05-29 [1] CRAN (R 4.0.0)
   praise         1.0.0     2015-08-11 [1] CRAN (R 4.0.0)
   prettyunits    1.1.1     2020-01-24 [1] CRAN (R 4.0.0)
   processx       3.4.4     2020-09-03 [1] CRAN (R 4.0.2)
   ps             1.3.4     2020-08-11 [1] CRAN (R 4.0.2)
   R6             2.4.1     2019-11-12 [1] CRAN (R 4.0.0)
   RColorBrewer   1.1-2     2014-12-07 [1] CRAN (R 4.0.0)
   Rcpp           1.0.5     2020-07-06 [1] CRAN (R 4.0.2)
   RcppEigen      0.3.3.7.0 2019-11-16 [1] CRAN (R 4.0.2)
 D RcppParallel   5.0.2     2020-06-24 [1] CRAN (R 4.0.2)
   rlang          0.4.7     2020-07-09 [1] CRAN (R 4.0.2)
   rprojroot      1.3-2     2018-01-03 [1] CRAN (R 4.0.0)
   rstan        * 2.21.2    2020-07-27 [1] CRAN (R 4.0.2)
   rstudioapi     0.11      2020-02-07 [1] CRAN (R 4.0.0)
   scales         1.1.1     2020-05-11 [1] CRAN (R 4.0.0)
   StanHeaders  * 2.21.0-6  2020-08-16 [1] CRAN (R 4.0.2)
   testthat       2.3.2     2020-03-02 [1] CRAN (R 4.0.0)
   tibble         3.0.3     2020-07-10 [1] CRAN (R 4.0.2)
   utf8           1.1.4     2018-05-24 [1] CRAN (R 4.0.0)
   V8             3.2.0     2020-06-19 [1] CRAN (R 4.0.2)
   vctrs          0.3.4     2020-08-29 [1] CRAN (R 4.0.2)
   viridisLite    0.3.0     2018-02-01 [1] CRAN (R 4.0.0)
   withr          2.3.0     2020-09-22 [1] CRAN (R 4.0.2)

[1] C:/Users/John/Documents/R/win-library/4.0
[2] C:/Program Files/R/R-4.0.2/library

 D -- DLL MD5 mismatch, broken installation.

Yes, I noticed the “broken installation” of RcppParallel, but am not sure what to do about it, and not sure if it’s a red herring because I get an error even when not trying to run stan on parallel cores as shown above.

Thank you!

andrjohns · October 8, 2020, 4:38pm

You can see the issue in this line:

echo C:/Rtools/mingw_64/bin/g++  -shared

Which means that R is trying to use RTools 3.5 with R4.0. So the first step is to make sure that you have RTools4 installed.

If you do, to troubleshoot why it’s not being picked up, can you post the outputs from:

Sys.getenv("PATH")
Sys.getenv("BINPREF")
readLines("~/.R/Makevars.win")
readLines("~/.Rprofile")
readLines("~/.Renviron")

JohnKruschke · October 8, 2020, 4:53pm

Thank you for your quick reply.

It made me smile when you said "You can see the issue in this line: echo C:/Rtools/mingw_64/bin/g++ -shared" because, of course, I do not have the expertise to see anything meaningful there. I’m very glad you can! :-)

I definitely have RTools4 installed. Did that before, based on https://cran.r-project.org/bin/windows/Rtools/

Here are the outputs of the commands you requested:

> Sys.getenv("PATH")
[1] "
C:\\rtools40\\usr\\bin;
C:\\Program Files\\R\\R-4.0.2\\bin\\x64;
C:\\Rtools\\bin;
C:\\Rtools\\bin;
C:\\Rtools\\mingw_32\\bin;
..."

> Sys.getenv("BINPREF")
[1] "C:/Rtools/mingw_$(WIN)/bin/"

> readLines("~/.R/Makevars.win")
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
  cannot open file 'C:/Users/John/Documents/.R/Makevars.win': No such file or directory

I removed Makevars.win based on advice in New error: cleanup_makevar(old) : argument "RMU" is missing, with no default

> readLines("~/.Rprofile")
[1] "Sys.setenv(BINPREF = \"C:/Rtools/mingw_$(WIN)/bin/\")"
[2] "Sys.setenv(BINPREF = \"C:/Rtools/mingw_$(WIN)/bin/\")"

> readLines("~/.Renviron")
[1] "PATH=\"${RTOOLS40_HOME}\\usr\\bin;${PATH}\""

andrjohns · October 8, 2020, 4:58pm

Happy to have the answer! The issue here is that the .Rprofile file is setting the BINPREF environment variable which then tells R to use the RTools3.5 compilers. Try deleting the .Rprofile file (should be in your Documents folder), then restart R and try your model again.

jonah · October 8, 2020, 5:20pm

@andrjohns @bgoodri Is this a problem that you’ve seen before? Can we add something to the installation instructions to warn them to look out for this?

andrjohns · October 8, 2020, 5:23pm

Yeah it seems to be the main issue people are running into when upgrading from R3.6 -> R4.0, since the .Rprofile configuration is part of the install instructions for 3.6.

A warning would be a great idea!

jonah · October 8, 2020, 5:25pm

Do you mean part of our RStan install instructions for people using 3.6 or part of R’s install instructions for 3.6?

andrjohns · October 8, 2020, 5:32pm

An earlier version of the install instructions (don’t think its there any more) included the configuration step:

cat('Sys.setenv(BINPREF = "C:/Rtools/mingw_$(WIN)/bin/")',
    file = file.path(Sys.getenv("HOME"), ".Rprofile"), 
    sep = "\n", append = TRUE)

So there are a bunch of users that still have this config floating around

EDIT: The RStan install instructions, I mean

jonah · October 8, 2020, 5:40pm

andrjohns:

An earlier version of the install instructions (don’t think its there any more) included the configuration step:
cat('Sys.setenv(BINPREF = "C:/Rtools/mingw_$(WIN)/bin/")',
    file = file.path(Sys.getenv("HOME"), ".Rprofile"), 
    sep = "\n", append = TRUE)
So there are a bunch of users that still have this config floating around

Good point. @bgoodri going forward can we be extra careful not only to update installation instructions but also to mention problems that previous install instructions may cause? If I were a windows user I wouldn’t expect my previous RStan installation steps to break my current RStan installation, at least not without warning.

JohnKruschke · October 8, 2020, 5:50pm

As instructed, I deleted .Rprofile from Documents folder, restarted R, tried stan()again. (Still with no Makevars.win file, and with no options set after library("rstan").) The result …drum roll…

stan() runs and returns an object. Yay!

However, it does give this warning:

Warning message:
In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  'C:/rtools40/usr/mingw_/bin/g++' not found

which seems like something that should be addressed even if it doesn’t cause processing to halt?

Thanks again.

jonah · October 8, 2020, 5:55pm

Glad it’s working and sorry for the hassle! And yeah that warning is unfortunate. It’s super annoying but it can be ignored. I believe it’s fixed already in the development version and should be gone in the next release.

jonah · October 8, 2020, 6:01pm

@andrjohns or @bgoodri can correct me if I’m wrong (I’m not a Windows user) but now that Stan is running for you I think there are a few things you can add back to the Makevars.win file to speed things up. At a minimum I think CXX14FLAGS=-O3. @andrjohns Anything else?

JohnKruschke · October 8, 2020, 6:01pm

Since y’all are discussing installation instructions in this thread, I thought I’d point out that it would also solve many headaches to update https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started#checking-the-c-toolchain for updating to RTools4, and update https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started#configuration-of-the-c-toolchain so it’s appropriate for R 4.0.x. Thank you again!

bgoodri · October 8, 2020, 6:07pm

I use

CXX14FLAGS = -O3 -g0 -Wno-ignored-attributes -Wno-deprecated-declarations

on Windows.

jonah · October 8, 2020, 6:22pm

Thanks @bgoodri. Is this particular configuration recommended somewhere?

jonah · October 8, 2020, 6:27pm

Yeah if those links have incorrect instructions we need to update them ASAP!

@bgoodri or @andrjohns or someone else who understands the current Windows situation, can you take a look and update those pages if necessary? Thank you!

Also what can we do going forward to keep all these installation instructions, recommended Makevars configurations, etc., up to date? Is the problem that the best practices are changing too quickly or that we just don’t have anyone making sure everything is current?

szuniga07 · December 22, 2020, 2:51am

Loved your book, Doing Bayesian Data Analysis, 2nd Ed. Partially how I first made it to Stan.org. 100% how I now have a coffee mug with puppies on it…I don’t even drink coffee! I had a similar issue, glad to see that the great team at Stan helped you out, too. Happy new year!

JohnKruschke · December 22, 2020, 1:31pm

Thanks very much for your kind remarks, Steve. I’m glad that DBDA2E served you well, and that Bayesian puppies grace your coffee mug. If anyone else out there likes DBDA2E and Stan, take a look at Solomon Kurz’s re-casting much of the book in brms (a package for Stan) and the tidyverse. Thanks again to you and all the folks at Stan.org. Happy New Year to all!

jonah · December 22, 2020, 6:58pm

Thanks John. Same to you!

strengejacke · January 3, 2021, 9:55pm

Keeping only this line still results in the error

Error in cleanup_makevar(old) :
argument “RMU” is missing, with no default

Currently, I renamed makevars.win so it’s not loaded, which at least makes brms compile models.

BTW, I also have

CXX14FLAGS = -O3 -g0 -Wno-ignored-attributes -Wno-deprecated-declarations

in my makevars.win, I think this must have been part of the Stan installation instructions somewhere, since I did not manually specify this.