Continued compiling problems rstan 2.21.2 with R 4.0.3 and rtools40 (windows)

Interfaces RStan
,
1

Dear all,

I love Bayes & Stan immensely, and thank you for you work. This is my first time posting. Before I begin, let me say I’ve read and tried everything in the related threads by @bgoodri including the further problems experience and solved by @Antoine and @kartermycroft.

I had the same situation as @Antoine : updated to R 4.0.3, rtools40 and RStudio 1.3.1093, and then my models (including simple simulated ones to test this) would take a loooong time to compile and gave the same error (see below). My OS is Windows 10 Enterprise 2016 LTSB. I’m trying to run models through brms, but had the same problems using McElreath’s rethinking package.

Warning message:
In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  'C:/rtools40/usr/mingw_/bin/g++' not found

I followed @bgoodri’s detailed advice in the other threads, updating my Makevars.win. See below (I don’t think the incomplete final line is a problem?)

writeLines(readLines("C:/Program Files/R/R-4.0.3/src/library/windlgs/src/Makevars.win"))
CXX14FLAGS += -mtune=native -march=native -Wno-ignored-attributes -Wno-deprecated-declarations
CXX14 = C:\Rtools\mingw64\bin\g++ -m$(WIN)
Warning message:
In readLines("C:/Program Files/R/R-4.0.3/src/library/windlgs/src/Makevars.win") :
  incomplete final line found on 'C:/Program Files/R/R-4.0.3/src/library/windlgs/src/Makevars.win'

Notice above that my Makevars.win doesn’t live in a ~/.R folder (my computer doesn’t seem to have one of those).

I also tried simply deleting Makevars.win, and also, given that my rtools40 does not have have a mingw_ folder (it has a mingw32 and mingw64 folder), creating a copy of the mingw64 folder named mingw_; that didn’t change anything.

Reading @kartermycroft’s issues, I uninstalled R, rtools, Rstudio, removed all packages and vestiges of R including packages (except scripts, projects, and .rds files which are just raw data, they don’t compile anything), then reinstalled everything, and the problem was unchanged. The script I’m trying to debug with isn’t linked to any project or .rds files.

I thank you so so much in advance for your help…I’m excited to get back to the Bayes!

Graden

2 Likes
2

Hi Graden, great to hear you’re loving Stan, sorry you’ve hit this roadblock!

A key point here is that the warning:

Warning message:
In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  'C:/rtools40/usr/mingw_/bin/g++' not found

Is completely harmless and safe to ignore, it was just a misconfiguration that will be removed in the next release.

What I will recommend though is re-configuring your Makevars.win file. The .R folder is something that you need to create and then put your Makevars.win file. When you see a file path like ~/.R, the ~ is referring to your ‘Home’ directory, normally your Documents folder, but we can do all this automatically from R.

Additionally, your current specification will cause some issues, so we’ll want to use to different settings.

To create a Makevars.win file with the current best settings, you can just run the following in R:

writeLines("CXX14FLAGS += -mtune=native -O3 -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -Wno-ignored-attributes -Wno-deprecated-declarations -Wno-unused-function",
           con = "~/.R/Makevars.win")

Next, the latest version of the withr package is causing problems with the use of Makevars files on Windows, so you’ll need to downgrade (one version):

devtools::install_version("withr",version="2.2.0")

Can you run those two commands, and then check that your installation is working with:

library(rstan)
example(stan_model, run.dontrun=T)

Everything should run and finish sampling (but you’ll still get that warning message at the end)

1 Like
3

Hi Andrew,

Thank you so much for your help, it means a lot! I made* your suggested Makevars.win file (and also updated the old one in the other location) and downgraded withr.

The example model runs, but takes ~10 minutes, and my own models still take very long: a simple regression can now take over 1 hour, where as before it was a few minutes or less.

Is there anything else I should try?

*note to anyone else reading as inexperienced with in-the-weeds stuff as me, I had to first use the command line in windows to create a folder starting with a dot: mkdir .R

4

Hm, that is odd. When you say ‘before’, is this with an older RStan version or different R version?

5

Yes, sorry, I should have clarified. ‘Before’ meant before I updated: my R version would have been very old (probably the late 2s), and Rstan decently old but I don’t know exactly (unfortunately, I didn’t record what versions I was working with before updating everything - a mistake to correct next time).

6

Still strange for it to be taking so long. One thing we can check is how long it takes to compile some basic c++ (not stan-related).

Can you run the following commands and post the output and how long it takes:

system("touch foo.cpp")
system("R CMD SHLIB foo.cpp")
7 
> start <- Sys.time()
> system("touch foo.cpp")
[1] 0
> system("R CMD SHLIB foo.cpp")
"C:/rtools40/mingw64/bin/"g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-4.0.3/include" -DNDEBUG          -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign -c foo.cpp -o foo.o
C:/rtools40/mingw64/bin/g++ -std=gnu++11 -shared -s -static-libgcc -o foo.dll tmp.def foo.o -LC:/PROGRA~1/R/R-4.0.3/bin/x64 -lR
[1] 0
> end <- Sys.time()
> time <- end - start
> time
Time difference of 2.517258 mins

I just rechecked the example stan_model as well, and run time was 4.23542 mins.

8

Oh wow, ok so it looks there’s a non-Stan issue here. That example should take less than a second:

> start <- Sys.time()
> system("touch foo.cpp")
[1] 0
> system("R CMD SHLIB foo.cpp")
"C:/rtools40/mingw64/bin/"g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-40~1.3/include" -DNDEBUG          -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign -c foo.cpp -o foo.o
C:/rtools40/mingw64/bin/g++ -std=gnu++11 -shared -s -static-libgcc -o foo.dll tmp.def foo.o -LC:/PROGRA~1/R/R-40~1.3/bin/x64 -lR
[1] 0
> Sys.time() - start
Time difference of 0.8319819 secs

I see that you’re on an Enterprise edition, is this a work/university computer? If so, do you know if you’re working out of a networked drive or home directory?

A quick way to check would be to see where your home directory is mounted by running:

setwd("~")
getwd()
9

Oh no, here I’ve been pestering you for an issue that’s not even Stan! I am so appreciative for your help thus far, and please do feel free to go on your way!

It’s a university machine that I own, but I’ve been using Rtools without problems on it for 3 years. To troubleshoot I’m just working out of my home directory "C:/Users/my_name/Documents"; usually I’m working out of an Rproject in a folder linked to a private GitLab

10

That’s all good, it’s always helpful for me to know what the issues are, since there’s always a chance someone else will encounter the same thing.

Is your Documents folder being backed up via Dropbox/OneDrive or the like? That can also cause issues.

If that’s not the case, I’ll tag the main RStan dev (Ben Goodrich) to see if this something he’s run into before. If Ben can’t help, then you might have to get in touch with your IT department (if that’s an option) to see if they would have any insight on the issue.

@bgoodri This user is experiencing very slow c++ compilation and RStan model running times, when using Windows 10 (Enterprise), everything installed locally and RTools4. Is this something you’ve seen before?

3 Likes
11

Thanks so much Andrew. No backup happening. I’ll see what Ben says then reach out to my IT.

I guess it’s in the moment when something stops working that you realized how dearly you love it and took it for granted! Bayes will be so sweet once I’m back up and running.

3 Likes
12

Update: I tried downloaded a c++ compiler direct from MinGW, and pasting that folder (with all the g++ and associated applications) into rtool40 as the folder named mingw64. First line remained slow and second gave an error.

> #g++ downloaded from MinGW and pasted into rtool40 
> start <- Sys.time()
> system("touch foo.cpp")
[1] 0
> Sys.time() - start
Time difference of 1.252465 mins
> start <- Sys.time()
> system("R CMD SHLIB foo.cpp")
"C:/rtools40/mingw64/bin/"g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-4.0.3/include" -DNDEBUG          -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign -c foo.cpp -o foo.o
make: *** [C:/PROGRA~1/R/R-4.0.3/etc/x64/Makeconf:229: foo.o] Error 1
[1] 1
> Sys.time() - start
Time difference of 1.264654 mins

I tried 32-bit, no luck (ignorant grasping at straws haha).

Is this all a sign I need to transfer to windows subsystem to linux? It looks appealing https://blog.jdblischak.com/posts/wsl-r/, but I’m scared that given my utter lack of experience in all this I’d just find more ways to mess things up.

13

If you’re interested in going down the WSL route (definitely recommended for the increased model speed in general), a good approach is to install RStudio Server inside your WSL setup, so you can interact with it through a familiar RStudio interface. Their support page has a great guide on getting WSL installed and setting up RStudio: https://support.rstudio.com/hc/en-us/articles/360049776974-Using-RStudio-Server-in-Windows-WSL2

2 Likes
14

I too can highly recommend CmdStanR inside of WSL! There are a ton of resources that help you setting it up. I wrote a blog post which could be a little outdated but might still be helpful to you.

Cheers,
Max

1 Like
15

Hey @andrjohns I wanted to let you know that I finally sat down yesterday and fully removed Windows from my machine. I’m living the Linux life on Ubuntu now, and it is sweet. Models compiling and running faster than ever.

PS @jonah: during the last few weeks I’ve just been pushing scripts fitting models for team members to run and pulling down the objects to play with, and for that bayesplot has been such a useful joy. Thank you for this package and its documentation!

1 Like
16

Thanks @graden, really glad to hear that!