Running rstan on university remote virtual lab

Interfaces RStan
1

I’ve got a hierarchical model that takes a long time to run that I need to run several times on different data (the longest version is ~8 hours and makes my laptop otherwise unusable).

I do have access to a virtual remote lab with RStudio via my university, but I can’t get rstan running. I suspect it has to do with a mismatch in the library locations. Here’s the code I’m running

library(brms)

bre10 <- sthfc[sthfc$e_l10 == "e" & !is.na(sthfc$e_l10),]

formula_logit2 <- bf(correct_log ~ exp(logalpha) * eta,
                     eta ~ 1 + time_rem + (0 + time_rem |i| discover_id) + (1 | user_id),
                     logalpha ~ 1 + (1 |i| discover_id),
                     nl = TRUE)

prior_log <- prior("normal(0, 5)", class = "b", nlpar = "eta") +
  prior("normal(0, 1)", class = "b", nlpar = "logalpha") +
  prior("constant(1)", class = "sd", group = "user_id", nlpar = "eta") +
  prior("normal(0, 3)", class = "sd", group = "discover_id", nlpar = "eta") +
  prior("normal(0, 1)", class = "sd", group = "discover_id", nlpar = "logalpha")


brm_log10 <- brm(formula_logit2, data = bre10,
                 family = gaussian(),
                 prior = prior_log,
                 iter = 3000)

And the error I get is

Compiling Stan program...
make cmd is
  make -f "C:/PROGRA~1/R/R-40~1.2/etc/x64/Makeconf" -f "C:/PROGRA~1/R/R-40~1.2/share/make/winshlib.mk" -f "C:\Users\MHIMME~1\AppData\Local\Temp\230\RtmpKCXL0S\Makevarsaa482c6e410b" CXX='$(CXX14) $(CXX14STD)' CXXFLAGS='$(CXX14FLAGS)' CXXPICFLAGS='$(CXX14PICFLAGS)' SHLIB_LDFLAGS='$(SHLIB_CXX14LDFLAGS)' SHLIB_LD='$(SHLIB_CXX14LD)' SHLIB="fileaa483e5918c9.dll" WIN=64 TCLBIN=64 OBJECTS="fileaa483e5918c9.o"

make would use
Warning message:
In system(paste(cmd, "-n")) :
  'CreateProcess' failed to run 'C:\rtools40\usr\bin\make.exe -f "C:/PROGRA~1/R/R-40~1.2/etc/x64/Makeconf" -f "C:/PROGRA~1/R/R-40~1.2/share/make/winshlib.mk" -f "C:\Users\MHIMME~1\AppData\Local\Temp\230\RtmpKCXL0S\Makevarsaa482c6e410b" CXX='$(CXX14) $(CXX14STD)' CXXFLAGS='$(CXX14FLAGS)' CXXPICFLAGS='$(CXX14PICFLAGS)' SHLIB_LDFLAGS='$(SHLIB_CXX14LDFLAGS)' SHLIB_LD='$(SHLIB_CXX14LD)' SHLIB="fileaa483e5918c9.dll" WIN=64 TCLBIN=64 OBJECTS="fileaa483e5918c9.o" -n'

I also ran

devtools::session_info("rstan")

and get

- Session info -------------------------------------------------
 setting  value                       
 version  R version 4.0.2 (2020-06-22)
 os       Windows Server x64          
 system   x86_64, mingw32             
 ui       RStudio                     
 language (EN)                        
 collate  English_United States.1252  
 ctype    English_United States.1252  
 tz       America/New_York            
 date     2020-09-07                  

- Packages -----------------------------------------------------
 ! package      * version   date       lib
   assertthat     0.2.1     2019-03-21 [1]
   backports      1.1.9     2020-08-24 [1]
   BH             1.72.0-3  2020-01-08 [1]
   callr          3.4.3     2020-03-28 [1]
   checkmate      2.0.0     2020-02-06 [1]
   cli            2.0.2     2020-02-28 [1]
   colorspace     1.4-1     2019-03-18 [1]
   crayon         1.3.4     2017-09-16 [1]
   curl           4.3       2019-12-02 [1]
   desc           1.2.0     2018-05-01 [1]
   digest         0.6.25    2020-02-23 [1]
   ellipsis       0.3.1     2020-05-15 [1]
   evaluate       0.14      2019-05-28 [1]
   fansi          0.4.1     2020-01-08 [1]
   farver         2.0.3     2020-01-16 [1]
   ggplot2        3.3.2     2020-06-19 [1]
   glue           1.4.2     2020-08-27 [1]
   gridExtra      2.3       2017-09-09 [1]
   gtable         0.3.0     2019-03-25 [1]
   inline         0.3.15    2020-09-07 [1]
   isoband        0.2.2     2020-06-20 [1]
   jsonlite       1.7.0     2020-06-25 [1]
   labeling       0.3       2014-08-23 [1]
   lattice        0.20-41   2020-04-02 [2]
   lifecycle      0.2.0     2020-03-06 [1]
   loo            2.3.1     2020-07-14 [1]
   magrittr       1.5       2014-11-22 [1]
   MASS           7.3-51.6  2020-04-26 [2]
   Matrix         1.2-18    2019-11-27 [2]
   matrixStats    0.56.0    2020-03-13 [1]
   mgcv           1.8-31    2019-11-09 [2]
   munsell        0.5.0     2018-06-12 [1]
   nlme           3.1-148   2020-05-24 [2]
   pillar         1.4.6     2020-07-10 [1]
   pkgbuild       1.1.0     2020-07-13 [1]
   pkgconfig      2.0.3     2019-09-22 [1]
   pkgload        1.1.0     2020-05-29 [1]
   praise         1.0.0     2015-08-11 [1]
   prettyunits    1.1.1     2020-01-24 [1]
   processx       3.4.4     2020-09-03 [1]
   ps             1.3.4     2020-08-11 [1]
   R6             2.4.1     2019-11-12 [1]
   RColorBrewer   1.1-2     2014-12-07 [1]
   Rcpp         * 1.0.5     2020-07-06 [1]
   RcppEigen      0.3.3.7.0 2019-11-16 [1]
 D RcppParallel   5.0.2     2020-06-24 [1]
   rlang          0.4.7     2020-07-09 [1]
   rprojroot      1.3-2     2018-01-03 [1]
   rstan          2.21.2    2020-07-27 [1]
   rstudioapi     0.11      2020-02-07 [1]
   scales         1.1.1     2020-05-11 [1]
   StanHeaders    2.21.0-6  2020-08-16 [1]
   testthat       2.3.2     2020-03-02 [1]
   tibble         3.0.3     2020-07-10 [1]
   utf8           1.1.4     2018-05-24 [1]
   V8             3.2.0     2020-06-19 [1]
   vctrs          0.3.4     2020-08-29 [1]
   viridisLite    0.3.0     2018-02-01 [1]
   withr          2.2.0     2020-04-20 [1]
 source                         
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.0)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 Github (bgoodri/inline@21105c7)
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.0)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.0)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 
 CRAN (R 4.0.2)                 

[1] \\NV-DC1/DFS$/Root/FORDHAM/Users/mhimmelste_fordham/Documents/R/win-library/4.0
[2] C:/Program Files/R/R-4.0.2/library

 D -- DLL MD5 mismatch, broken installation.

Which seems like it might indicate the problem, though I still don’t know what to do about it.

Any help would be much appreciated.

2

Do you know whether Rtools4 is installed on that server? You can check if R can find it via:

pkgbuild::has_rtools()
3

It appears so, running that line returns TRUE.

My hunch was this was related to the issue, which was from the end of running

devtools::session_info(“rstan”)

[1] \\NV-DC1/DFS$/Root/FORDHAM/Users/mhimmelste_fordham/Documents/R/win-library/4.0
[2] C:/Program Files/R/R-4.0.2/library

 D -- DLL MD5 mismatch, broken installation.`

But I could be totally wrong

I also tried Ben’s suggestion from this post

But no luck

4

Alright, the next steps are to check some configurations:

Sys.which("make")
Sys.getenv("PATH")

And try to compile some C++ using just the compiler through R:

system("touch foo.cpp")
system("R CMD SHLIB foo.cpp")

And using the inline package:

fx <- inline::cxxfunction( signature(x = "integer", y = "numeric" ) , 'return ScalarReal( INTEGER(x)[0] * REAL(y)[0] ) ;' )
5 
Sys.which("make")

appears to be empty (just returns “”)

Sys.getenv("PATH")
[1] "C:\\Program Files\\R\\R-4.0.2\\bin\\x64;C:\\Program Files\\Python38\\Scripts\\;C:\\Program Files\\Python38\\;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;C:\\Program Files\\Amazon\\cfn-bootstrap\\;C:\\Program Files\\ESET\\ESET Security\\;C:\\Program Files (x86)\\PharosSystems\\Core;C:\\Program Files (x86)\\Microsoft SQL Server\\150\\DTS\\Binn\\;C:\\Program Files\\SASHome\\SASFoundation\\9.4\\ets\\sasexe;C:\\Program Files\\SASHome\\Secure\\ccme4;C:\\Program Files\\SASHome\\x86\\Secure\\ccme4;C:\\Program Files\\Microsoft SQL Server\\Client SDK\\ODBC\\170\\Tools\\Binn\\;C:\\Program Files (x86)\\Microsoft SQL Server\\150\\Tools\\Binn\\;C:\\Program Files\\Microsoft SQL Server\\150\\Tools\\Binn\\;C:\\Program Files\\Microsoft SQL Server\\150\\DTS\\Binn\\;C:\\Program Files\\PuTTY\\;C:\\Users\\mhimmelste_fordham\\AppData\\Local\\Microsoft\\WindowsApps;"

system("touch foo.cpp")
[1] 127
> system("R CMD SHLIB foo.cpp")
'\\apporto.com\dfs\FORDHAM\Users\mhimmelste_fordham\Documents'
CMD.EXE was started with the above path as the current directory.
UNC paths are not supported.  Defaulting to Windows directory.
Warning message:
In system(cmd) : 'make' not found
[1] 1

So it looks like something is up with ‘make’ being missing?


> fx <- inline::cxxfunction( signature(x = "integer", y = "numeric" ) , 'return ScalarReal( INTEGER(x)[0] * REAL(y)[0] ) ;' )

ERROR(s) during compilation: source code errors or compiler configuration errors!
make cmd is
  make -f "C:/PROGRA~1/R/R-40~1.2/etc/x64/Makeconf" -f "C:/PROGRA~1/R/R-40~1.2/share/make/winshlib.mk" -f "//apporto.com/dfs/FORDHAM/Users/mhimmelste_fordham/Documents/.R/Makevars.win" SHLIB_LDFLAGS='$(SHLIB_CXXLDFLAGS)' SHLIB_LD='$(SHLIB_CXXLD)' SHLIB="file2fa429f13a20.dll" WIN=64 TCLBIN=64 OBJECTS="file2fa429f13a20.o"

make would use
Warning message:
In system(paste(cmd, "-n")) : 'make' not found

Program source:
  1: 
  2: // includes from the plugin
  3: #include <R.h>
  4: #include <Rdefines.h>
  5: #include <R_ext/Error.h>
  6: 
  7: 
  8: // user includes
  9: 
 10: 
 11: // declarations
 12: extern "C" {
 13: SEXP file2fa429f13a20( SEXP x, SEXP y) ;
 14: }
 15: 
 16: // definition
 17: 
 18: SEXP file2fa429f13a20( SEXP x, SEXP y ){
 19: return ScalarReal( INTEGER(x)[0] * REAL(y)[0] ) ; 
 20: Rf_warning("your C++ program does not return anything"); 
 21:  return R_NilValue ; 
 22: }
 23: 
 24: 

Compilation ERROR, function(s)/method(s) not created!
Error in compileCode(f, code, language = language, verbose = verbose) : 
  Warning message:In system(cmd) : 'make' not found

Same warning message about ‘make’

Does that help?

6

Great! This looks like it might be an easy fix. You just need to add the RTools directory to the PATH environment variable so that R knows where to look for the compiler.

Try running:

writeLines('PATH="${RTOOLS40_HOME}\\usr\\bin;${PATH}"', con = "~/.Renviron")

Then restart the R session

And try Sys.which("make") again

7

So I did that, and now make shows as

"C:\\rtools40\\usr\\bin\\make.exe"

But it still doesn’t seem like it’s being found when I try to compile

For example, when I run

 
> system("R CMD SHLIB foo.cpp")
'\\apporto.com\dfs\FORDHAM\Users\mhimmelste_fordham\Documents'
CMD.EXE was started with the above path as the current directory.
UNC paths are not supported.  Defaulting to Windows directory.
Warning message:
In system(cmd) :
  'CreateProcess' failed to run 'C:\rtools40\usr\bin\make.exe -f "C:/PROGRA~1/R/R-40~1.2/etc/x64/Makeconf" -f "C:/PROGRA~1/R/R-40~1.2/share/make/winshlib.mk" -f "//apporto.com/dfs/FORDHAM/Users/mhimmelste_fordham/Documents/.R/Makevars.win" SHLIB_LDFLAGS='$(SHLIB_CXXLDFLAGS)' SHLIB_LD='$(SHLIB_CXXLD)' SHLIB="foo.dll" WIN=64 TCLBIN=64 OBJECTS="foo.o"'
[1] 1
>

So the warning message is different, now its “failed to run” instead of “not found”

8

What happens with the example model:

library(rstan)
example(stan_model,run.dontrun=TRUE,verbose=TRUE)
9

If you need it I can post the rest of the output but it hits a similar looking error after line 372


372: 

Compilation ERROR, function(s)/method(s) not created!
Error in compileCode(f, code, language = language, verbose = verbose) : 
  Warning message:In system(cmd) :  'CreateProcess' failed to run 'C:\rtools40\usr\bin\make.exe -f "C:/PROGRA~1/R/R-40~1.2/etc/x64/Makeconf" -f "C:/PROGRA~1/R/R-40~1.2/share/make/winshlib.mk" -f "C:\Users\MHIMME~1\AppData\Local\Temp\115\RtmpcrSaNL\Makevars87a052935b1c" CXX='$(CXX14) $(CXX14STD)' CXXFLAGS='$(CXX14FLAGS)' CXXPICFLAGS='$(CXX14PICFLAGS)' SHLIB_LDFLAGS='$(SHLIB_CXX14LDFLAGS)' SHLIB_LD='$(SHLIB_CXX14LD)' SHLIB="file87a022a07e78.dll" WIN=64 TCLBIN=64 OBJECTS="file87a022a07e78.o"'
In addition: Warning messages:
1: In find.package(package, lib.loc, verbose = verbose) :
  package ‘base’ found more than once, using the first from
  “C:/PROGRA~1/R/R-40~1.2/library/base”,
  “C:/Program Files/R/R-4.0.2/library/base”
2: In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  'C:/rtools40/usr/mingw_/bin/g++' not found
10

Can you try updating the inline package to 0.3.16 and trying again? You can do this manually via:

install.packages("https://cran.r-project.org/bin/windows/contrib/4.1/inline_0.3.16.zip",
                 type="source",repos=NULL)
11

Looks the same (I tried removing and reinstalling rstan as well and then doing it again and it seems the same no matter what)

372: 

Compilation ERROR, function(s)/method(s) not created!
Error in compileCode(f, code, language = language, verbose = verbose) : 
  Warning message:In system(cmd) :  'CreateProcess' failed to run 'C:\rtools40\usr\bin\make.exe -f "C:/PROGRA~1/R/R-40~1.2/etc/x64/Makeconf" -f "C:/PROGRA~1/R/R-40~1.2/share/make/winshlib.mk" -f "C:\Users\MHIMME~1\AppData\Local\Temp\115\RtmpaU8LIP\Makevars7aec6ed341da" CXX='$(CXX14) $(CXX14STD)' CXXFLAGS='$(CXX14FLAGS)' CXXPICFLAGS='$(CXX14PICFLAGS)' SHLIB_LDFLAGS='$(SHLIB_CXX14LDFLAGS)' SHLIB_LD='$(SHLIB_CXX14LD)' SHLIB="file7aec76da3a5b.dll" WIN=64 TCLBIN=64 OBJECTS="file7aec76da3a5b.o"'
In addition: Warning messages:
1: In find.package(package, lib.loc, verbose = verbose) :
  package ‘base’ found more than once, using the first from
  “C:/PROGRA~1/R/R-40~1.2/library/base”,
  “C:/Program Files/R/R-4.0.2/library/base”
2: In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  'C:/rtools40/usr/mingw_/bin/g++' not found
12

And you get the same error with compilation via the inline package?:

fx <- inline::cxxfunction( signature(x = "integer", y = "numeric" ) , 'return ScalarReal( INTEGER(x)[0] * REAL(y)[0] ) ;' )
13

Yeah looks the same when I run that


Compilation ERROR, function(s)/method(s) not created!
Error in compileCode(f, code, language = language, verbose = verbose) : 
  Warning message:In system(cmd) :  'CreateProcess' failed to run 'C:\rtools40\usr\bin\make.exe -f "C:/PROGRA~1/R/R-40~1.2/etc/x64/Makeconf" -f "C:/PROGRA~1/R/R-40~1.2/share/make/winshlib.mk" -f "//apporto.com/dfs/FORDHAM/Users/mhimmelste_fordham/Documents/.R/Makevars.win" SHLIB_LDFLAGS='$(SHLIB_CXXLDFLAGS)' SHLIB_LD='$(SHLIB_CXXLD)' SHLIB="file7aeceb360c7.dll" WIN=64 TCLBIN=64 OBJECTS="file7aeceb360c7.o"'
14

I wonder if this is because you’re in a networked directory, can you change the working directory to somewhere on the local drive (like setwd("C:/...")) and try:

system("touch foo.cpp")

?

15

I think everything is the same. So sorry for the headache on this! I really appreciate your patience with it.

 setwd("C:/Users/mhimmelste_fordham")
> system("touch foo.cpp")
[1] 127
> system("R CMD SHLIB foo.cpp")
Warning message:
In system(cmd) :
  'CreateProcess' failed to run 'C:\rtools40\usr\bin\make.exe -f "C:/PROGRA~1/R/R-40~1.2/etc/x64/Makeconf" -f "C:/PROGRA~1/R/R-40~1.2/share/make/winshlib.mk" -f "//apporto.com/dfs/FORDHAM/Users/mhimmelste_fordham/Documents/.R/Makevars.win" SHLIB_LDFLAGS='$(SHLIB_CXXLDFLAGS)' SHLIB_LD='$(SHLIB_CXXLD)' SHLIB="foo.dll" WIN=64 TCLBIN=64 OBJECTS="foo.o"'
[1] 1
16

Hmm, the error [1] 127 means that it’s not even able to create the file, which was the same error when you ran it in the networked drive.

Two things to try here, can you first see if you can create the file just with R:

cat("",file="foo.cpp")

If that works, can you switch to the ‘Terminal’ tab in RStudio and run:

cd C:\Users\mhimmelste_fordham
touch foo.cpp

?

17

I’m guessing this is maybe going to be the answer and its something I have to ask the IT folks at Fordham about?

C:\Users\mhimmelste_fordham>touch foo.cpp
This program is blocked by group policy. For more information, contact your system administrator.
18

Ah that would do it. I’m guessing there are restrictions on the programs/executables that you’re able to run. Yep, you’ll have to get in touch with IT to find out what the restrictions are and see if they’re able to relax them for you.

So close, but so far!

1 Like
19

Got it, I’ll reach out and keep my fingers crossed. Thanks again for all the help!

1 Like