Stan now (mostly) working on RStudio Cloud

General
41

Not sure if you were planning on fixing this, but I note that there has been an update since I last posted. Nevertheless, I still have problems when trying to run the demos:

> fit1 <- stan_demo(68)
Do you want to download the example models? 

1: Download to include directory of the rstan package
2: Download to the temporary directory
3: Do not download

Selection: 2
trying URL 'https://github.com/stan-dev/example-models/archive/master.zip'
downloaded 27.8 MB


> "N" <-   8

> "sigma_y" <-   c(15,10,16,11,9,11,10,18)

> "y" <-   c(28,8,-3,7,-1,1,18,12)

Error in compileCode(f, code, language = language, verbose = verbose) : 
  Compilation ERROR, function(s)/method(s) not created! /usr/bin/ld.gold: error: filecab2dc710.o:1:3: invalid character
/usr/bin/ld.gold: error: filecab2dc710.o:1:3: syntax error, unexpected $end
/usr/bin/ld.gold: error: filecab2dc710.o: not an object or archive
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [filecab2dc710.so] Error 1
In addition: Warning message:
In system(cmd, intern = !verbose) :
  running command '/opt/R/3.5.2/lib/R/bin/R CMD SHLIB filecab2dc710.cpp 2> filecab2dc710.cpp.err.txt' had status 1
Error in sink(type = "output") : invalid connection

> fit1 <- stan_demo(1)

> "score1" <-
+ c(0.446, 0.334, -0.233, -0.304, -0.256, 0.515, -0.652, 0.271, -0.556, 
+ 0.955, 0.307, -0.434, -0.272, -0.404, 0.415, 0.39, -0.755, 0. .... [TRUNCATED] 

> "party" <-
+ c(1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 
+ 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 
+ 1, 0, 1, 0, 1, 1, 0 .... [TRUNCATED] 

> "x" <-
+ c(0.62073, 0.50803, 0.38243, 0.29769, 0.32751, 0.53766, 0.20003, 0.52211, 
+ 0.31267, 0.64143, 0.69113, 0.4386, 0.30166, 0.25836, 0.51547,  .... [TRUNCATED] 

> "N" <-  357

Error in compileCode(f, code, language = language, verbose = verbose) : 
  Compilation ERROR, function(s)/method(s) not created! clang: error: unable to execute command: Killed
clang: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 5.0.0-3~16.04.1 (tags/RELEASE_500/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
clang: note: diagnostic msg: PLEASE submit a bug report to http://llvm.org/bugs/ and include the crash backtrace, preprocessed source, and associated run script.
clang: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /tmp/fileca40943554-1385b3.cpp
clang: note: diagnostic msg: /tmp/fileca40943554-1385b3.sh
clang: note: diagnostic msg: 

********************
make: *** [fileca40943554.o] Error 254
In addition: Warning message:
In system(cmd, intern = !verbose) :
  running command '/opt/R/3.5.2/lib/R/bin/R CMD SHLIB fileca40943554.cpp 2> fileca40943554.cpp.err.txt' had status 1
Error in sink(type = "output") : invalid connection
42

It is still not working. I am trying to put together a customized version that utilizes less RAM.

43

Hello :)

I want to prepare a course using brms and Rstudio Cloud, but I am unable to compile any model.

I used the link in the first post of this thread, made a personnal copy and I tried with demos from stan_demo and with very simple models with only two parameters, but I always get this error message :

Error in compileCode(f, code, language = language, verbose = verbose) : 
  Compilation ERROR, function(s)/method(s) not created! In file included from file815cbab7cf.cpp:8:
In file included from /home/rstudio-user/R/x86_64-pc-linux-gnu-library/3.5/StanHeaders/include/src/stan/model/model_header.hpp:4:
In file included from /home/rstudio-user/R/x86_64-pc-linux-gnu-library/3.5/StanHeaders/include/stan/math.hpp:4:
In file included from /home/rstudio-user/R/x86_64-pc-linux-gnu-library/3.5/StanHeaders/include/stan/math/rev/mat.hpp:12:
In file included from /home/rstudio-user/R/x86_64-pc-linux-gnu-library/3.5/StanHeaders/include/stan/math/prim/mat.hpp:276:
In file included from /home/rstudio-user/R/x86_64-pc-linux-gnu-library/3.5/StanHeaders/include/stan/math/prim/mat/prob/dirichlet_rng.hpp:5:
In file included from /home/rstudio-user/R/x86_64-pc-linux-gnu-library/3.5/BH/include/boost/random/gamma_distribution.hpp:25:
In file included from /home/rstudio-user/R/x86_64-pc-linux-gnu-library/3.5/BH/include/boost/random/exponential_distribution.hpp:27:
In file 
In addition: Warning message:
In system(cmd, intern = !verbose) :
  running command '/opt/R/3.5.2/lib/R/bin/R CMD SHLIB file815cbab7cf.cpp 2> file815cbab7cf.cpp.err.txt' had status 1
Error in sink(type = "output") : invalid connection

Any idea??

Thank you very much!
Lucas

EDIT : everything works fine while using rstanarm, which does not have to compile.

44

I have not been able to get much of anything to compile on RStudio Cloud with rstan >= 2.18 due to RAM limits. One option would be for me to ask the RStudio Cloud people to downgrade everything Stan related to 2.17. However, that is not great because recent versions of brms and related packages are going to require rstan >= 2.18.

So, there is no good solution. After rstan 2.19 is released I am going to try again to reduce the RAM consumed in the compilation step. When does your course start?

45

Thank you for your response!

I plan to use brms the 5th of april. It would be very convenient to use Rstudio cloud, because I always loose 10-15 minutes at the beginning of each session because of installation problems.

However, I already used rstan in a session without the cloud, so I am not too worried :)

Would there be any change in compilation related RAM managenent in stan 2.19? I am unfortunatly not verse in low level programmation, and cannot understand what changed between 2.17 and 2.18 relatively to this issue!

46

One thought : is the problem related only to rstan? Would this problem be solved with a light R interface to Cmdstan?

47

In 2.18 we started using C++14 (or really C++11), which increased the amount of RAM needed to compile Stan programs, which is going to affect every conceivable and actual interface to Stan when constrained by a 1GB RAM limit.

48

Hi Ben, I took your class a couple of years ago. I noticed that the simple linear regression example on https://rstudio.cloud/project/56157 no longer works. It was working a few months ago, but now I get an error message. I’m wondering, are you aware of this issue ?

49

Hi Laura. Yes, I’m aware. At the moment, it is impossible to use RStudio Cloud with Stan, but I am hopeful that Bob’s refactor of the translation units will save RAM as well as compilation time, so that we can work within the 1GB limit that RStudio Cloud imposes.

2 Likes
50

Thanks for the info ! Will try again later on then.

51

Any updates on this?

52

Close, but still mostly not working.

53

Hi, sorry for being late to this discussion. Is the problem installation or model compilation?

Do you know if -ltfo or (for clang) -ltfo-thin work while building a model?

54

The problem is / was model compilation under a 1GB of RAM constraint for users. With rstan > 2.19.x where the model_base_crtp class is compiled once at installation by the RStudio Admins (who are not subject to RAM constraints), users should be able to compile the class that implements their log_prob function with about 0.8 GB of RAM using clang++.

1 Like
55

I will say I can use rstanarm using the Cloud project provided in this thread, since those models come pre-compiled. Uncompiled models (like the examples, and likely brms models) don’t seem to work. Just an FYI.

56

For its worth, I was recently granted 4GB of RAM and 2 cores and can now compile the example code provided in the Stan Cloud project. Stan also appears to be threading normally when mc.cores \ge 2.

57

I also requested the RStudio Cloud administrator to gain memory.
The code properly ran under 3 GB RAM and 1 CPU.

1 Like
58

Sorry to hijack this threat. My company apparently has its own RStudio Cloud server that runs on UNIX with a lot of cores and RAM (or so I was told). I am trying to get RStan to work there. What are the steps to set it up properly? Are they the same steps listed on https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started for Windows? Or are those listed for Linux? Thank you.

59

Linux