Problem with R update to 4.0.2 on MacOS: stan stuck in model fit

Interfaces RStan
1

Hi community. This is my first post here. I updated R to v4.0.2 and configured the C++ toolchain, following the RStan Getting Started page.

When testing the install with the tutorial-provided eight schools example, the model appears stuck, there’s no better word for it. No warnings are produced initially, and the CPU running suggests the chains are going. But this continues for as much as 6hrs. If I escape out of the process, I get these warnings:

> stan_samples <- stan(model_code = model_string, data = list(y = y, n = length(y)) )

Warning messages:
1: In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  system call failed: Undefined error: 0
2: In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  error in running command

I’m using the R interface, not R Studio. System info is as follows:
R version 4.0.2 (2020-06-22) – “Taking Off Again”
Copyright © 2020 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin17.0 (64-bit)
[R.app GUI 1.72 (7847) x86_64-apple-darwin17.0]
MacOS Mojave 10.14.6

Reproduced from RStan Getting Started

// saved as schools.stan
data {
  int<lower=0> J;         // number of schools 
  real y[J];              // estimated treatment effects
  real<lower=0> sigma[J]; // standard error of effect estimates 
}
parameters {
  real mu;                // population treatment effect
  real<lower=0> tau;      // standard deviation in treatment effects
  vector[J] eta;          // unscaled deviation from mu by school
}
transformed parameters {
  vector[J] theta = mu + tau * eta;        // school treatment effects
}
model {
  target += normal_lpdf(eta | 0, 1);       // prior log-density
  target += normal_lpdf(y | theta, sigma); // log-likelihood
}

schools_dat <- list(J = 8, 
                    y = c(28,  8, -3,  7, -1,  1, 18, 12),
                    sigma = c(15, 10, 16, 11,  9, 11, 10, 18))

fit <- stan(file = 'schools.stan', data = schools_dat)

Thanks for any help, very much appreciated.

2

Does running the schools.stan example produce the same behaviour, or is it just your particular model:

stan_samples <- stan(model_code = model_string, data = list(y = y, n = length(y)) )

Are you able to post your full model?

3

Thanks for the reply.

The schools.stan example with the posted model produces these errors. I’ve run a few other test models, to the same end. In the example I posted, model_string was just the schools.stan text file, sorry for the confusion.

4

As an initial troubleshooting step, what output do you get if you run:

example(stan_model, run.dontrun=TRUE, verbose=TRUE)
5 
> example(stan_model, run.dontrun=TRUE, verbose=TRUE)
Found file = ‘/Library/Frameworks/R.framework/Versions/4.0/Resources/library/rstan/help/stan_model’ 
'envir' chosen:<environment: R_GlobalEnv>
encoding = "UTF-8" chosen
--> parsed 4 expressions; now eval(.)ing them:
has srcrefs:
List of 4
 $ : 'srcref' int [1:8] 8 1 8 84 1 84 8 8
  ..- attr(*, "srcfile")=Classes 'srcfilecopy', 'srcfile' <environment: 0x7fe43a513ab8> 
 $ : 'srcref' int [1:8] 9 1 9 56 1 56 9 9
  ..- attr(*, "srcfile")=Classes 'srcfilecopy', 'srcfile' <environment: 0x7fe43a513ab8> 
 $ : 'srcref' int [1:8] 10 1 10 45 1 45 10 10
  ..- attr(*, "srcfile")=Classes 'srcfilecopy', 'srcfile' <environment: 0x7fe43a513ab8> 
 $ : 'srcref' int [1:8] 11 1 11 46 1 46 11 11
  ..- attr(*, "srcfile")=Classes 'srcfilecopy', 'srcfile' <environment: 0x7fe43a513ab8> 

>>>> eval(expression_nr. 1 )
		 =================

stn_md> stancode <- 'data {real y_mean;} parameters {real y;} model {y ~ normal(y_mean,1);}'
curr.fun: symbol <-
 .. after ‘expression(stancode <- 'data {real y_mean;} parameters {real y;} model {y ~ normal(y_mean,1);}')’

>>>> eval(expression_nr. 2 )
		 =================

stn_md> mod <- stan_model(model_code = stancode, verbose = TRUE)

TRANSLATING MODEL '16a540c6086086816528e4524def24d9' FROM Stan CODE TO C++ CODE NOW.
successful in parsing the Stan model '16a540c6086086816528e4524def24d9'.
6

And apologies, that post of the output may have been premature. That output is given initially, but the command is still running, similar to what I’ve observed before- apparently ‘stuck’, with a process still running on the CPU, and if I escape out I get:

Warning messages:
1: In find.package(package, lib.loc, verbose = verbose) :
  package ‘base’ found more than once, using the first from
  “/Library/Frameworks/R.framework/Resources/library/base”,
  “/Library/Frameworks/R.framework/Versions/4.0/Resources/library/base”
2: In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  system call failed: Undefined error: 0
3: In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
  error in running command
7

Is there anything else in your output? For example, on my system I also have:

make cmd is
  make -f "C:/PROGRA~1...

make would use
"C:/rtools40/mingw64/bin/"g++ ...

Can I also check what version of RStan that you’re using?

8

That is the entire output, though the process still runs (ie, it’s not complete, and R is chewing away on something). The behavior is similar it seems to the initial example (schools.stan) above.

rstan 2.21.2, following an R update to 4.0.2

9

When you say update to 4.0.2, were you previously on a 4.0.x version or an 3.x version of R? Additionally, did you install RStan before or after the upgrade?

If I recall correctly, you need to reinstall your R packages when moving from R3.x -> R4.x, so that might be the issue here.

If that’s not the case, can you also post the output from:

readLines("~/.R/Makevars")
10

I was running R 3.5.1 and an rstan version of similar vintage (can’t recall which). You’re correct, installing R 4.0.2 required reinstalling all packages. However, the rstan installation following Installing RStan from source on a Mac appeared to go as per instructions (until the test phase and problems we’re discussing now).

> readLines("~/.R/Makevars")
Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
  cannot open file '/Users/jdsal/.R/Makevars': No such file or directory
11

Alrighty, so it’s not the update from 3.5, and that error message means that you don’t have a Makevars file (which is a good thing at this stage). Can you also check whether there is an existing .Renviron file?

readLines("~/.Renviron")

We basically need to see if there are any configurations left over from the old version that are causing issues

12

“Alrighty” sounds like progress, this is good.

readLines(“~/.Renviron”)
Error in file(con, “r”) : cannot open the connection
In addition: Warning message:
In file(con, “r”) :
cannot open file ‘/Users/jdsal/.Renviron’: No such file or directory

Thanks much, hugely appreciate the help.

13

Unfortunately that’s the extent of my knowledge with macs sorry! I believe @jonah is on Mac, so he might be able to get you over the line here.

Sorry that you can’t get started with Stan just yet!

14

So if I understand correctly:

  • R 4.0.2
  • Mac OS X Mojave
  • No Makevars file
  • No Renviron file
  • Fresh install of the latest RStan

and running the schools example model just hangs for ~6 hrs?

That’s pretty strange! I don’t think I’ve seen that before, although I’ve seen a few people report issues with Mojave. One thing to try is reinstalling Rcpp from source and then, if necessary, also reinstalling RStan from source again after installing Rcpp from source:

install.packages("Rcpp", type = "source")
# try running model again, if same issue then
install.packages("rstan", type = "source")
15

And if that gets it working then I recommend creating a Makevars file:

# create the file
file.create(file.path(Sys.getenv("HOME"), ".R", "Makevars"))
# open the file for editing
file.edit(file.path(Sys.getenv("HOME"), ".R", "Makevars"))

add these lines

CXX14FLAGS=-O3 -march=native -mtune=native 
CXX14FLAGS += -arch x86_64 -ftemplate-depth-256

That should speed up your models (once you get them working).

16

Jonah, hi there. Thanks for coming on board the thread.

The fresh install of Rcpp from source, then trying the model, yielded the same result as before- the model appears stuck. And, the fresh install of Rcpp then rstan from source, yielded the same as well.

A couple points maybe relevant. First, I’m experiencing these identical problems on two machines that I’m troubleshooting in parallel, both with new R 4.0.2 installed, one running OSX 10.13.6 High Sierra and the other 10.14.6 Mojave.

Second, perhaps relevant, even after quitting out of the R console, the processes continue to run in the background, or at least separate R processes continue to suck up CPU space shown on the system activity monitor.

17

Yeah, we used to have a lot of trouble with the newer OS X Catalina, but now it’s the opposite and Catalina seems to be fine and the older Mac OSes seem to result in more issues. Often installing Rcpp and RStan from source fixes the issue, but if that didn’t fix it for you then I’m not sure what else to try. If we can’t figure this out soon but you want to run Stan models from R then another option is to try the new CmdStanR interface.

But @bgoodri might have more suggestions. Ben have you seen this before? Any ideas? Here’s a summary:

And installing Rcpp and then RStan from source didn’t fix the problem.

18

Maybe

remotes::install_github("bgoodri/inline")

and passing verbose = TRUE to stan or sampling will reveal something about how it was (mis)compiled. Look for the part that says “make would call …”.

19

Hey, thanks for chiming in! The below model has been ‘running’ and appears stuck, as before. In the interest of being verbose, I’m including the whole process.

> remotes::install_github("bgoodri/inline")
Downloading GitHub repo bgoodri/inline@HEAD
─  preparing ‘inline’:/private/var/folders/h1/rq6w0dz12pbc3lpb129lhlhh0000gn/T/Rtmp6yGZDy/remotes16ec822742797/bgoodri-inline-f80c7f9/DESCRIPTION’ ...
─  installing the package to process help pages
   es
* installing *source* package ‘inline’ ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (inline)
> library(rstan)
Loading required package: StanHeaders
Loading required package: ggplot2
rstan (Version 2.21.2, GitRev: 2e1f913d3ca3)
For execution on a local, multicore CPU with excess RAM we recommend calling
options(mc.cores = parallel::detectCores()).
To avoid recompilation of unchanged Stan programs, we recommend calling
rstan_options(auto_write = TRUE)
> 
> model <- 
+ "data {
+   int<lower=0> J;
+   real y[J];
+   real<lower=0> sigma[J];
+ }
+ parameters {
+   real mu;
+   real<lower=0> tau;
+   vector[J] eta;
+ }
+ transformed parameters {
+   vector[J] theta = mu + tau * eta;
+ }
+ model {
+   target += normal_lpdf(eta | 0, 1);
+   target += normal_lpdf(y | theta, sigma);
+ }"
> 
> schools_dat <- list(J = 8, 
+                     y = c(28,  8, -3,  7, -1,  1, 18, 12),
+                     sigma = c(15, 10, 16, 11,  9, 11, 10, 18))
> fit <- stan(model_code=model, data = schools_dat, verbose=T)

TRANSLATING MODEL 'c6e5b88b131a8ef63e73c683a2589167' FROM Stan CODE TO C++ CODE NOW.
successful in parsing the Stan model 'c6e5b88b131a8ef63e73c683a2589167'.
20

Try first doing

rstan_options(javascript = FALSE)