Dealing with Catalina

Just a question: Do you want me to file this as an Issue on GitHub?

No

If you have Catalina and rstanarm yields any errors when you try to estimate a model. Try installing / testing it from source with

remotes::install_github("stan-dev/rstanarm", build_vignettes = FALSE)
library(rstanarm)
example(example_model)

plus whatever model was not working before.

If you have Catalina and Stan models are not working, try running this function on the object produced by stan_model:

fix_Mac <- function(sm) {
  stopifnot(is(sm, "stanmodel"))
  dso_last_path <- sm@dso@.CXXDSOMISC$dso_last_path
  CLANG_DIR <- tail(n = 1, grep("clang[456789]", 
                    x = list.dirs("/usr/local", recursive = FALSE), 
                    value = TRUE))
  if (length(CLANG_DIR) == 0L) stop("no clang from CRAN found")
  LIBCPP <- file.path(CLANG_DIR, "lib", "libc++.1.dylib")
  if (!file.exists(LIBCPP)) stop("no unique libc++.1?.dylib found")
  Rv <- R.version
  GOOD <- file.path("/Library", "Frameworks", "R.framework", "Versions", 
                     paste(Rv$major, substr(Rv$minor, 1, 1), sep = "."), 
                     "Resources", "lib", "libc++.1.dylib")
  if (!file.exists(GOOD)) stop(paste(GOOD, "not found"))
  cmd <- paste(
    "install_name_tool",
    "-change",
    LIBCPP,
    GOOD,
    dso_last_path
  )
  system(cmd)
  dyn.unload(dso_last_path)
  dyn.load(dso_last_path)
  return(invisible(NULL))
}

followed by sampling(...).

2 Likes

I’m curious for those who are having difficulties… What happens if you type in Terminal:

R RHOME

Are you getting…

/Library/Frameworks/R.framework/Resources

or something with anaconda? If the later, please avoid using R with anaconda.

So, if I do this:

library(rstan)
library(brms)
output ← c(0.5237868884927709,0.9487179487179488,0.5494563370710159,0.46386348238482383,0.4458125179443009,0.9830952380952381,0.9685185185185186,0.5118241235888294,0.8824457593688363,0.5644325518178729,0.6317411924119242,0.4893159632500718,0.9416071428571429,0.9145833333333333,0.4509011685482273,0.7682445759368836,0.2748216106014271,0.7207063008130081,0.6109249005373036,0.9156547619047619,0.9011574074074074,0.6936423054070112,0.9012820512820513,0.8805895344886171,0.7585789295392954,0.6982066978384808,0.9988095238095238,0.9606481481481481,0.7225985343632402,0.9347140039447732,0.7936459395174992,0.7401422764227643,0.7547029449161232,0.9897619047619047,0.9571759259259259,0.734383046147752,1.0,0.8722222222222221,0.6707232384823848,0.6119692588491038,1.0,1.0,0.5487621311150723,0.8960552268244577,0.6401885830784912,0.7439871273712737,0.6910643533899349,0.9875,0.9847222222222223,0.5955040602099425,0.9396449704142013,0.5083588175331294,0.6947069783197831,0.5883210286698658,0.9872023809523809,0.9643518518518519)
categs<-as.factor(rep(c(“FOO1”,“FOO2”,“FOO3”,“FOO4”,“FOO5”,“FOO6”,“FOO7”,“FOO8”),each=7));
blocks ← c(1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7,1,2,3,4,5,6,7)
df ← data.frame(y=output, algorithm=as.integer(categs), block_id=blocks)

And then

df ← make_standata(y ~ 1 + (1 | algorithm) + (1 | block_id), data = df)

And run model.stan (2.1 KB)

fit ← stan(file = ‘model.stan’, data = df)

I get a c++ exception (unknown reason) during sampling. But…

foo ← stan_model(‘model.stan’)

fix_Mac(foo)

returns nothing, so I guess ok?

sampling(foo, data=df)

Samples nicely.

Just tell me if I misunderstood something… I am somewhat confused but I guess the important thing here is that you’re not confused @bgoodri… ;)

1 Like

I tried it and got:

SAMPLING FOR MODEL ‘pareto_gfi’ NOW (CHAIN 1).
Chain 1:
Chain 1: Gradient evaluation took 7.8e-05 seconds
Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 0.78 seconds.
Chain 1: Adjust your expectations accordingly!
Chain 1:
Chain 1:
Chain 1: Iteration: 1 / 2000 [ 0%] (Warmup)
Chain 1: Iteration: 200 / 2000 [ 10%] (Warmup)
Chain 1: Iteration: 400 / 2000 [ 20%] (Warmup)
Chain 1: Iteration: 600 / 2000 [ 30%] (Warmup)
Chain 1: Iteration: 800 / 2000 [ 40%] (Warmup)
Chain 1: Iteration: 1000 / 2000 [ 50%] (Warmup)
Chain 1: Iteration: 1001 / 2000 [ 50%] (Sampling)
Chain 1: Iteration: 1200 / 2000 [ 60%] (Sampling)
Chain 1: Iteration: 1400 / 2000 [ 70%] (Sampling)
Chain 1: Iteration: 1600 / 2000 [ 80%] (Sampling)
Chain 1: Iteration: 1800 / 2000 [ 90%] (Sampling)
Chain 1: Iteration: 2000 / 2000 [100%] (Sampling)
Chain 1:
Chain 1: Elapsed Time: 0.292604 seconds (Warm-up)
Chain 1: 0.285105 seconds (Sampling)
Chain 1: 0.577709 seconds (Total)
Chain 1:

SAMPLING FOR MODEL ‘pareto_gfi’ NOW (CHAIN 2).
Chain 2:
Chain 2: Gradient evaluation took 4.4e-05 seconds
Chain 2: 1000 transitions using 10 leapfrog steps per transition would take 0.44 seconds.
Chain 2: Adjust your expectations accordingly!
Chain 2:
Chain 2:
Chain 2: Iteration: 1 / 2000 [ 0%] (Warmup)
Chain 2: Iteration: 200 / 2000 [ 10%] (Warmup)
Chain 2: Iteration: 400 / 2000 [ 20%] (Warmup)
Chain 2: Iteration: 600 / 2000 [ 30%] (Warmup)
Chain 2: Iteration: 800 / 2000 [ 40%] (Warmup)
Chain 2: Iteration: 1000 / 2000 [ 50%] (Warmup)
Chain 2: Iteration: 1001 / 2000 [ 50%] (Sampling)
Chain 2: Iteration: 1200 / 2000 [ 60%] (Sampling)
Chain 2: Iteration: 1400 / 2000 [ 70%] (Sampling)
Chain 2: Iteration: 1600 / 2000 [ 80%] (Sampling)
Chain 2: Iteration: 1800 / 2000 [ 90%] (Sampling)
Chain 2: Iteration: 2000 / 2000 [100%] (Sampling)
Chain 2:
Chain 2: Elapsed Time: 0.295372 seconds (Warm-up)
Chain 2: 0.290389 seconds (Sampling)
Chain 2: 0.585761 seconds (Total)
Chain 2:

SAMPLING FOR MODEL ‘pareto_gfi’ NOW (CHAIN 3).
Chain 3:
Chain 3: Gradient evaluation took 4.5e-05 seconds
Chain 3: 1000 transitions using 10 leapfrog steps per transition would take 0.45 seconds.
Chain 3: Adjust your expectations accordingly!
Chain 3:
Chain 3:
Chain 3: Iteration: 1 / 2000 [ 0%] (Warmup)
[1] “Error in sampler$call_sampler(args_list[[i]]) : "
[2] " c++ exception (unknown reason)”
error occurred during calling the sampler; sampling not done
Stan model ‘pareto_gfi’ does not contain samples.

SAMPLING FOR MODEL ‘pareto_gfi’ NOW (CHAIN 1).
Chain 1: empty_nested() must be true before calling recover_memory()
[1] “Error in sampler$call_sampler(args_list[[i]]) : "
[2] " empty_nested() must be true before calling recover_memory()”
error occurred during calling the sampler; sampling not done
Stan model ‘pareto_gfi’ does not contain samples.

SAMPLING FOR MODEL ‘pareto_gfi’ NOW (CHAIN 1).
Chain 1: empty_nested() must be true before calling recover_memory()
[1] “Error in sampler$call_sampler(args_list[[i]]) : "
[2] " empty_nested() must be true before calling recover_memory()”
error occurred during calling the sampler; sampling not done
Stan model ‘pareto_gfi’ does not contain samples.
Error in { :
task 1 failed - “non-numeric argument to mathematical function”

This is different than before - it crashed on 3rd chain instead of the usual 2nd.

I wouldn’t read too much into that because when exceptions are thrown is random. There could have not been any on the first two chains.

How about doing the above using Xcode’s clang++ and headers? You probably need to comment out everything in ~/.R/Makevars .

And the entry in ~/.Renviron governing the PATH.

mac OS v10.14.6 Mojave with

clang++ --version
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.7.0

After commenting out everything in ~/.R/Makevars, I get

Error in throw_exception() : this is an exception

According to @ssp3nc3r if

Rcpp::sourceCpp(code = 
'
#include <Rcpp.h>
using namespace Rcpp; 

// [[Rcpp::export]]
int throw_exception() { 
  std::stringstream errmsg; errmsg << "this is an exception";
  throw std::domain_error(errmsg.str()); 
  return 0;
}
'
)

is compiled with Xcode’s clang++, then

throw_exception()

yields the expected

This is an exception

like @roualdes found.

But if any other package is loaded that has compiled C++ code such as dplyr, readr, etc., then

throw_exception()

yields the unexpected

c++ exception (unknown reason)

So, my conclusion is that this is a Rcpp and / or AT&T clang and / or Catalina issue rather than a RStan one, but it affects RStan a lot because Stan branches on exception type and doesn’t know what to do with “c++ exception (unknown reason)” and our previous approach to dealing with it in 2017 won’t work if the user happens to have loaded any of the 1000+ other packages that use Rcpp to compile C++ code.

Do you have any suggestions? I don’t think Anaconda is at issue here (although that often messes things up for other reasons) because @ssp3nc3r is just using a plain R and RStudio setup.

I still get “this is an exception” even after loading dplyr and readr in R or RStudio.

Sorry, all. This sounds frustrating. Thanks @bgoodri for all your help.

The example’s failure to correctly throw the exception after loading other packages was on Catalina, not Mojave. Testing on Mojave, the example correctly throws the exception whether or not other packages are loaded.

What happens if you do

Sys.setenv(PKG_LDFLAGS = "-static")

before

Rcpp::sourceCpp(code = 
'
#include <Rcpp.h>
using namespace Rcpp; 

// [[Rcpp::export]]
int throw_exception() { 
  std::stringstream errmsg; errmsg << "this is an exception";
  throw std::domain_error(errmsg.str()); 
  return 0;
}
'
)

throw_exception()

on Catalina when Apple’s clang++ is used?

No change in result. Without loading a package first, the correct error throws. Afterwards, it fails.

I added verbose = T in case its output helps:

Restarting R session...

> library(rstan)
Loading required package: StanHeaders
rstan (Version 2.19.9, GitRev: 2e1f913d3ca3)
For execution on a local, multicore CPU with excess RAM we recommend calling
options(mc.cores = parallel::detectCores()).
To avoid recompilation of unchanged Stan programs, we recommend calling
rstan_options(auto_write = TRUE)
> 
> Sys.setenv(PKG_LDFLAGS = "-static")
> 
> Rcpp::sourceCpp(code = 
+                   '
+ #include <Rcpp.h>
+ using namespace Rcpp; 
+ 
+ // [[Rcpp::export]]
+ int throw_exception() { 
+   std::stringstream errmsg; errmsg << "this is an exception";
+   throw std::domain_error(errmsg.str()); 
+   return 0;
+ }
+ ',
+ verbose = T
+ )

Generated extern "C" functions 
--------------------------------------------------------


#include <Rcpp.h>
// throw_exception
int throw_exception();
RcppExport SEXP sourceCpp_1_throw_exception() {
BEGIN_RCPP
    Rcpp::RObject rcpp_result_gen;
    Rcpp::RNGScope rcpp_rngScope_gen;
    rcpp_result_gen = Rcpp::wrap(throw_exception());
    return rcpp_result_gen;
END_RCPP
}

Generated R functions 
-------------------------------------------------------

`.sourceCpp_1_DLLInfo` <- dyn.load('/private/var/folders/lc/h55t9rqs08df0y000pnp61d40000gn/T/RtmpWD2Ye1/sourceCpp-x86_64-apple-darwin15.6.0-1.0.2/sourcecpp_3e041171732a/sourceCpp_2.so')

throw_exception <- Rcpp:::sourceCppFunction(function() {}, FALSE, `.sourceCpp_1_DLLInfo`, 'sourceCpp_1_throw_exception')

rm(`.sourceCpp_1_DLLInfo`)

Building shared library
--------------------------------------------------------

DIR: /private/var/folders/lc/h55t9rqs08df0y000pnp61d40000gn/T/RtmpWD2Ye1/sourceCpp-x86_64-apple-darwin15.6.0-1.0.2/sourcecpp_3e041171732a

/Library/Frameworks/R.framework/Resources/bin/R CMD SHLIB -o 'sourceCpp_2.so'  'file3e0418632ce.cpp'  
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I"/Library/Frameworks/R.framework/Versions/3.6/Resources/library/Rcpp/include" -I"/private/var/folders/lc/h55t9rqs08df0y000pnp61d40000gn/T/RtmpWD2Ye1/sourceCpp-x86_64-apple-darwin15.6.0-1.0.2" -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c file3e0418632ce.cpp -o file3e0418632ce.o
clang++ -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o sourceCpp_2.so file3e0418632ce.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
> 
> throw_exception()
Error in throw_exception() : c++ exception (unknown reason)

It didn’t take. Maybe try it with

Sys.setenv(PKG_LDLIBS = “-static”)

instead of

Sys.setenv(PKG_LDFLAGS = “-static”)

This is not working for me when I run:

example(stan_model, run.dontrun = TRUE)

This is the resulting error I get at the end of it:

Error in compileCode(f, code, language = language, verbose = verbose) :
Compilation ERROR, function(s)/method(s) not created! In file included from file5cb6aa88d5a.cpp:8:
In file included from /Users/lukemansillo/Library/R/3.6/library/StanHeaders/include/src/stan/model/model_header.hpp:4:
In file included from /Users/lukemansillo/Library/R/3.6/library/StanHeaders/include/stan/math.hpp:4:
In file included from /Users/lukemansillo/Library/R/3.6/library/StanHeaders/include/stan/math/rev/mat.hpp:4:
In file included from /Users/lukemansillo/Library/R/3.6/library/StanHeaders/include/stan/math/rev/core.hpp:5:
In file included from /Users/lukemansillo/Library/R/3.6/library/StanHeaders/include/stan/math/rev/core/build_vari_array.hpp:4:
In file included from /Users/lukemansillo/Library/R/3.6/library/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:4:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/library/RcppEigen/include/Eigen/Dense:1:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/library/Rc

Does anyone with Catalina want to try a clang++ from the LLVM website

http://releases.llvm.org/

as opposed to the one from CRAN?

@bgoodri, any particular version of clang++ you want us to install?