Dealing with Catalina

[kevinushey]

Can you confirm whether the object compiled by Rcpp::sourceCpp() is using the system C++ standard library or the one provided by R? Try running this after compiling the C++ object:

dlls <- getLoadedDLLs()
paths <- vapply(dlls, `[[`, "path", FUN.VALUE = character(1))
invisible(lapply(paths, function(path) {
  
  if (!file.exists(path))
    return(FALSE)
  
  output <- system(paste("otool -L", shQuote(path), "| grep libc++ || true"), intern = TRUE)
  if (length(output) == 0)
    return(FALSE)
  
  writeLines(paste0(path, ":"))
  writeLines(output)
  
}))

I wonder if you’ll see something like this:

/private/var/folders/b4/2422hswx71z8mgwtv4rhxchr0000gn/T/RtmpuZz5kZ/sourceCpp-x86_64-apple-darwin15.6.0-1.0.2/sourcecpp_47b76cd81e72/sourceCpp_2.so:
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.6.0)

If you still see the issue even with everyone using R’s libc++, I will be surprised!

[bgoodri]

@ssp3nc3r says that he got

/Library/Frameworks/R.framework/Versions/3.6/Resources/library/Rcpp/libs/Rcpp.so:
/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
/Library/Frameworks/R.framework/Versions/3.6/Resources/library/readr/libs/readr.so:
/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
/private/var/folders/lc/h55t9rqs08df0y000pnp61d40000gn/T/Rtmp0GWy17/sourceCpp-x86_64-apple-darwin15.6.0-1.0.2/sourcecpp_ced87afdaccc/sourceCpp_2.so:
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.6.0)

which seems to confirm your haunch. So, do we need to make Rcpp::sourceCpp and inline::cxxfunction use R’s libc++.1.dylib somehow? Or is the only option to do the install_name_tool thing?

[kevinushey]

There should be a way to do this with some special flags in ~/.R/Makevars, but I’m not sure exactly how this could be accomplished yet. Overriding which standard library gets used by the compiler is a bit tricky, and requires some orchestration of flags both during compilation and linking.

install_name_tool is normally a simpler workaround, but I’m not sure if there’s a clear space for it to fit during Rcpp::sourceCpp().

[bigmanj]

Hi, sorry–as a new used I could not post more until 24hrs.

I think I figured out where to look (silly me, just needed verbose = TRUE).

I think it is here:

In file included from /usr/local/clang7/include/c++/v1/cstdlib:86:
/usr/local/clang7/include/c++/v1/stdlib.h:94:15: fatal error: ‘stdlib.h’ file not found
#include_next <stdlib.h>

Thank you!

[janhannig]

@kevinushey

dlls ← getLoadedDLLs() paths ← vapply(dlls, [[, “path”, FUN.VALUE = character(1)) invisible(lapply(paths, function(path) { if (!file.exists(path)) return(FALSE) output ← system(paste(“otool -L”, shQuote(path), “| grep libc++ || true”), intern = TRUE) if (length(output) == 0) return(FALSE) writeLines(paste0(path, “:”)) writeLines(output) }))

I ran this and got:

/Library/Frameworks/R.framework/Versions/3.6/Resources/library/Rcpp/libs/Rcpp.so:
	/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

I am intermediate computer user and I am not sure what I am supposed to do to fix this issue.

[bgoodri]

@kevinushey Have you seen a Mac with Catalina where the install_name_tool makes exceptions work? On @ssp3nc3r’s Mac, I get the attached log where install_name_tool seems to work but exceptions continue to not work.

exceptions.txt (5.5 KB)

[kevinushey]

I believe install_name_tool would need to be called before the compiled shared object is loaded.

An easier way to verify and test would be with an installed package that does the same thing, fixing up the .so before load.

[bgoodri]

Maybe. But Rcpp::sourceCpp and inline::cxxfunction are loading the DLLs automatically, so I don’t see how those would ever work on a Catalina machine. Prior to Catalina, rstan was working with essentially

1. inline::cxxfunction
2. install_name_tool ...
3. dyn.unload(path)
4. dyn.load(path)

[mattwilliamson13]

Does the new Stan release solve this issue or is Catalina still causing problems?

[bgoodri]

Unlikely

[bensilver95]

Hello, I am a new Stan user and just want to echo that I am having the exact same issue as other people here. It seems like conversation on this thread has fallen off in the past few days - was a permanent, consistent solution ever found? Thanks.

[bgoodri]

No.

[jonah]

@bensilver95

Hello, I am a new Stan user and just want to echo that I am having the exact same issue as other people here. It seems like conversation on this thread has fallen off in the past few days - was a permanent, consistent solution ever found? Thanks.

Not yet but @bgoodri and others are working on it!

If you want to try running Stan from R a different way in the meantime this might work:

Error in rstan

As a last resort, if CmdStan doesn’t have problems with Catalina (I would think more it’s likely to be ok than RStan) then you could try out the new (unreleased, and unannounced until now, and subject to change) CmdStanR interface, which can be installed from GitHub devtools::install_github("stan-dev/cmdstanr") There’s a brief intro vignette showing how to install cmdstan (there’s an install_cmdstan function) and then use cmdstanr to fit models from R (and obtain a stanfit object from the resu…

[torkar]

I use Docker for now, and it works nicely :)

https://cloud.docker.com/repository/docker/torkar/rstan

Install via the terminal by writing (on one line):

docker run -e PASSWORD=YOURPASSWORD -p 8787:8787 -v /PATH/TO/YOUR/HOMEDIR:/home/rstudio torkar/rstan:latest

Change YOURPASSWORD and /PATH/TO/YOUR/HOMEDIR. Point your browser to localhost:8787.

[cesmithjr]

Docker is the only way I can work right now on the machines I “upgraded” to Catalina. It’s just amazing that Apple has bricked so many people. I have several friends who just can’t work until there is a simple solution.

[mansillo]

I tried to load blavaan earlier today and got an error - adding it in case it points to a solution

library(blavaan)
Loading required package: Rcpp
Error: package or namespace load failed for ‘blavaan’ in dyn.load(file, DLLpath = DLLpath, …):
unable to load shared object ‘/Library/Frameworks/R.framework/Versions/3.6/Resources/library/quantreg/libs/quantreg.so’:
dlopen(/Library/Frameworks/R.framework/Versions/3.6/Resources/library/quantreg/libs/quantreg.so, 6): Library not loaded: /usr/local/gfortran/lib/libgfortran.5.dylib
Referenced from: /Library/Frameworks/R.framework/Versions/3.6/Resources/library/quantreg/libs/quantreg.so
Reason: image not found
install.packages(“blavaan”)
Installing package into ‘/Users/lukemansillo/Library/R/3.6/library’
(as ‘lib’ is unspecified)
trying URL ‘http://cran.rstudio.com/bin/macosx/el-capitan/contrib/3.6/blavaan_0.3-7.tgz’
Content type ‘application/x-gzip’ length 10100210 bytes (9.6 MB)
==================================================
downloaded 9.6 MB
The downloaded binary packages are in
/var/folders/bx/8ltnl1nd13b3k8vjzl8m7ffw0000gn/T//RtmpEpxFYL/downloaded_packages
library(blavaan)
Error: package or namespace load failed for ‘blavaan’ in dyn.load(file, DLLpath = DLLpath, …):
unable to load shared object ‘/Library/Frameworks/R.framework/Versions/3.6/Resources/library/quantreg/libs/quantreg.so’:
dlopen(/Library/Frameworks/R.framework/Versions/3.6/Resources/library/quantreg/libs/quantreg.so, 6): Library not loaded: /usr/local/gfortran/lib/libgfortran.5.dylib
Referenced from: /Library/Frameworks/R.framework/Versions/3.6/Resources/library/quantreg/libs/quantreg.so
Reason: image not found

[martinmodrak]

For transparency: I posted about the issues with Catalina on official Stan Twitter so that users are warned. If you think I’ve mischaracterized the situation, let me know and I will update the info there (people with access to official Twitter can obviously do it themselves).

Wish you good luck with hunting this down.

[mansillo]

@torkar I just installed Docker ran a brms job with it but the error emerged for one of the chains:

m_demos_secular_age ← brm(auth_theta_st |weights(weight)~ s(age,by=religion_sample),family=gaussian(),
data=survey_data,
cores = 4,chains=4, iter = 5000,
file=“models/m_demos_secular_age”)
Compiling the C++ model
recompiling to avoid crashing R session
Start sampling
starting worker pid=1162 on localhost:11460 at 01:37:30.530
starting worker pid=1176 on localhost:11460 at 01:37:30.722
starting worker pid=1190 on localhost:11460 at 01:37:30.911
starting worker pid=1204 on localhost:11460 at 01:37:31.099
SAMPLING FOR MODEL ‘0c266c38e0747884b21cf03aad6cbcc2’ NOW (CHAIN 1).
Chain 1:
Chain 1: Gradient evaluation took 0.00253 seconds
Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 25.3 seconds.
Chain 1: Adjust your expectations accordingly!
Chain 1:
Chain 1:
Chain 1: Iteration: 1 / 5000 [ 0%] (Warmup)
SAMPLING FOR MODEL ‘0c266c38e0747884b21cf03aad6cbcc2’ NOW (CHAIN 2).
Chain 2:
Chain 2: Gradient evaluation took 0.002729 seconds
Chain 2: 1000 transitions using 10 leapfrog steps per transition would take 27.29 seconds.
Chain 2: Adjust your expectations accordingly!
Chain 2:
Chain 2:
Chain 2: Iteration: 1 / 5000 [ 0%] (Warmup)
SAMPLING FOR MODEL ‘0c266c38e0747884b21cf03aad6cbcc2’ NOW (CHAIN 3).
Chain 3:
Chain 3: Gradient evaluation took 0.003116 seconds
Chain 3: 1000 transitions using 10 leapfrog steps per transition would take 31.16 seconds.
Chain 3: Adjust your expectations accordingly!
Chain 3:
Chain 3:
[1] “Error in sampler$call_sampler(args_list[[i]]) : "
[2] " c++ exception (unknown reason)”
error occurred during calling the sampler; sampling not done
SAMPLING FOR MODEL ‘0c266c38e0747884b21cf03aad6cbcc2’ NOW (CHAIN 4).
Chain 4:
Chain 4: Gradient evaluation took 0.002956 seconds
Chain 4: 1000 transitions using 10 leapfrog steps per transition would take 29.56 seconds.
Chain 4: Adjust your expectations accordingly!
Chain 4:
Chain 4:
Chain 4: Iteration: 1 / 5000 [ 0%] (Warmup)

Am I possibly doing/done something wrong? Have you had success sampling with each implementation of rstan? Does this extend to brms or any other package such as blavaan that acts as a wrapper for RStan?

[torkar]

You are running this on localhost:8787?!? I, and others, have used the Docker image now for a number of days and I haven’t seen that error on Docker :( Can you send a MWE (incl. data) that I can try to run to see what happens?

[mansillo]

I was hoping it could be that I was doing something wrong since I’m unfamiliar with much computer science. What do you mean by running it on localhost:8787 exactly?