R 4.0.0 and CRAN MacOS binaries

… there is apparently light at the end of the tunnel:

See http://mac.r-project.org :

Important note about R 4.0.0
Starting with R 4.0.0 alpha we are building R using standard Apple tools (Xcode 11.4) and GNU Fortran 8.2 from fxcoudert and the target is macOS 10.13 (High Sierra). All dependent static libraries are available in the libs-4 directory. Please make sure you remove any modifications to build flags from your home since no custom compilers are used anymore.

I would very much hope that this resolves the current macOS Catalina hiccups with stan. Hopefully someone has time to test this…

Sebastian

6 Likes

Hi!

This was actually a call to the community to test this. It is really straightforward to do so. Anyone with Catalina problems should join in resolving the matter.

Now, I went ahead and installed the latest R 4.0.0 RC on my macOS Catalina. As CRAN now provides binary packages pre-build with the new toolchain (based on clang macOS High Sierra toolchain) this promises to solve all problems… and there is a lot of light here:

  • IT WORKS TO DOWNLOAD PRE-BUILD RSTAN BASED PACKAGES AND RUN THEM JUST FINE ON CATALINA
  • compling a Stan model with stan_model works fine
  • sampling a Stan model with Sampling works just fine - with/without running chains in parallel
  • sampling with my own packages OncoBayes2 + RBesT works only fine when avoiding parallel sampling strangely
  • sampling with rstanarm in parallel does work

@bgoodri while there is some oddity going on with OncoBayes2 & RBesT for parallel sampling, I think this all looks very promising!

All I have in my ~/.R/Makevars is:

CFLAGS+=-O3 -march=native -mtune=native
CCFLAGS+=-O3 -march=native -mtune=native
CXXFLAGS+=-O3 -march=native -mtune=native

CXX14FLAGS+=-O3 -march=native -mtune=native

CXX14FLAGS += -DSTAN_THREADS
CXX14FLAGS += -arch x86_64 -ftemplate-depth-256
CXX14FLAGS += -DBOOST_MATH_PROMOTE_DOUBLE_POLICY=false

Below is the R script with the key outputs as produced by my R session:

pkgs <- c("rstan", "RBesT", "OncoBayes2", "rstanarm")

for(p in pkgs) {
    if(!require(p, character.only=TRUE))
        install.packages(p)
    require(p, character.only=TRUE)
}

## ok
example_model("combo2_trial")


options(mc.cores=4)
example_model("combo3")
## fails with
## Error in unserialize(node$con) : error reading from connection
## Calls: <Anonymous> -> slaveLoop -> makeSOCKmaster
## Execution halted


options(mc.cores=1)

## ok
set.seed(34563)
map_AS <- gMAP(cbind(r, n-r) ~ 1 | study,
               family=binomial,
               data=AS,
               tau.dist="HalfNormal", tau.prior=1,
               beta.prior=2)

map_AS

forest_plot(map_AS)

save_stan_model  <- function(fit) {
    model_name  <- attr(fit, "model_name")
    stan_model_file  <- paste0(model_name, ".stan")
    cat(paste0(get_stancode(fit), "\n"), file=stan_model_file)
    stan_model_file
}

library(rstan)
stan_model_file <- save_stan_model(map_AS$fit)

sm <- stan_model(stan_model_file)

fit_AS  <- sampling(sm, data=map_AS$fit.data, control=list(adapt_delta=0.9), cores=4)

print(fit_AS, pars="theta_pred")



## rstanarm - all ok
data(roaches)
roaches$roach1 <- roaches$roach1 / 100
roaches$log_exposure2 <- log(roaches$exposure2)
post <- stan_gamm4(
  y ~ s(roach1) + treatment + log_exposure2,
  random = ~(1 | senior),
  data = roaches,
  family = neg_binomial_2,
  QR = TRUE,
  cores = 2,
  chains = 2,
  adapt_delta = 0.99,
  seed = 12345)

R session:

> sessionInfo()
R version 4.0.0 RC (2020-04-21 r78267)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.4

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] C/UTF-8/C/C/C/C

attached base packages:
[1] stats     graphics  datasets  grDevices utils     methods   base     

other attached packages:
 [1] rstan_2.19.3       ggplot2_3.3.0      StanHeaders_2.19.2 abind_1.4-5       
 [5] tidyr_1.0.2        dplyr_0.8.5        tibble_3.0.1       OncoBayes2_0.6-4  
 [9] RBesT_1.6-0        Rcpp_1.0.4.6      

loaded via a namespace (and not attached):
 [1] tidyselect_1.0.0   purrr_0.3.4        reshape2_1.4.4     tcltk_4.0.0       
 [5] colorspace_1.4-1   vctrs_0.2.4        stats4_4.0.0       loo_2.2.0         
 [9] rlang_0.4.5        pkgbuild_1.0.6     pillar_1.4.3       withr_2.2.0       
[13] glue_1.4.0         matrixStats_0.56.0 lifecycle_0.2.0    plyr_1.8.6        
[17] stringr_1.4.0      munsell_0.5.0      gtable_0.3.0       mvtnorm_1.1-0     
[21] codetools_0.2-16   labeling_0.3       inline_0.3.15      callr_3.4.3       
[25] ps_1.3.2           parallel_4.0.0     fansi_0.4.1        bayesplot_1.7.1   
[29] rstantools_2.0.0   scales_1.1.0       backports_1.1.6    checkmate_2.0.0   
[33] farver_2.0.3       gridExtra_2.3      digest_0.6.25      stringi_1.4.6     
[37] processx_3.4.2     grid_4.0.0         cli_2.0.2          tools_4.0.0       
[41] magrittr_1.5       Formula_1.2-3      crayon_1.3.4       pkgconfig_2.0.3   
[45] ellipsis_0.3.0     prettyunits_1.1.1  ggridges_0.5.2     assertthat_0.2.1  
[49] R6_2.4.1           compiler_4.0.0    
> 
5 Likes

OK, I will try to figure it out. My only guess so far is that there is some difference between packages depending on whether they were (re)created before or after rstantools 2.0.

Ok, thanks for looking into that. Just to make sure: serial sampling does seem to work just fine for RBesT & OncoBayes2; just the parallel stuff is going stray which I don’t get as I do not really do anything different than rstanarm when calling out to the sampler (at least this is what I think).

This is indeed the case at the cost of OpenMP being available for all users. So, one step forward two steps back.

Yeah, that is unfortunate. But since Stan is not using OpenMP, I guess this is going to lead to fewer posts around here.

Agreed. Plus, the installation goes down to:

  1. Install Xcode CLI
  2. Install gfortran

Done.

Gfortran for Stan, really?

No, that is for other stuff

I installed R 4.0.0 RC on Debian bullseye (11). I used this in .R/Makevars

CXX14FLAGS=-O3 -march=native -mtune=native -fPIC
CXX14=g++

I used install.packages("rstan", type = "source"), which installed rstan 2.19.3 and StanHeaders 2.19.2.

The 8schools example ran without a problem.

2 Likes

What version of g++ do you have? I got a student’s Mac to work using clang++ from Xcode with some effort.

g++ (Debian 9.3.0-10) 9.3.0

Hi This is my first post to this group and I am a novice. I am working with Imperial’s covid model 3 on an Imac and have just updated to R 4.0.0
I had to move to an old machine not running Catalina to run and experiment with their earlier models because of the reasons that have been discussed between Stan and Catalina. However this machine is MUCH more powerful so need to move my research now onto this machine. So Rstan now works but I have the same problem by bgoodri above- I cannot get 4 cores to sample just 1 works. Is there anything I can do to overcome this please??? this is the error message from R

[1] “First non-zero cases is on day 64, and 30 days before 10 deaths is day 53”
[1] “Portugal has 64 days of data”
[1] “First non-zero cases is on day 60, and 30 days before 10 deaths is day 46”
[1] “Netherlands has 71 days of data”
starting worker pid=45437 on localhost:11120 at 20:50:04.418
starting worker pid=45434 on localhost:11120 at 20:50:04.418
starting worker pid=45436 on localhost:11120 at 20:50:04.419
starting worker pid=45435 on localhost:11120 at 20:50:04.419
Error in unserialize(node$con) : error reading from connection
Calls: -> slaveLoop -> makeSOCKmaster
Error in unserialize(node$con) : error reading from connection
Calls: -> slaveLoop -> makeSOCKmaster
Execution halted
Execution halted
Error in unserialize(node$con) : error reading from connection
Calls: -> slaveLoop -> makeSOCKmaster
Error in unserialize(node$con) : error reading from connection
Calls: -> slaveLoop -> makeSOCKmaster
Execution halted
Execution halted
Error in makePSOCKcluster(names = spec, …) :
Cluster setup failed. 4 of 4 workers failed to connect.


and this is my set up

sessionInfo()
R version 4.0.0 (2020-04-24)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.4

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages:
[1] parallel stats graphics grDevices utils datasets methods base

other attached packages:
[1] rstanarm_2.19.3 xts_0.12-0 zoo_1.8-7 rstantools_2.0.0 Rcpp_1.0.4.6 abind_1.4-5 ggpubr_0.2.5 magrittr_1.5
[9] bayesplot_1.7.1 scales_1.1.0 stringr_1.4.0 optparse_1.6.6 EnvStats_2.3.1 tidyr_1.0.2 dplyr_0.8.5 gdata_2.18.0
[17] data.table_1.12.8 rstan_2.19.3 ggplot2_3.3.0 StanHeaders_2.19.2 lubridate_1.7.8 httr_1.4.1 readr_1.3.1

loaded via a namespace (and not attached):
[1] nlme_3.1-147 matrixStats_0.56.0 threejs_0.3.3 tools_4.0.0 R6_2.4.1 DT_0.13 colorspace_1.4-1 withr_2.2.0
[9] tidyselect_1.0.0 gridExtra_2.3 prettyunits_1.1.1 processx_3.4.2 curl_4.3 compiler_4.0.0 cli_2.0.2 shinyjs_1.1
[17] colourpicker_1.0 dygraphs_1.1.1.6 ggridges_0.5.2 callr_3.4.3 digest_0.6.25 minqa_1.2.4 base64enc_0.1-3 pkgconfig_2.0.3
[25] htmltools_0.4.0 lme4_1.1-23 fastmap_1.0.1 htmlwidgets_1.5.1 rlang_0.4.5 shiny_1.4.0.2 generics_0.0.2 crosstalk_1.1.0.1
[33] gtools_3.8.2 inline_0.3.15 loo_2.2.0 Matrix_1.2-18 munsell_0.5.0 fansi_0.4.1 lifecycle_0.2.0 stringi_1.4.6
[41] MASS_7.3-51.5 pkgbuild_1.0.6 plyr_1.8.6 grid_4.0.0 promises_1.1.0 crayon_1.3.4 miniUI_0.1.1.1 lattice_0.20-41
[49] splines_4.0.0 hms_0.5.3 ps_1.3.2 pillar_1.4.3 igraph_1.2.5 boot_1.3-24 markdown_1.1 ggsignif_0.6.0
[57] shinystan_2.5.0 reshape2_1.4.4 codetools_0.2-16 stats4_4.0.0 glue_1.4.0 packrat_0.5.0 nloptr_1.2.2.1 vctrs_0.2.4
[65] httpuv_1.5.2 gtable_0.3.0 getopt_1.20.3 purrr_0.3.3 assertthat_0.2.1 mime_0.9 xtable_1.8-4 later_1.0.0
[73] survival_3.1-12 rsconnect_0.8.16 tibble_3.0.1 shinythemes_1.1.2 statmod_1.4.34 ellipsis_0.3.0

many thanks for your help
Mark

This is an RStudio bug

It should run with multiple cores if either you knit, run R from the Terminal, or tell RStudio to run your script as a separate job. If RStudio can’t fix this soon, we’ll put a workaround for it into rstan.

1 Like

Many thanks Ben,
Mark

Or the R GUI for Mac should work

Sorry Ben not having much luck-- is this
not an OpenMP problem in data.table - get this now on Mac GUI–
but stopped I think for a different reason…

  > library(data.table)

  data.table 1.12.8 using 1 threads (see ?getDTthreads).  Latest

news: r-datatable.com

I don’t know if the data.table package has any incompatibilities with the new R stuff.

I also was having issues with multiple chains running in parallel in MacOS using latest R-4.0.0 and RStan.

I’ve found out that this is a problem of the package parallel from R-4.0.0

I’ve found this recently issue in RStudio’s GitHub that solved my problem:


cl <- parallel::makeCluster(2, setup_strategy = "sequential")
If this works, you could add the following to your ~/.Rprofile until this is fixed in R/RStudio:

## WORKAROUND: https://github.com/rstudio/rstudio/issues/6692
## Revert to 'sequential' setup of PSOCK cluster in RStudio Console on macOS and R 4.0.0
if (Sys.getenv("RSTUDIO") == "1" && !nzchar(Sys.getenv("RSTUDIO_TERM")) && 
    Sys.info()["sysname"] == "Darwin" && getRversion() == "4.0.0") {
  parallel:::setDefaultClusterOptions(setup_strategy = "sequential")
}

Hope this helps clarifying the issues surrounding RStan, MacOS and R-4.0.0

10 Likes

Thank you very much- you also solved my problem as well- after three weeks of pain with various Rstan, R and Rstudio problems I can get back to work!