Problems running brms models with cores > 4

  • Operating System: OS High Sierra
  • brms Version: 2.11.1

Hi there,

I love brms, and have been using it weekly for a couple of years now. Recently, I have had a sporadic issue when I run a model with e.g. 4 chains across 4 cores, on my fairly good Mac desktop (32GB RAM). The problem is that the model runs fine, but the execution never completes after the chains have all finished. That is, I get the notifications from Stan in R’s console that all 4 chains have ended, but the code never completes and returns to the > prompt, and I need to force-quit R. If I run the same model with cores = 1, everything works fine. This occurs when running even basic models using all the brms defaults except for cores > 1. I enclose my sessionInfo and the contents of ~/.R/Makevars (I have a poor understanding of what is in there - I only edit it when trying to load odd packages and programs, and this could be the issue). Grateful for any ideas, as the multicore functionality is really handy, and until recently (I’m not sure when) it has always “just worked”, even for far bigger models (in terms of RAM, complexity, and/or number of iterations) than I am currently working on.

All the best!

Luke

My makevars:

This file’s location is ~/.R/Makevars

Default variables (no omp support):

CXX=clang++

CC=clang

CC="/usr/local/opt/llvm/bin/clang"
CXX="/usr/local/opt/llvm/bin/clang++"
CXX14FLAGS=-O3 -march=native -mtune=native
CXX14="/usr/local/opt/llvm/bin/clang++"

My session info:

R version 3.6.2 (2019-12-12)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.6

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_AU.UTF-8/en_AU.UTF-8/en_AU.UTF-8/C/en_AU.UTF-8/en_AU.UTF-8

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] kableExtra_1.1.0 gridExtra_2.3 brms_2.11.1 Rcpp_1.0.3 tidybayes_2.0.1 ggbeeswarm_0.6.0 forcats_0.4.0
[8] stringr_1.4.0 dplyr_0.8.4 purrr_0.3.3 readr_1.3.1 tidyr_1.0.2 tibble_2.1.3 ggplot2_3.2.1
[15] tidyverse_1.3.0

loaded via a namespace (and not attached):
[1] colorspace_1.4-1 ggridges_0.5.2 rsconnect_0.8.16 markdown_1.1 base64enc_0.1-3 fs_1.3.1
[7] rstudioapi_0.11 rstan_2.19.2 svUnit_0.7-12 DT_0.12 fansi_0.4.1 mvtnorm_1.0-12
[13] lubridate_1.7.4 xml2_1.2.2 bridgesampling_0.8-1 knitr_1.28 shinythemes_1.1.2 bayesplot_1.7.1
[19] jsonlite_1.6.1 packrat_0.5.0 broom_0.5.4 dbplyr_1.4.2 shiny_1.4.0 compiler_3.6.2
[25] httr_1.4.1 backports_1.1.5 assertthat_0.2.1 Matrix_1.2-18 fastmap_1.0.1 lazyeval_0.2.2
[31] cli_2.0.1 later_1.0.0 htmltools_0.4.0 prettyunits_1.1.1 tools_3.6.2 igraph_1.2.4.2
[37] coda_0.19-3 gtable_0.3.0 glue_1.3.1 reshape2_1.4.3 cellranger_1.1.0 vctrs_0.2.2
[43] nlme_3.1-144 crosstalk_1.0.0 xfun_0.12 ps_1.3.0 rvest_0.3.5 mime_0.9
[49] miniUI_0.1.1.1 lifecycle_0.1.0 gtools_3.8.1 zoo_1.8-7 scales_1.1.0 colourpicker_1.0
[55] hms_0.5.3 promises_1.1.0 Brobdingnag_1.2-6 parallel_3.6.2 inline_0.3.15 shinystan_2.5.0
[61] loo_2.2.0 StanHeaders_2.21.0-1 stringi_1.4.5 dygraphs_1.1.1.6 pkgbuild_1.0.6 rlang_0.4.4
[67] pkgconfig_2.0.3 matrixStats_0.55.0 evaluate_0.14 lattice_0.20-38 rstantools_2.0.0 htmlwidgets_1.5.1
[73] tidyselect_1.0.0 processx_3.4.2 plyr_1.8.5 magrittr_1.5 R6_2.4.1 generics_0.0.2
[79] DBI_1.1.0 pillar_1.4.3 haven_2.2.0 withr_2.1.2 xts_0.12-0 abind_1.4-5
[85] modelr_0.1.5 crayon_1.3.4 arrayhelpers_1.1-0 rmarkdown_2.1 grid_3.6.2 readxl_1.3.1
[91] callr_3.4.1 threejs_0.3.3 webshot_0.5.2 reprex_0.3.0 digest_0.6.23 xtable_1.8-4
[97] httpuv_1.5.2 stats4_3.6.2 munsell_0.5.0 viridisLite_0.3.0 beeswarm_0.2.3 vipor_0.4.5
[103] shinyjs_1.1

Hi Luke!

Are you also running into problems using rstan or rstanarm?

I’m definitely not the right person to solve this issue, but let’s see if we can get the right people in here to make this work for you again.

Cheers,
Max

Hi Max,

I haven’t tried those recently - I can’t write Stan code, and I like the extra complexity offered by brms over rstanarm. Also I just noticed a typo in the title of the post - it should say cores > 1. I can give rstanarm a try next week to help isolate the problem, that’s a good idea.

Cheers!

Luke, I experienced the same the other day and just killed everything. Yesterday before leaving work the same thing happened (brms GitHub d/l yesterday). I left it and when I came this morning I had the prompt waiting for me. Could be a regression in the code perhaps, but then I guess @paul.buerkner might know if anyone has reported this before?

1 Like