Problems running LOO on >1 core

General
1

Evening all,

I ran into a strange bug I’m hoping someone knows the answer to. I’m trying to use the loo package to diagnose my models, but for some reason I can only run the commands on a single core, like so:
loo1 <- loo(Main_EffectsModel, save_psis = TRUE, cores = 1)
loo2 <- loo(Interaction_Model, save_psis = TRUE, cores = 1)
loo_compare(loo1,loo2)

If I change the “cores” option to anything greater than 1, I get the following error code:
Error in get(name, envir = envir) : object 'draws' not found

Any chance anyone knows why specifying >1 core is causing this command to break?

2

I am having a similar issue with kfold. I am posting here for the first time because I just can’t figure this out! (apologies - I’m sure I’m going to get something wrong about documenting what’s happening)

Here’s the error I get - something wrong with the parallelization -

Fitting K = 10 models distributed over 10 cores
Error in checkForRemoteErrors(val) : 
    10 nodes produced errors; first error: object 't_prior' not found

I run rstanarm on a cluster through a Windows remote desktop. I ran a couple models last week before the cluster was taken through monthly maintenance overnight Friday. When I got back on, R was updated to 3.6.3 instead of 3.6.2, and rstan, StanHeaders, and rstanarm were all updated. The cluster has 10 machines. On one of them, I can run my kfold commands:

folds_Period <- loo::kfold_split_stratified(K = 10, 
                                        x = model_data_scaled$Period15yrs) 
options(mc.cores = 10)
fit_observing_kfold <- kfold(fit_observing, folds = folds_Period)

That machine is working using StanHeaders 2.18.1-10, rstan 2.19.2, rstanarm 2.18.2. When I try to run on the other ones StanHeaders 2.19.2, rstan 2.19.3, rstanarm 2.19.3, my kfold command works only with specifying cores = 1.
If I run

fit_hybrid_kfold <- kfold(fit_hybrid, folds = folds_Period)
Fitting K = 10 models distributed over 10 cores

it silently fails, and pretty quickly. No error message or anything, just a new > ready to go!

Thanks for any advice! Please let me know what other info would be helpful! Hopefully it’s some novice mistake…

- Session info ---------------------------------------------------------
 setting  value                       
 version  R version 3.6.2 (2019-12-12)
 os       Windows Server x64          
 system   x86_64, mingw32             
 ui       RStudio                     
 language (EN)                        
 collate  English_United States.1252  
 ctype    English_United States.1252  
 tz       America/Los_Angeles         
 date     2020-03-29                  

- Packages -------------------------------------------------------------
 ! package        * version  date       lib source        
   abind          * 1.4-5    2016-07-21 [1] CRAN (R 3.6.0)
   assertthat       0.2.1    2019-03-21 [1] CRAN (R 3.6.1)
   backports        1.1.5    2019-10-02 [1] CRAN (R 3.6.1)
   base64enc        0.1-3    2015-07-28 [1] CRAN (R 3.6.0)
   bayesplot      * 1.7.1    2019-12-01 [1] CRAN (R 3.6.3)
   bayestestR       0.5.2    2020-02-12 [1] CRAN (R 3.6.3)
   boot             1.3-23   2019-07-05 [1] CRAN (R 3.6.2)
   broom            0.5.3    2019-12-14 [1] CRAN (R 3.6.2)
 V callr            3.4.0    2020-03-28 [1] CRAN (R 3.6.2)
   cellranger       1.1.0    2016-07-27 [1] CRAN (R 3.6.1)
   cli              2.0.2    2020-02-28 [1] CRAN (R 3.6.2)
   coda             0.19-3   2019-07-05 [1] CRAN (R 3.6.1)
   codetools        0.2-16   2018-12-24 [1] CRAN (R 3.6.2)
   colorspace       1.4-1    2019-03-18 [1] CRAN (R 3.6.1)
   colourpicker     1.0      2017-09-27 [1] CRAN (R 3.6.3)
   corrplot       * 0.84     2017-10-16 [1] CRAN (R 3.6.3)
   cowplot        * 1.0.0    2019-07-11 [1] CRAN (R 3.6.1)
   crayon           1.3.4    2017-09-16 [1] CRAN (R 3.6.1)
   crosstalk        1.0.0    2016-12-21 [1] CRAN (R 3.6.1)
   DBI              1.1.0    2019-12-15 [1] CRAN (R 3.6.2)
   dbplyr           1.4.2    2019-06-17 [1] CRAN (R 3.6.2)
   deldir           0.1-23   2019-07-31 [1] CRAN (R 3.6.1)
   desc             1.2.0    2018-05-01 [1] CRAN (R 3.6.1)
   devtools       * 2.2.2    2020-02-17 [1] CRAN (R 3.6.3)
   digest           0.6.25   2020-02-23 [1] CRAN (R 3.6.3)
   dplyr          * 0.8.5    2020-03-07 [1] CRAN (R 3.6.3)
   DT               0.10     2019-11-12 [1] CRAN (R 3.6.1)
   dygraphs         1.1.1.6  2018-07-11 [1] CRAN (R 3.6.3)
   effectsize       0.2.0    2020-02-25 [1] CRAN (R 3.6.3)
   ellipsis         0.3.0    2019-09-20 [1] CRAN (R 3.6.1)
   emmeans          1.4.5    2020-03-04 [1] CRAN (R 3.6.3)
   estimability     1.3      2018-02-11 [1] CRAN (R 3.6.0)
   evaluate         0.14     2019-05-28 [1] CRAN (R 3.6.1)
   fansi            0.4.1    2020-01-08 [1] CRAN (R 3.6.2)
   fastmap          1.0.1    2019-10-08 [1] CRAN (R 3.6.1)
   forcats        * 0.4.0    2019-02-17 [1] CRAN (R 3.6.1)
   fs               1.3.1    2019-05-06 [1] CRAN (R 3.6.1)
   gdistance      * 1.3-1    2020-02-29 [1] CRAN (R 3.6.3)
   generics         0.0.2    2018-11-29 [1] CRAN (R 3.6.1)
   geosphere      * 1.5-10   2019-05-26 [1] CRAN (R 3.6.3)
   ggeffects      * 0.14.2   2020-03-14 [1] CRAN (R 3.6.3)
   ggplot2        * 3.3.0    2020-03-05 [1] CRAN (R 3.6.2)
   ggridges         0.5.2    2020-01-12 [1] CRAN (R 3.6.3)
   glue             1.3.2    2020-03-12 [1] CRAN (R 3.6.3)
   goftest          1.2-2    2019-12-02 [1] CRAN (R 3.6.1)
   gridExtra      * 2.3      2017-09-09 [1] CRAN (R 3.6.1)
   gtable           0.3.0    2019-03-25 [1] CRAN (R 3.6.1)
   gtools           3.8.1    2018-06-26 [1] CRAN (R 3.6.0)
   haven            2.2.0    2019-11-08 [1] CRAN (R 3.6.1)
   here           * 0.1      2017-05-28 [1] CRAN (R 3.6.1)
   hms              0.5.2    2019-10-30 [1] CRAN (R 3.6.1)
   htmltools        0.4.0    2019-10-04 [1] CRAN (R 3.6.1)
   htmlwidgets      1.5.1    2019-10-08 [1] CRAN (R 3.6.1)
   httpuv           1.5.2    2019-09-11 [1] CRAN (R 3.6.1)
   httr             1.4.1    2019-08-05 [1] CRAN (R 3.6.1)
   igraph         * 1.2.4.2  2019-11-27 [1] CRAN (R 3.6.3)
   inline           0.3.15   2018-05-18 [1] CRAN (R 3.6.3)
   insight          0.8.2    2020-03-06 [1] CRAN (R 3.6.3)
   jsonlite         1.6.1    2020-02-02 [1] CRAN (R 3.6.3)
   kableExtra     * 1.1.0    2019-03-16 [1] CRAN (R 3.6.3)
   knitr          * 1.28     2020-02-06 [1] CRAN (R 3.6.3)
   later            1.0.0    2019-10-04 [1] CRAN (R 3.6.1)
   lattice          0.20-40  2020-02-19 [1] CRAN (R 3.6.3)
 V lifecycle        0.1.0    2020-03-06 [1] CRAN (R 3.6.2)
   lme4             1.1-21   2019-03-05 [1] CRAN (R 3.6.1)
   loo              2.2.0    2019-12-19 [1] CRAN (R 3.6.3)
   lubridate        1.7.4    2018-04-11 [1] CRAN (R 3.6.2)
   magrittr       * 1.5      2014-11-22 [1] CRAN (R 3.6.1)
   markdown         1.1      2019-08-07 [1] CRAN (R 3.6.1)
   MASS             7.3-51.5 2019-12-20 [1] CRAN (R 3.6.3)
   Matrix         * 1.2-18   2019-11-27 [1] CRAN (R 3.6.2)
   matrixStats      0.56.0   2020-03-13 [1] CRAN (R 3.6.2)
   memoise          1.1.0    2017-04-21 [1] CRAN (R 3.6.1)
   mgcv             1.8-31   2019-11-09 [1] CRAN (R 3.6.2)
   mime             0.7      2019-06-11 [1] CRAN (R 3.6.0)
   miniUI           0.1.1.1  2018-05-18 [1] CRAN (R 3.6.2)
   minqa            1.2.4    2014-10-09 [1] CRAN (R 3.6.1)
   modelr           0.1.5    2019-08-08 [1] CRAN (R 3.6.2)
   multcomp         1.4-11   2019-12-10 [1] CRAN (R 3.6.1)
   munsell          0.5.0    2018-06-12 [1] CRAN (R 3.6.1)
   mvtnorm          1.0-11   2019-06-19 [1] CRAN (R 3.6.0)
   ncdf4          * 1.17     2019-10-23 [1] CRAN (R 3.6.1)
   nlme           * 3.1-145  2020-03-04 [1] CRAN (R 3.6.3)
   nloptr           1.2.2.1  2020-03-11 [1] CRAN (R 3.6.3)
   packrat          0.5.0    2018-11-14 [1] CRAN (R 3.6.3)
   pacman         * 0.5.1    2019-03-11 [1] CRAN (R 3.6.3)
   pander         * 0.6.3    2018-11-06 [1] CRAN (R 3.6.2)
   parameters       0.6.0    2020-03-12 [1] CRAN (R 3.6.3)
   performance      0.4.5    2020-03-28 [1] CRAN (R 3.6.2)
   pillar           1.4.3    2019-12-20 [1] CRAN (R 3.6.2)
   pkgbuild         1.0.6    2019-10-09 [1] CRAN (R 3.6.1)
   pkgconfig        2.0.3    2019-09-22 [1] CRAN (R 3.6.1)
   pkgload          1.0.2    2018-10-29 [1] CRAN (R 3.6.1)
   plyr           * 1.8.6    2020-03-03 [1] CRAN (R 3.6.3)
   polyclip         1.10-0   2019-03-14 [1] CRAN (R 3.6.0)
   prettyunits      1.1.1    2020-01-24 [1] CRAN (R 3.6.2)
   processx       * 3.4.2    2020-02-09 [1] CRAN (R 3.6.3)
   promises         1.1.0    2019-10-04 [1] CRAN (R 3.6.1)
   ps               1.3.2    2020-02-13 [1] CRAN (R 3.6.3)
   purrr          * 0.3.3    2019-10-18 [1] CRAN (R 3.6.1)
   R6               2.4.1    2019-11-12 [1] CRAN (R 3.6.1)
   raster         * 3.0-7    2019-09-24 [1] CRAN (R 3.6.2)
   Rcpp           * 1.0.4    2020-03-17 [1] CRAN (R 3.6.3)
   readr          * 1.3.1    2018-12-21 [1] CRAN (R 3.6.1)
   readxl           1.3.1    2019-03-13 [1] CRAN (R 3.6.1)
   remotes          2.1.1    2020-02-15 [1] CRAN (R 3.6.3)
   reprex           0.3.0    2019-05-16 [1] CRAN (R 3.6.2)
   reshape2       * 1.4.3    2017-12-11 [1] CRAN (R 3.6.1)
   rgdal          * 1.4-8    2019-11-27 [1] CRAN (R 3.6.2)
   rgeos          * 0.5-2    2019-10-03 [1] CRAN (R 3.6.3)
   riverdist      * 0.15.0   2017-07-18 [1] CRAN (R 3.6.3)
   rlang            0.4.5    2020-03-01 [1] CRAN (R 3.6.3)
   rmarkdown        2.0      2019-12-12 [1] CRAN (R 3.6.2)
   rpart          * 4.1-15   2019-04-12 [1] CRAN (R 3.6.2)
   rprojroot        1.3-2    2018-01-03 [1] CRAN (R 3.6.1)
   rsconnect        0.8.16   2019-12-13 [1] CRAN (R 3.6.3)
   rstan          * 2.19.3   2020-02-11 [1] CRAN (R 3.6.3)
   rstanarm       * 2.19.3   2020-02-11 [1] CRAN (R 3.6.3)
   rstantools       2.0.0    2019-09-15 [1] CRAN (R 3.6.3)
   rstudioapi       0.11     2020-02-07 [1] CRAN (R 3.6.2)
   rvest            0.3.5    2019-11-08 [1] CRAN (R 3.6.1)
   sandwich         2.5-1    2019-04-06 [1] CRAN (R 3.6.1)
   scales           1.1.0    2019-11-18 [1] CRAN (R 3.6.1)
   sessioninfo      1.1.1    2018-11-05 [1] CRAN (R 3.6.1)
   shiny            1.4.0    2019-10-10 [1] CRAN (R 3.6.1)
   shinyjs          1.1      2020-01-13 [1] CRAN (R 3.6.3)
   shinystan        2.5.0    2018-05-01 [1] CRAN (R 3.6.3)
   shinythemes      1.1.2    2018-11-06 [1] CRAN (R 3.6.3)
   sjlabelled       1.1.3    2020-01-28 [1] CRAN (R 3.6.3)
   sjmisc           2.8.3    2020-01-10 [1] CRAN (R 3.6.3)
   sjPlot         * 2.8.3    2020-03-09 [1] CRAN (R 3.6.3)
   sjstats        * 0.17.9   2020-02-06 [1] CRAN (R 3.6.3)
   sp             * 1.3-2    2019-11-07 [1] CRAN (R 3.6.1)
   spatstat       * 1.62-2   2019-12-10 [1] CRAN (R 3.6.2)
   spatstat.data  * 1.4-0    2018-10-04 [1] CRAN (R 3.6.2)
   spatstat.utils   1.15-0   2019-12-02 [1] CRAN (R 3.6.2)
   StanHeaders    * 2.19.2   2019-09-07 [1] CRAN (R 3.6.3)
   stargazer      * 5.2.2    2018-05-30 [1] CRAN (R 3.6.0)
   stringi          1.4.6    2020-02-17 [1] CRAN (R 3.6.2)
   stringr        * 1.4.0    2019-02-10 [1] CRAN (R 3.6.1)
   survival         3.1-11   2020-03-07 [1] CRAN (R 3.6.3)
   tensor           1.5      2012-05-05 [1] CRAN (R 3.6.0)
   testthat         2.3.2    2020-03-02 [1] CRAN (R 3.6.3)
   texreg         * 1.36.23  2017-03-03 [1] CRAN (R 3.6.3)
   TH.data          1.0-10   2019-01-21 [1] CRAN (R 3.6.1)
   threejs          0.3.3    2020-01-21 [1] CRAN (R 3.6.3)
   tibble         * 2.1.3    2019-06-06 [1] CRAN (R 3.6.1)
   tidyr          * 1.0.2    2020-01-24 [1] CRAN (R 3.6.3)
   tidyselect       1.0.0    2020-01-27 [1] CRAN (R 3.6.2)
   tidyverse      * 1.3.0    2019-11-21 [1] CRAN (R 3.6.3)
   usethis        * 1.5.1    2019-07-04 [1] CRAN (R 3.6.3)
   vctrs            0.2.4    2020-03-10 [1] CRAN (R 3.6.3)
   viridisLite      0.3.0    2018-02-01 [1] CRAN (R 3.6.1)
   webshot          0.5.2    2019-11-22 [1] CRAN (R 3.6.1)
   withr            2.1.2    2018-03-15 [1] CRAN (R 3.6.1)
   xfun             0.12     2020-01-13 [1] CRAN (R 3.6.3)
   xml2             1.2.5    2020-03-11 [1] CRAN (R 3.6.3)
   xtable           1.8-4    2019-04-21 [1] CRAN (R 3.6.1)
   xts              0.12-0   2020-01-19 [1] CRAN (R 3.6.3)
   zoo            * 1.8-7    2020-01-10 [1] CRAN (R 3.6.3)

[1] C:/Program Files/R/R-3.6.2/library

 V -- Loaded and on-disk version mismatch.
3

There have been several reports that Windows fails running parallel jobs. Unfortunately, in most cases it’s something we cannot fix in R except by disabling parallel for all Windows users :(
Please report if in this case you find out something more

4

That’s unfortunate :( If I can find out more I will let you know.

Is this an R problem generally with parallel, or something specific to rstanarm? I was thinking about taking a Data Camp class on how to run parallel code recently, so if it’s the former I may not bother.

5

Thanks for the tip, @avehtari!

@Longshot408, I was able to get kfold to run (on Windows Server 16) in R 3.6.3 by reverting to prior rstan etc versions - StanHeaders 2.18.1-10, rstan 2.19.3, rstanarm 2.18.2. So the Windows problem isn’t 3.6.3 generally (as I mentioned, I had gotten it to run again on 3.6.2), nor is it rstan 2.19.3. I have several of these I want to run, so I’m going to see if I can get my code to run with StanHeaders, rstan, and rstanarm all on 2.19.xx. I’ll update again if I find other configurations that work.

6

It seems like the Windows parallel issue might be with rstanarm 2.19.xx

For me, kfold IS running in parallel on Windows, R 3.6.3 under:

  • StanHeaders 2.18.1-10, rstan 2.19.2, rstanarm 2.18.2
  • StanHeaders 2.19.2, rstan 2.19.2, rstanarm 2.18.2

kfold IS NOT running in parallel under:

  • StanHeaders 2.19.2, rstan 2.19.3, rstanarm 2.19.3
  • StanHeaders 2.19.2, rstan 2.19.3, rstanarm 2.19.2
  • StanHeaders 2.18.1-10, rstan 2.19.3, rstanarm 2.19.2

I followed the directions to install rstan from source: https://github.com/stan-dev/rstan/wiki/Installing-RStan-from-source-on-Windows#configuration

I removed my rstan, rstanarm, and StanHeaders. I reinstalled rstan, then StanHeaders, then rstanarm for the specific versions I wanted. I did not update any packages during the install process when prompted.

2 Likes
7

@bgoodri and @jonah might remember more about Windows parallel problems

8

I am having this same issue with R 4.0.2, rstanarm 2.21.1 and loo 2.3.1 (the current versions of each package as of right now) - loo will run on 1 core but not multiple cores, and when you try multiple cores it spits out the same error message “Error in get(name, envir = envir) : object ‘draws’ not found”. Did anyone ever figure out this problem?

9

Hmm, sorry you’re running into that error. Is there any more to the error message than that? Or can you run traceback() after you get the error and see if there’s more info?

Also @paul.buerkner are you also having trouble running it in parallel on Windows?

10

I encounter the same issue. No other error message than the one provided by Tim_Tinker. I first encountered the problem working with stan_polr, but I had the same issue when attempting the examples here: http://mc-stan.org/rstanarm/reference/loo.stanreg.html. Setting mc.cores = 1 solved the problem.

In case it’s helpful, here is some platform and version info:
R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18363)

other attached packages:
[1] loo_2.3.1 bayesplot_1.7.1 rstanarm_2.21.1 Rcpp_1.0.4.6 rstan_2.21.2
[6] StanHeaders_2.21.0-5 readxl_1.3.1 forcats_0.5.0 stringr_1.4.0 dplyr_0.8.5
[11] purrr_0.3.4 readr_1.3.1 tidyr_1.1.0 tibble_3.0.1 ggplot2_3.3.0
[16] tidyverse_1.3.0

11

I just tried updating R and all packages to the latest version, and I’m still encountering what the others have posted recently. I ran the traceback command and warnings, and this is what I got:

> loo3=loo(Main_EffectsModel, save_psis=TRUE,cores=16)

Error in get(name, envir = envir) : object 'draws' not found

traceback()
12: get(name, envir = envir)
11: serialize(data, node$con)
10: sendData.SOCKnode(con, list(type = type, data = value, tag = tag))
9: sendData(con, list(type = type, data = value, tag = tag))
8: postNode(con, “EXEC”, list(fun = fun, args = args, return = return,
tag = tag))
7: sendCall(cl[[i]], fun, list(…))
6: clusterCall(cl, gets, name, get(name, envir = envir))
5: parallel::clusterExport(cl, “draws”)
4: relative_eff.function(x = likfun, chain_id = chain_id, data = args$data,
draws = args$draws, cores = cores, …)
3: loo::relative_eff(x = likfun, chain_id = chain_id, data = args$data,
draws = args$draws, cores = cores, …)
2: loo.stanreg(Main_EffectsModel, save_psis = TRUE, cores = 16)
1: loo(Main_EffectsModel, save_psis = TRUE, cores = 16)
There were 16 warnings (use warnings() to see them)
warnings()
Warning messages:
1: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 19 (<-DESKTOP-NIBF9LB:11815)
2: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 18 (<-DESKTOP-NIBF9LB:11815)
3: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 17 (<-DESKTOP-NIBF9LB:11815)
4: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 16 (<-DESKTOP-NIBF9LB:11815)
5: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 15 (<-DESKTOP-NIBF9LB:11815)
6: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 14 (<-DESKTOP-NIBF9LB:11815)
7: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 13 (<-DESKTOP-NIBF9LB:11815)
8: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 12 (<-DESKTOP-NIBF9LB:11815)
9: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 11 (<-DESKTOP-NIBF9LB:11815)
10: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 10 (<-DESKTOP-NIBF9LB:11815)
11: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 9 (<-DESKTOP-NIBF9LB:11815)
12: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 8 (<-DESKTOP-NIBF9LB:11815)
13: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 7 (<-DESKTOP-NIBF9LB:11815)
14: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 6 (<-DESKTOP-NIBF9LB:11815)
15: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 5 (<-DESKTOP-NIBF9LB:11815)
16: In .Internal(grep(as.character(pattern), x, ignore.case, … :
closing unused connection 4 (<-DESKTOP-NIBF9LB:11815)

12

It does work with brms at least so may not be an issue of loo necessarily but perhaps of rstanarm?

13

Hmm, yeah possibly an rstanarm issue then. I’ll look into it but unfortunately I don’t have access to a Windows machine at the moment, so I might not be able to reproduce this.

14

Because I don’t have access to Windows at the moment, it would help me a lot to if someone who is getting this error could try running the code below. This should help me isolate whether this is a problem in rstanarm or in the loo package:

library(rstanarm)
fit <- stan_glm(mpg ~ wt, data = mtcars, chains = 1, iter = 1000)

loo(fit, cores = 2) # this will probably error since cores > 1

# now manually extract log_lik, compute r_eff, and run loo.matrix
loglik <- log_lik(fit)
reff <- loo::relative_eff(loglik, chain_id = rep(1, 500), cores = 2)
loo::loo.matrix(loglik, r_eff = reff, cores = 2)

Question: do either of the last two lines (loo::relative_eff or loo::loo_matrix) result in the error?

And one more code chunk to run if you don’t mind (to test the loo.function method):

# Simulate data and draw from posterior
N <- 50; K <- 10; S <- 100; a0 <- 3; b0 <- 2
p <- rbeta(1, a0, b0)
y <- rbinom(N, size = K, prob = p)
a <- a0 + sum(y); b <- b0 + N * K - sum(y)
fake_posterior <- as.matrix(rbeta(S, a, b))
dim(fake_posterior) # S x 1
fake_data <- data.frame(y,K)
dim(fake_data) # N x 2

llfun <- function(data_i, draws) {
  # each time called internally within loo the arguments will be equal to:
  # data_i: ith row of fake_data (fake_data[i,, drop=FALSE])
  # draws: entire fake_posterior matrix
  dbinom(data_i$y, size = data_i$K, prob = draws, log = TRUE)
}

reff <- loo::relative_eff(llfun, chain_id = rep(1, S), cores = 2, 
                          data = fake_data, draws = fake_posterior)
loo_with_fn <- loo::loo.function(llfun, r_eff = reff, cores = 2, 
                                 draws = fake_posterior, data = fake_data)

Do either of the last two lines here result in the error?

Thanks a lot and hopefully this will help us fix this!

15

@paul.buerkner This seems similar to this loo issue

And also, I glanced at the brms code and it seems like you’re using the matridx method for loo::relative_eff() and not the function method. Is that correct? If so that’s a difference between brms and rstanarm here. So it’s possible this is all coming from the loo::relative_eff.function() method, like we saw in that issue I linked to.

16

Here’s the first block:

fit ← stan_glm(mpg ~ wt, data = mtcars, chains = 1, iter = 1000)

> SAMPLING FOR MODEL 'continuous' NOW (CHAIN 1).
> Chain 1: 
> Chain 1: Gradient evaluation took 0 seconds
> Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 0 seconds.
> Chain 1: Adjust your expectations accordingly!
> Chain 1: 
> Chain 1: 
> Chain 1: Iteration:   1 / 1000 [  0%]  (Warmup)
> Chain 1: Iteration: 100 / 1000 [ 10%]  (Warmup)
> Chain 1: Iteration: 200 / 1000 [ 20%]  (Warmup)
> Chain 1: Iteration: 300 / 1000 [ 30%]  (Warmup)
> Chain 1: Iteration: 400 / 1000 [ 40%]  (Warmup)
> Chain 1: Iteration: 500 / 1000 [ 50%]  (Warmup)
> Chain 1: Iteration: 501 / 1000 [ 50%]  (Sampling)
> Chain 1: Iteration: 600 / 1000 [ 60%]  (Sampling)
> Chain 1: Iteration: 700 / 1000 [ 70%]  (Sampling)
> Chain 1: Iteration: 800 / 1000 [ 80%]  (Sampling)
> Chain 1: Iteration: 900 / 1000 [ 90%]  (Sampling)
> Chain 1: Iteration: 1000 / 1000 [100%]  (Sampling)
> Chain 1: 
> Chain 1:  Elapsed Time: 0.106 seconds (Warm-up)
> Chain 1:                0.065 seconds (Sampling)
> Chain 1:                0.171 seconds (Total)
> Chain 1: 
> > loo(fit, cores = 2) # this will probably error since cores > 1
> Error in get(name, envir = envir) : object 'draws' not found
> > loglik <- log_lik(fit)
> > reff <- loo::relative_eff(loglik, chain_id = rep(1, 500), cores = 2)
> > loo::loo.matrix(loglik, r_eff = reff, cores = 2)
> 
> Computed from 500 by 32 log-likelihood matrix
> 
>          Estimate  SE
> elpd_loo    -83.6 4.2
> p_loo         3.3 1.2
> looic       167.2 8.5
> ------
> Monte Carlo SE of elpd_loo is 0.1.
> 
> Pareto k diagnostic values:
>                          Count Pct.    Min. n_eff
> (-Inf, 0.5]   (good)     31    96.9%   161       
>  (0.5, 0.7]   (ok)        1     3.1%   146       
>    (0.7, 1]   (bad)       0     0.0%   <NA>      
>    (1, Inf)   (very bad)  0     0.0%   <NA>      
> 
> All Pareto k estimates are ok (k < 0.7).
> See help('pareto-k-diagnostic') for details.
> Warning messages:
> 1: Some Pareto k diagnostic values are slightly high. See help('pareto-k-diagnostic') for details.
>  
> 2: In for (i in seq_len(n)) { :
>   closing unused connection 5 (<-DESKTOP-NIBF9LB:11494)
> 3: In for (i in seq_len(n)) { :
>   closing unused connection 4 (<-DESKTOP-NIBF9LB:11494)

It looks like that actually worked! I’m going to run the second block in a minute

17

Less luck with the second block, I got the error again:

> reff <- loo::relative_eff(llfun, chain_id = rep(1, S), cores = 2, 
+                           data = fake_data, draws = fake_posterior)
Error in get(name, envir = envir) : object 'draws' not found
> loo_with_fn <- loo::loo.function(llfun, r_eff = reff, cores = 2, 
+                                  draws = fake_posterior, data = fake_data)
Error: 'r_eff' must have one value per observation.
18

Thanks @Longshot408, that’s super helpful. I think these results confirm my suspicion about the loo:::relative_eff.function() method. Seems like this problem again

but the previous fix wasn’t actually a fix apparently.

@paul.buerkner rstanarm is calling the function method of relative_eff and I think brms is not, so that explains why this only seems to affect rstanarm. I can change to the matrix method in rstanarm temporarily but hopefully we can figure out how to fix this in loo. Any ideas why the fix when we closed that issue seems not to work in some cases?

1 Like
19

Glad I could help. Is there any known way to fix?

I’m a little bummed to find out that the extra cores I paid for when I upgraded my CPU this spring can’t really be used :(

20

So a short term solution for anyone affected by this is to extract the log likelihood matrix from the fitted model object (rstanarm stanreg object) using log_lik(fit) and pass that to loo::loo.matrix().

1 Like