Cross-validation for hierarchical models (Replicating Aki's post)

Interfaces rstanarm
1

Hello,
I am preparing to perform some cross-validation analyses for some multilevel models using rstanarm. As a part of this, I am first mimicking the very helpful blog post by Aki found here:
https://avehtari.github.io/modelselection/rats_kcv.html#setup

I’ve been able to reproduce the code/results up through section 5.4 at which point, running the following code results in the following error:

cv10xfolds ← kfold_split_grouped(K = 10, x = dfrats$rat)
cv10xfolds[dfrats$age==8] ← 11
cv30xfolds ← kfold_split_grouped(K = 30, x = dfrats$rat)
cv30xfolds[dfrats$age==8] ← 31
cv10x_1 ← rstanarm::kfold(fit_1, K=10, folds = cv10xfolds)
Error in rstanarm::kfold(fit_1, K = 10, folds = cv10xfolds) :
all(folds %in% 1L:K) is not TRUE
cv10x_2 ← rstanarm::kfold(fit_2, K=10, folds = cv10xfolds)
Error in rstanarm::kfold(fit_2, K = 10, folds = cv10xfolds) :
all(folds %in% 1L:K) is not TRUE
cv10x_3 ← rstanarm::kfold(fit_3, K=10, folds = cv10xfolds)
Error in rstanarm::kfold(fit_3, K = 10, folds = cv10xfolds) :
all(folds %in% 1L:K) is not TRUE

I have run this in two different Windows computers running similar versions. My session info is below:
R version 3.5.2 (2018-12-20)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] rstanarm_2.18.2 bayesplot_1.6.0 ggplot2_3.1.0 Rcpp_1.0.0 loo_2.1.0.9000 rprojroot_1.3-2

loaded via a namespace (and not attached):
[1] splines_3.5.2 gtools_3.8.1 StanHeaders_2.18.1 threejs_0.3.1 shiny_1.2.0 assertthat_0.2.0
[7] stats4_3.5.2 yaml_2.2.0 pillar_1.3.1 backports_1.1.3 lattice_0.20-38 glue_1.3.1
[13] digest_0.6.18 promises_1.0.1 minqa_1.2.4 colorspace_1.4-0 htmltools_0.3.6 httpuv_1.4.5.1
[19] Matrix_1.2-15 plyr_1.8.4 dygraphs_1.1.1.6 pkgconfig_2.0.2 rstan_2.18.2 purrr_0.2.5
[25] xtable_1.8-3 scales_1.0.0 processx_3.3.0 later_0.8.0 lme4_1.1-21 tibble_2.0.1
[31] DT_0.5 withr_2.1.2 shinyjs_1.0 lazyeval_0.2.1 cli_1.0.1 survival_2.43-3
[37] magrittr_1.5 crayon_1.3.4 mime_0.6 ps_1.3.0 nlme_3.1-137 MASS_7.3-51.1
[43] xts_0.11-0 pkgbuild_1.0.2 colourpicker_1.0 prettyunits_1.0.2 rsconnect_0.8.13 tools_3.5.2
[49] matrixStats_0.54.0 stringr_1.3.1 munsell_0.5.0 callr_3.1.1 compiler_3.5.2 rlang_0.3.1
[55] grid_3.5.2 nloptr_1.2.1 ggridges_0.5.1 rstudioapi_0.9.0 htmlwidgets_1.3 crosstalk_1.0.0
[61] igraph_1.2.4 miniUI_0.1.1.1 base64enc_0.1-3 boot_1.3-20 codetools_0.2-15 gtable_0.2.0
[67] inline_0.3.15 markdown_0.9 reshape2_1.4.3 R6_2.2.2 gridExtra_2.3 rstantools_1.5.1
[73] zoo_1.8-3 dplyr_0.8.0.1 shinystan_2.5.0 shinythemes_1.1.2 stringi_1.1.7 parallel_3.5.2
[79] tidyselect_0.2.5

Any idea where the issue may lie? I understand the error, conceptually, but I’m not sure why I would be throwing it but Aki’s Rmarkdown document is not.

Thanks,
Matt

1 Like
2

Sorry that it’s not mentioned in the actual notebook, I only mentioned it in the twitter and discourse announcements New tutorial: Cross-validation for hierarchical/multilevel models that you need the latest version from github

3

Hi Aki,
I did notice that on Twitter, but it looked like the package versions in your environment:
loo_2.0.0.9000
rstanarm_2.18.2

Were earlier than my own, so I assumed I was safe:
rstanarm_2.18.2
loo_2.1.0.9000

Was that incorrect? I’m unable to load the github rstanarm version for reasons noted here:

I’m also not able to compile C code (i.e. use the github rstanarm) on my work computer, so maybe my reviewing my later versions of packages was false hope. Is there an expected timeline for the updates to hit CRAN?
Thanks again,
Matt

4

Version number doesn’t go up with each commit to git. There will be soon rstanarm 2.19 release

1 Like