First of all, thank you to the developers of BRMS for creating such an accessible package for Bayesian analyses!
I am currently attempting to perform model comparison between 9 models using K-fold cross validation (K=10) because performing LOO yields a high number of Pareto K > 0.7.
Presumably on account of the random data selection using the K-fold process, the ranked order of the models changes if one compares one run of the K-fold validation for all models to another (examples attached, where g.m1 is the simplest model). Thus, I was wondering if there is a way to “average” across multiple K-fold model comparisons in BRMS, which I believe is a process known as “Repeated k-Fold Cross-Validation” in other statistical softwares? I should note that, in most cases, the most simple model is not significantly different from the “best”-performing model. Ideally, my goal is to present an averaged ELPD_diff and SE_diff model comparison to readers. If there is no way to average across K-fold comparisons, are there any other recommended approaches for how to deal with randomness in the ELPD order of selected models?
Thanks again and please let me know if more information is required!