I’m working on model selection for mixed models fit with brms. In most cases using loo works well, though I have to reloo sometimes. For at least one model, the number of reloos is quite high and I don’t seem to have enough memory to make it through. I’m thinking that using kfold might be best in those situations, but I’m not that excited about using kfold for all of my models (slow…).
Would it be appropriate to compare elpd values that were calculated differently for different candidate models? E.g. to compare one model that has elpd_loo to another with elpd_kfold?
If you can derive standard errors from elpd_kfold then you’re on equal footing with elpd_loo, which compares to within standard error intervals. See this post for understanding the role of the standard errors for the estimates.
Yes you can compare K-fold results to LOO results. If I remember correctly, we have the loo_compare() function throw a warning if you try to do this, basically telling you that you can get a more accurate comparison if you use LOO for all models or K-fold with the same number of folds for all models. But the situation where you have a lot of high Pareto k values for one of the models but not the other(s) (which sounds like your situation) is precisely when this approach can be useful.