Looking at the new rstanarm & loo package I am really pleased to see that we can now run k-fold cross-validation with the folds chosen by the user. I am interested in hierarchical models and with those I would like stratify the k-folds by the units the hierarchical model is wrt (the patients in my case).
Now, my question: Can I also apply the loo evaluation for rstanarm models on the patient level? So instead of definig log_lik per data point I would like to sum together the log_lik values for each patient and use that. Doing it on the data-point level answers the question “Do we fit observed data from patients well?” while I am also interested in “Do we fit patients well?”.
One can probably do this with some manual work, I suppose… but can rstanarm “just do it” with some cool feature (I am asking for a lot, I know).
Bonus question: If my patients have different number of observations will this be a problem in whatever regard?
Thanks! Looking forward to use this toolset.
Please also provide the following information in addition to your question:
- Operating System:
- rstanarm Version: