If I understand correctly, one should be able to use the kfold-helpers in the Loo 2 package to do a grouped k-fold cv. However, I do not understand exactly how to do it.
As a toy example, say that I have an experiment with some participants and I am interested in whether them being correct influences their response time.
I define the two models:
# Define null model rtm0 <- stan_glmer(rt~+(1|id), data=data, family=gaussian) # Define model with one predictor rtm1 <- update(rtm0,.~.+acc)
The classic way of comparing them would then be
### Compare with classic loo loo_rtm0 <- loo(rtm0) loo_rtm1 <- loo(rtm1) compare(loo_rtm0,loo_rtm1)
But if I understand correctly, this answers how the participants should respond if they are getting one additional trial. The more interesting question is how a new participant will do.
To test this, I hope to use the kfold_split_stratified function something like this:
### Compare with grouped k-fold cv kfold_split_stratified(K = nrparticipants, x = data$id)
But I am not sure what to do afterwards? Having gone through the vignettes and some googling has thus far not helped.