Leave-one-group-out, but for multiple, different groups?

Hi all. Thankyou for your advice on my earlier question. I have a model for analyzing performance in a psycholinguistic experiment:

reactiontime ~ Condition + (Condition | SubjectID) + (Condition | StimuliID)

I am following the information in the loo vignettes - since I would wish to determine how well this model will predict for new subjects (not just the subjects in the experiment), I am using leave-one-group-out crossvalidation. But the issue is I would also like to determine how well the model will predict for new stimuli.

Is standard practice to run separate leave-one-group-out crossvalidation tests for each group (i.e. run one logo-cv for SubjectID and then a separate logo-cv for StimuliID) Is there any way of combining them?

Are there any research papers that in any field that anyone could point me towards that have an analysis which does two different leave-one-group-out cv analyses of the same model?

Thanks

1 Like

I think it makes complete sense to run both logo. They are going to answer different questions.I don’t think you combine them in one, since the grouping is different.

Thankyou!

you’re welcome!, by the way, I added the tag cognitive-science to your post. And after you submit a preprint, you can add your paper to this website I’m maintaining: https://cognitive-science-stan.github.io/

2 Likes