I’m trying to apply LOO-CV approximated by PSIS as an information criterion for a multilevel model of stock returns (with each security as a group). I have been thinking that it makes more sense to perform leave-one-group-out cross-validation instead of LOO because of the multilevel nature of the model, but I haven’t been able to find any good resources on the topic. Unfortunately there are roughly 3,000 groups in the model, each with 300 or so observations, which makes re-fitting the model extremely costly (a single model takes 4 days to fit).
Is this a problem anyone has encountered before? Is there a way to do this quickly, or should I just rely on approximate LOO to compare the models I’m fitting? The Pareto shape diagnostics look fine (below 0.5) when fitting it that way.
Currently I’m fitting my model with BRMS, but I’m relatively agnostic about the interface or package.
If you want to predict the future performance, you should consider also leave-future-out cross-validation as discussed in https://arxiv.org/abs/1902.06281
You can try PSIS for leave-one-group-out cross-validation, too. Just sum the log_lik values in each group and then use loo package as usual. This works if the observations for each group are not very informative compared to the hierarchical prior.
You could also use sub-sampling LOGO, by leaving only some of the groups. See subsampling loo paper https://arxiv.org/abs/1904.10679 and just replace loo with logo
Thanks so much for the references! I have seen the paper on leaving out future observations. I am also planning to test the time-series aspect, but right now I’m more focused on cross-sectional prediction since most existing academic financial models look at the cross-section.
I hadn’t seen the second one, but it looks extremely useful. I’ll give it a read.
My suspicion was to try summing by group and looking at the shape diagnostic, but I haven’t tried it yet, I’ll give that a go and see if it works.
Thanks for the help!