LFO-CV for mixed models

A question regarding leave-future-out cross validation.

In the example in https://mc-stan.org/loo/articles/loo2-lfo.html, there is a single time series, and with approximate lfo-cv, the model is fit to the first L observations and the model performance is evaluated on how good it is able to subsequently predict the following observations.

What is a reasonable approach if there are multiple time series, for example in a mixed linear model where a group intercept and slope is fit to each time series? To me it seems there are two options:

  1. Leaving out just future observations (from L on) of one time series at a time, refitting the model and evaluating the model on the left out observations. Repeat this for every time series. Cost is that refitting is necessary for each time series.

  2. Leaving out all future observations for each time series from L on, and then evaluate predictive performance on all L+ observations. If L is small, then fitting might be more difficult because there is fewer data.

This depends on what is your predictive decision task. For example: 2. if you want use the model to predict actual future, so that you don’t have future observations from any series. 1. If you want to analyse goodness of the part of the model describing the between series similarities.

In addition to what Aki said, it is not clear to me yet how we should optimally approach the approximate LFO for multiple time-series such as those coming from a multilevel model. But I am open for suggestions on sensible approaches.

So 2. meets my needs better, although it means refitting multiple times. But is the approximation not fair to use in multilevels anyway as I understand @paul.buerkner?

In your case, you should be able to use the approximation but of course it is unclear how well it will work if you interpolate a lot of observations at once (1+ for each time-series). the khat diagnostic will tell you if it doesn’t work (and then refit the model). So it is just a matter of trying it out at this point I think. Nothing to loose really except that a lot of refits may be required.

Okay thanks a lot!