Interpreting output from compare() of loo

If I am comparing two models using compare() of the loo package, how should I interpret the SE values.
I compare three models A, B, C and the compare() output is:

compare(looB, looA)
elpd_diff se
-431.1 84.7

compare(looC, looB)
elpd_diff se
42.3 14.3

How can I interpret the SE values of these output?

?loo::compare says

The difference will be positive if the expected predictive accuracy for the second model is higher.

and

To compute the standard error of this difference we can use a paired estimate to take advantage of the fact that the same set of N data points was used to fit both models. These calculations should be most useful when N is large, because then non-normality of the distribution is not such an issue when estimating the uncertainty in these sums.

You can read more in http://arxiv.org/abs/1507.04544

1 Like

one related question. If SE is bigger than the difference between models, then one would say that the models are similar (and vice versa)?

First, instead of SE, it’s better to consider something like 2SE or more cautious 4SE, where 4 comes from the fact that SE for LOO can be underestimated for small n or under bad model misspecification. Second, the models can be very different and the predictions can be very different, it’s just that the average predictive accuracies are close to each other. Third, SE describe uncertainty, so if SE is large then it’s likely that the models do have big difference in predictive accuracy, but we don’t know whether the difference is negative or positive.

5 Likes

Thanks - this is very helpful advice!