WAIC unreliable?

Keep in mind that PSIS-LOO, WAIC, DIC, AIC, etc are all approximations to a fundamentally-uncalculatable number that exactly quantifies predictive accuracy. And each of those approximations require different assumptions to be reasonably accurate.

A common assumption that’s necessary is that the data are roughly IID so that, for example, looking at a subset of the data gives a reasonable quantification of what a draw from the full data would do. In many models, however, such as the one that Ben mentions this assumption is not valid and all of the predictive performance estimators should be suspect.

One of the great features of PSIS-LOO is that is has the self-diagnostic that can identify the failure of some of these assumptions and that makes it much more useful in practice than many of the other estimators. And in practice if we can’t trust PSIS-LOO then we probably can’t trust any of the others like WAIC.

5 Likes