Comparing non-nested models using elpd_diff

I just read the help for the loo_compare() function in the brms package and it says

To compute the standard error of the difference in ELPD — which should not be expected to equal the difference of the standard errors — we use a paired estimate to take advantage of the fact that the same set of N data points was used to fit both models. These calculations should be most useful when N is large, because then non-normality of the distribution is not such an issue when estimating the uncertainty in these sums. These standard errors, for all their flaws, should give a better sense of uncertainty than what is obtained using the current standard approach of comparing differences of deviances to a Chi-squared distribution, a practice derived for Gaussian linear models or asymptotically, and which only applies to nested models in any case.

Does this mean you can compare non-nested models using the loo_compare() function? And how far does this extend? Could you compare a linear model to a non-linear model for example? Or an exponential regression model to a gamma regression model.

Somewhere, sometime, I was taught that it’s ok, given sufficiently large N, to compare non-nested models, as well as models coming from the same family group, e.g., exponential and gamma both belong to the same family of distributions.

It would be nice if @avehtari could give us a hint about this :)

1 Like

Yes. Just use the same outcome. See more in CV-FAQ, especially, Q12 and Q13.

1 Like