Can WAIC/LOOIC be used to compare models with different likelihoods?

Modeling
1

Dear all,

I’d like to echo an issue opened on StackExchange: https://stats.stackexchange.com/questions/325012/can-waic-be-used-to-compare-bayesian-linear-regression-models-with-different-lik/325220

In brief, I am wondering whether WAIC (or LOOIC) can be used to compare model with different likelihoods ? In McElreath’ book (chapter 9), there is a section (cf. attached screenshot) explaining that information criteria, including WAIC, cannot be used to compare models with different likelihoods.

23
Screenshot 2019-01-18 at 20.32.23.png1392×890 228 KB

However, I have found (sparse) contradictory information online (e.g., in the SE thread linked above).

As an example, I am currently analysing data where I would like to compare Normal and Skew-Normal models. Could I use WAIC or LOOIC in that situation ? If not, what would you recommend for model comparison ?

I would be glad to hear your thoughts about this question.

Ladislas

2 Likes
2

First to make the terms more clear, p(y|theta) as a function of y is an observation model and p(y|theta) as a function of theta is a likelihood. waic and loo focus on (log) predictive density (in continuous case) or probability (in discrete case). Your question would make more sense if you would ask “whether WAIC (or LOOIC) can be used to compare model with different observation models ?”

  • You can compare models given different discrete observation models and it’s also allowed to have different y as long as the mapping is bijective.
  • You can’t mix densities and probabilities, so you can’t compare model given continuous and discrete observation models, unless you compute probabilities in intervals from the continuous model.
  • You can compare models given different continuous observation models and but you have exactly the same y. If y is transformed, then the Jacobian of that transformation needs to be included.

I repeat that you need to focus on the observation model. As a bonus there is at least one case where a discrete observation model and a continuous observation model have the same continuous likelihood, so knowing the likelihood doesn’t help deciding whether the two models can be compared.

7 Likes