In this post I learnt from an answer by @jsocolar that I can compare prediction of models from continuous families (i.e. with no discrete component) using loo-cv.
In this notebook Aki Vehtari discusses an exception to this rule where you can compare count data modeled as both a gaussian regression and a betabinomial regression.
I was wondering if the same applies to comparing different kinds of count models. For example can you compare, using loo-cv, a poisson or negative binomial model to a beta-binomial model, and, most importantly, trust the results?
Thanks in advance.
Yes, comparing these models is totally straightforward :)
In fact, when we compare continuous models to discrete models, what we are doing is finding a sensible discretization of the continuous model in order that we can treat the comparison as a comparison between to discrete models. The important thing is to make sure that we are comparing probability mass to probability mass (discrete models, including discretized continuous models), or probability density to probability density (continuous models). What we cannot do is compare probability density to probability mass.
Aki’s notebook is not pointing out an exception to this; rather, it’s pointing out that in many cases the discretization of a continuous model to a discrete model is numerically trivial to do, so that we can form the log probability density matrix for the continuous model, and what we create will be numerically identical to a good approximation to the log probability mass matrix of a discretized version.
Thank you @jsocolar. This is such a powerful tool for improving inference. It will be part of my workflow from now on.