In this post I learnt from an answer by @jsocolar that I can compare prediction of models from continuous families (i.e. with no discrete component) using loo-cv.
In this notebook Aki Vehtari discusses an exception to this rule where you can compare count data modeled as both a gaussian regression and a betabinomial regression.
I was wondering if the same applies to comparing different kinds of count models. For example can you compare, using loo-cv, a poisson or negative binomial model to a beta-binomial model, and, most importantly, trust the results?
Yes, comparing these models is totally straightforward :)
In fact, when we compare continuous models to discrete models, what we are doing is finding a sensible discretization of the continuous model in order that we can treat the comparison as a comparison between to discrete models. The important thing is to make sure that we are comparing probability mass to probability mass (discrete models, including discretized continuous models), or probability density to probability density (continuous models). What we cannot do is compare probability density to probability mass.
Aki’s notebook is not pointing out an exception to this; rather, it’s pointing out that in many cases the discretization of a continuous model to a discrete model is numerically trivial to do, so that we can form the log probability density matrix for the continuous model, and what we create will be numerically identical to a good approximation to the log probability mass matrix of a discretized version.
@jscolar and @avehtari is the ability to compare gamma to gaussian regression (discussed above) compromised if you specify a log link function in the gamma regression? i.e. do both models need to have an identity link function to compare them via loo_compare()?
Link functions can be different, the predicted quantities need to be the same. For example, you can compare normal (identity link) and log-normal (log link) model for y. If you did corresponding modeling using normal (identity link) for log(y) (transformed y), you would need to take into account the Jacobian
Thank you @avehtari. So if I generate predictions from the posterior (using add_epred_draws() for example) from the gamma model with the log link function, will they be on the same scale as the original or on a log scale?
I think I know the answer (yes, on the original scale) but I just need to check. I’m submitting results of a clinical trial very soon and need to know what I am reporting.
As is often the case @avehtari I am not sure I understand what you are saying, which is due to my ignorance, rather than your explanation.
So are you saying if I did a log transformation of the outcome y and then used those transformed y values in a gamma model with an identity link function I could not compare the gamma to a gaussian also with an identity link function, BUT if I instead used a log link function in a gamma model with nontransformed (i.e. raw) y values I can compare it to a gaussian with an identity link function?
And if I generate marginal predictions from the posterior using add_epred_draws() in tidybayes will the predictions of y from the gamma with the log link function be backtransformed into the original form of y?
Yes. The data must be the same (not the same up to a nonlinear transformation–actually the same) for the comparison to be straightforward. The details of the link function, on the other hand, don’t impede our ability to compare the models.
Sorry if I’m reading too much into your question, but it seems like you might be supposing that the two models are essentially the same. They are not. The first has y log-gamma distributed conditional on the predictors. The second has y gamma distributed conditional on the predictors. So for example the second model can produce values of y less than 1, but the first model cannot, because the log of y must be positive (in the support of the gamma distribution). This is by no means the only difference–the response distributions are completely different in many ways–but it’s one of the starkest examples to show the difference simply.
add_epred_draws() doc says it corresponds to posterior_epred(), which doc says " predicted mean response values" where the the word “response” means they are on the original response scale (and not in the latent scale)
You can compare, but if you use different transformations of y in different models, you need to explicitly include the Jacobian of that transformation yourself (not straightforward as @jsocolar put it). See Chapter 12 in the Regression and Other Stories book and the associated code example