Worst LOO-PIT plots ever. PP Checks great

I am fitting zero-inflated negative binomial models in brms for count data that has many many zeros (90% zeroes). I fit two models:
m1<-brm(bf(Count ~ 1 + Treatment + (1|Location) + (1|id) + (1|Day/Time_Day), zi ~ 1 + (1|id))
m2<-brm(bf(Count ~ 1 + (1|id), zi ~ 1 + (1|id), shape ~ 1 + (1|id))
I know beforehand that the zero-inflation component depends heavily on the group (id) member as does likely the shape parameter (hence m2).

Both of these models give very good posterior predictive checks.

However, the LOO-PIT plots are incredibly bad. I have already done some looking and found these links:

My understanding from these discussions and the paper that @avehtari references are that the LOO-PIT plots (as done in the loo package) are not valid for zero-inflated negative binomial models for count data.

Have I interpreted this correctly? Should I just ignore the LOO-PIT plots for these models?
If this is not correct, then how do I interpret the bizarre shape of these plots?

Many thanks for any insight!

1 Like

That is correct and we’re working on better plots that are valid for discrete outcomes. Meanwhile you can use the randomized approach described in Predictive Model Assessment for Count Data to make valid plots. The randomized approach is not optimal as it adds randomness, but it is valid.

I was going to recommend plots for discrete outcomes, but then you already mentioned that they look good. As you have (1|id) terms it is possible that your model is overfitting (seeing loo diagnostics would help) and PPC don’t reveal problems, and thus LOO based diagnostics would be useful, too.

1 Like

Ok, great. Thank you for the information on LOO-PIT for zero-inflated models!

Below is the LOO diagnostics for m1 and m2.

Somehow I missed the PPC plots specifically for discrete outcomes! I was using the typical histogram, density, and stat PPC plots before.
The plots for discrete outcomes are below. I think these don’t look too terrible either, but I did have to ‘zoom in’ quite a bit, as my model predicts occasional extremely (implausibly) high values. Even though the PIT plots aren’t valid, would these high values cause the weird hump at the right end of the PIT plot??

“As you have (1|id) terms it is possible that your model is overfitting”
Well, it would just seem incorrect to leave this out in any situation. This is bacterial count data from hospital surfaces in unique groups (multiple measurements per group). The amount of this bacteria shed into the environment is very dependent upon the specific group id even given the same condition and factors among group id’s. This leads to count distribution and zero-inflation that varies greatly among id. This is why I have used the group-level effects in both the negative binomial and zero-inflation parts of the model. Using this type of model, I was hoping to tease out the treatment effect, as we have a within-id design (same group id serves for it’s own control and treatment scenario both).

Thank you for the helpful response.

I didn’t mean to suggest that you would leave that out. What I meant with that is that given you have such term, there can be a bigger difference between posterior and LOO predictive checks and specifically PPC can look like a perfect fit.

Seeing the LOO diagnostics, the concern about the overfit is reduced, as the p_loo (~25) divided by the number of observations (731) is quite small and most Pareto k’s are less than 0.5. Based on this I would expect not much difference between posterior and LOO predictive checks. There is still a difference that those PPC plots you show are looking at the marginal distributions and sometimes they can look quite good while conditional calibration is showing different problems.

That shape is mostly an artifact due to discreteness. For example, the very large number of zeros makes the smallest the smallest computed PIT values quite large and there are no PIT values between 0 and that smallest computed PIT value. The randomized PIT for discrete outcomes replaces PIT values with uniform random values between PIT values, and then those many PIT values corresponding to the all the zeros, would also spread between 0 and the smallest computed value.

1 Like

Ok, great. I see what you are getting at. That seems like a very important point about the difference between posterior and LOO predictive checks that I should keep in mind for the future.

Thank you so much for the detailed explanation and helpful information! I really appreciate it.