I’ve fitted a regression model and now have observations y and samples from the posterior predictive distribution y_{rep}.

One idea to test the model was to see if the predictive intervals are correctly calibrated, e.g. a posterior predictive interval that covers 10% density should contain the observation y in 10% of the cases (is this true?).

For the plot below I went through all pairs of (y_i, y_{rep_{i}}), calculated an x% interval (based on quantiles) for the draws of y_{rep_{i}} and counted how often y_{i} is in that interval.

The error bars are 95% intervals of a Beta distribution with \alpha = #observations inside the interval + 1 and \beta = #observation outside the interval + 1.

Interestingly, my posterior predictive distributions seem to be a bit too wide, e.g. ~55% of my data points are within the 50% predictive interval.

- Is my approach even valid? I’ve only seen this in binned form for logistic regression models before (e.g. in @avehtari’s tutorial here: https://avehtari.github.io/modelselection/diabetes.html)
- Should I be concerned for this specific case? All my other PPCs look okay.