Clarity on `pp_check` barplots for ordinal models & prediction accuracy

Modeling
,
1

Very simple question - I am hoping to get some simple clarity on what the pp_check bar plots are telling me. Here is a brms example from the documentation

library(brms)
fit2 <- brm(rating ~ treat + period + carry,
              data = inhaler, family = cumulative(),
              prior = set_prior("normal(0,5)"), refresh = 0)

pp_check(fit2, type = 'bars')

image
image770×475 8.43 KB

This plot is showing the distribution of the ratings 1-4 in the inhaler data (by count) and the predicted number of each rating from the model (median as dot + error bars), i.e. the predicted distribution.

However, the y-rep tells me nothing about the accuracy of the predictions. So the model may predict that many emperical 1’s are actually 2’s, and visa-versa, getting many of the predictions wrong, but I cannot know the accuracy from this plot. Am I thinking of this correctly?

Thanks!

2

Yes. You could create a confusion matrix for seeing in which ways the predictions are wrong. And for the overall accuracy you could use cross-validation (see e.g. loo package

2 Likes