Advice on plotting predicted versus observed by item

I am seeking advice on some good strategies for plotting predicted versus observed values, grouped by item, in a way that is consistent with Bayesian philosophy. I am working with brms.

A simplified version of the formula for the model (continuous response variable) is:

response ~ 1 + (X1 + X2) * X3 + (1 + X1 + X2 | prtcpnt) + (1 + X3 | item)

I want to plot the observed values against the predicted values per item. It is straightforward to make a scatter plot with the within-item mean observed value (y-axis) against the within-item mean predicted value (x-axis), all colour- and shape-coded by item (and that works very well). However, my question is how one should also include uncertainties in this plot.

For example, I could calculate the 95% confidence (not credibility) intervals for the mean response in each item and use vertical error bars. But is that an unprincipled mixing of frequentist and Bayesian methods?

Alternatively, I could use the posterior predictive distribution for each item and use horizontal error bars, but it’s not at all clear to me how those errors would would be averaged for each item – either conceptually or in terms of processing the draws.

Perhaps there is a completely different way I haven’t considered. Note that there are 48 items, so a set of 48 panels is not what I am after. I want a scatter plot with points and error bars, and each point colour- and shape-coded by item, and with some indication of uncertainty.

Any thoughts or suggestions would be gratefully received.

1 Like