Calibration plot for Bayesian binomial models


I have a Bernoulli predictive model created using BART, and I would like to show a calibration plot for it. I am struggling with how to include information regarding the whole posterior in it.

My current approach is the following:

  • distribute observations into risk quantile using the median of the individual posterior distribution of the expected risk.
  • estimate the unconditional risk in each quantile using a spline model to map the quantiles to the observed number of positive cases.
  • for each posterior sample, take the mean of the individual expected risk in each quantile to have the posterior distribution of the mean predicted risk by quantile.
  • compare the mean predicted risk distribution with the estimated risk by quantile produced by the spline model.
  • bonus: add summary indicators like median, credibility intervals, posterior density, etc

This is the outcome:

Another option is to visualize the ratio of the predicted over the expected risk:

which I think is more informative and easier to interpret.
Another addition I am thinking about is adding different “acceptability” bands for each risk quantile. In the left most quantiles the observed risk is much higher than the predicted, but I guess it’s just because I have 2 events on 10000 observations, which I think is totally acceptable in a random process.

I’d like to know your opinion on this technique. I couldn’t find any work on calibration plots for Bayesian models, so I would be grateful if someone knew any good work or alternative approach I could cite.


You didn’t mention the number of trials, but ROS discusses discrete data residual plots (Section 14.5) and a case study shows 3 different calibration plots for Bernoulli case (I like the CORP approach).

Your plots look sensible, too.

Thanks! I’ll give it a look!

Sorry, it’s a Bernoulli model, so dichotomous outcome. I’ll fix the question.

1 Like