# Calibration plot for Bayesian binomial models

**URL:** <https://discourse.mc-stan.org/t/calibration-plot-for-bayesian-binomial-models/27714>\
**Category:** General\
**Tags:** posterior-predictive\
**Created:** [June 3, 2022, 5:20pm UTC](https://discourse.mc-stan.org/t/calibration-plot-for-bayesian-binomial-models/27714 "2022-06-03T17:20:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Angelo\_D\_Ambrosio](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/angelo_d_ambrosio/32/5298_2.png) [@Angelo\_D\_Ambrosio](https://discourse.mc-stan.org/u/Angelo_D_Ambrosio)\
**Post date:** [June 3, 2022, 5:20pm UTC](https://discourse.mc-stan.org/t/calibration-plot-for-bayesian-binomial-models/27714/1 "2022-06-03T17:20:56Z")

</div>

Hello,

I have a Bernoulli predictive model created using BART, and I would like to show a calibration plot for it. I am struggling with how to include information regarding the whole posterior in it.

My current approach is the following:

- distribute observations into risk quantile using the median of the individual posterior distribution of the expected risk.
- estimate the unconditional risk in each quantile using a spline model to map the quantiles to the observed number of positive cases.
- for each posterior sample, take the mean of the individual expected risk in each quantile to have the posterior distribution of the mean predicted risk by quantile.
- compare the mean predicted risk distribution with the estimated risk by quantile produced by the spline model.
- bonus: add summary indicators like median, credibility intervals, posterior density, etc

This is the outcome:

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/original/2X/6/6d68cae525a974756836da92207614c6ec1e1e0e.jpeg)

Another option is to visualize the ratio of the predicted over the expected risk:

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/original/2X/0/03250759c662f9b650be2d3e868f23b6cd64f7bb.jpeg)  
which I think is more informative and easier to interpret.  
Another addition I am thinking about is adding different “acceptability” bands for each risk quantile. In the left most quantiles the observed risk is much higher than the predicted, but I guess it’s just because I have 2 events on 10000 observations, which I think is totally acceptable in a random process.

I’d like to know your opinion on this technique. I couldn’t find any work on calibration plots for Bayesian models, so I would be grateful if someone knew any good work or alternative approach I could cite.

---

<div class="post-metadata">

**Author:** ![avehtari](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/avehtari/32/5935_2.png) [@avehtari](https://discourse.mc-stan.org/u/avehtari)\
**Post date:** [June 6, 2022, 6:40pm UTC](https://discourse.mc-stan.org/t/calibration-plot-for-bayesian-binomial-models/27714/2 "2022-06-06T18:40:58Z")

</div>

> [@Angelo\_D\_Ambrosio](#):
>
> I couldn’t find any work on calibration plots for Bayesian models

You didn’t mention the number of trials, but [ROS](https://avehtari.github.io/ROS-Examples/) discusses discrete data residual plots (Section 14.5) and [a case study](https://avehtari.github.io/modelselection/diabetes.html#44_Calibration_of_predictions) shows 3 different calibration plots for Bernoulli case (I like the CORP approach).

Your plots look sensible, too.

---

<div class="post-metadata">

**Author:** ![Angelo\_D\_Ambrosio](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/angelo_d_ambrosio/32/5298_2.png) [@Angelo\_D\_Ambrosio](https://discourse.mc-stan.org/u/Angelo_D_Ambrosio)\
**Post date:** [June 7, 2022, 4:39pm UTC](https://discourse.mc-stan.org/t/calibration-plot-for-bayesian-binomial-models/27714/3 "2022-06-07T16:39:25Z")

</div>

Thanks! I’ll give it a look!

Sorry, it’s a Bernoulli model, so dichotomous outcome. I’ll fix the question.
