I have a Bernoulli predictive model created using BART, and I would like to show a calibration plot for it. I am struggling with how to include information regarding the whole posterior in it.
My current approach is the following:
- distribute observations into risk quantile using the median of the individual posterior distribution of the expected risk.
- estimate the unconditional risk in each quantile using a spline model to map the quantiles to the observed number of positive cases.
- for each posterior sample, take the mean of the individual expected risk in each quantile to have the posterior distribution of the mean predicted risk by quantile.
- compare the mean predicted risk distribution with the estimated risk by quantile produced by the spline model.
- bonus: add summary indicators like median, credibility intervals, posterior density, etc
This is the outcome:
Another option is to visualize the ratio of the predicted over the expected risk:
which I think is more informative and easier to interpret.
Another addition I am thinking about is adding different “acceptability” bands for each risk quantile. In the left most quantiles the observed risk is much higher than the predicted, but I guess it’s just because I have 2 events on 10000 observations, which I think is totally acceptable in a random process.
I’d like to know your opinion on this technique. I couldn’t find any work on calibration plots for Bayesian models, so I would be grateful if someone knew any good work or alternative approach I could cite.