When trying to do simulation based calibration, I personally felt that having to create multiple stan files was hard to keep track, especially when you’re making small changes here and there to the model.
Rstan’s sbc() does take care of the multi-file problem, given that the user calculates rank statistics within the stan code itself, and adhere to minor naming rules.
With that in mind, @hyunji.moon and I have created a small package which extends rstan::sbc() to provide support for cmdstanr models, while attempting to dull the burden of calculating ranks and prior/posterior predictive sampling from the user. In addition, we’re trying to keep the possibility of extending the features of SBC itself open, by adding some new plots and tacking on minor algorithms/metrics.
As of now, the package includes functions that streamline the predictive sampling process, new plots, notably the ECDF plot by Aki Vehtari, and a few empirical uniformity metrics. Some experimental methods, like bootstrap posterior predictive sampling, are also being worked on.
Unfortunately, the caveat is that the current implementation, being limited to just the “user-side” of the rstan interface, or in other words, no access to the C++ code of the model or other abstractions that make the model easier to manipulate, some “hacky” methods are being utilized to enable streamlined sampling. For example, in order to draw from the posterior predictive distribution, a single iteration is ran with explicitly set initial values. I’m afraid such methods can end up being inconsistent; thus having a significant impact on the results.
If you happen to have the time to take a look at the vignette included within the repo, we’d appreciate any feedback/suggestion/critique, especially on practicality.
As simulation based calibration itself is widely applicable once you have the preliminary steps implemented, we’re hoping that this packages will enable fluid incorporation of SBC into existing workflows with less effort than before.
I’m pinging @bnicenboim who was interested in this issue.