Do you use simulated data to check that your model works correctly? If no, then you should :-) If yes, then you are just a small step away from being able to validate the implementation of your model more thoroughly with SBC (simulation-based calibration). The SBC R package (https://hyunjimoon.github.io/SBC/) developed by me, @hyunji.moon, @Dashadower and Teemu Säilynoja is there to let you access the powerful SBC machinery as easily as possible and (unlike rstan::sbc) without the need to make any changes to your Stan model.
To put it yet another way - SBC is in my opinion the closest we have to unit tests for statistical models.
We have made a bunch of case studies/vignettes for the package that you may find interesting:
- Basic usage and philosophy of the package and introduction to rank visualizations
- Small model implementation workflow - this is IMHO the biggest selling point for the package. We show how to iteratively build a non-trivial model, validating each step of the implementation, starting with a single simulation and smoothly ramping up to hundreds of simulations. This mirrors the process of how I’ve been building my models for the past few years and summarises a lot of the ideas that I am contributing to the upcoming Bayesian worfklow book.
- If you want something short, Discovering bad parametrization and Discovering indexing errors are very small examples showing how simple but common errors can be caught.
- SBC for ADVI and optimizing shows how to use the package to learn how much precision you loose when using an approximate algorithm (without ever needing to compare the results to those of MCMC).
- SBC with discrete parameters in Stan and JAGS discusses what can be done about models with discrete parameters - both explicit and marginalized out - and how to test that you implemented the marginalization correctly.
- Limits of SBC shows some problems in your model/fitting algorithm that cannot be caught by vanilla SBC and proposes including generated quantities in SBC as a remedy for some of those.
The package currently supports models in rstan, cmdstanr, JAGS (via rjags) and brms models and it can be connected to additional modelling/fitting packages with ease.
If you need help running/understanding/extending the package, please ask a question here on Discourse and tag it with #simulation-based-calibration . Bug reports are welcome at https://github.com/hyunjimoon/SBC/issues