Extending Rstan's Simulation Based Calibration

Algorithms
1

When trying to do simulation based calibration, I personally felt that having to create multiple stan files was hard to keep track, especially when you’re making small changes here and there to the model.
Rstan’s sbc() does take care of the multi-file problem, given that the user calculates rank statistics within the stan code itself, and adhere to minor naming rules.

With that in mind, @hyunji.moon and I have created a small package which extends rstan::sbc() to provide support for cmdstanr models, while attempting to dull the burden of calculating ranks and prior/posterior predictive sampling from the user. In addition, we’re trying to keep the possibility of extending the features of SBC itself open, by adding some new plots and tacking on minor algorithms/metrics.

As of now, the package includes functions that streamline the predictive sampling process, new plots, notably the ECDF plot by Aki Vehtari, and a few empirical uniformity metrics. Some experimental methods, like bootstrap posterior predictive sampling, are also being worked on.

Unfortunately, the caveat is that the current implementation, being limited to just the “user-side” of the rstan interface, or in other words, no access to the C++ code of the model or other abstractions that make the model easier to manipulate, some “hacky” methods are being utilized to enable streamlined sampling. For example, in order to draw from the posterior predictive distribution, a single iteration is ran with explicitly set initial values. I’m afraid such methods can end up being inconsistent; thus having a significant impact on the results.

If you happen to have the time to take a look at the vignette included within the repo, we’d appreciate any feedback/suggestion/critique, especially on practicality.

As simulation based calibration itself is widely applicable once you have the preliminary steps implemented, we’re hoping that this packages will enable fluid incorporation of SBC into existing workflows with less effort than before.

I’m pinging @bnicenboim who was interested in this issue.

9 Likes
2

Link to the vignette!

3 Likes
3

Is the a plan to have it on CRAN? - It looks very good and I would like to try it but I’m aloud to download packages from CRAN only

2 Likes
4

Yes, but at best in a medium term outlook - submitting to CRAN seems to require a lot of work, and we will have to handle a couple issues (like the vignettes requiring a ton of computing resources to build, so they cannot be built on CRAN). Also, I am waiting a bit to get feedback on the current APIs and make any big changes before CRAN release to make sure when we release on CRAN we can guarantee reasonable stability.

2 Likes
5

You may have already seen this, but we get around this issue in many Stan R packages by generating the vignettes locally when building the package for submission to CRAN and inserting some code to avoid them being evaluated on CRAN. For example, at the top of the Rmd file you can set

params:
  EVAL: !r identical(Sys.getenv("NOT_CRAN"), "true")

and then use the knitr chunk settings to turn off evaluating all the chunks if on CRAN:

knitr::opts_chunk$set(
  eval = if (isTRUE(exists("params"))) params$EVAL else FALSE
)

We definitely do this in loo and rstanarm and I think other packages as well.

2 Likes