Hi All,

I want to get a discussion going on how Stan users on the forum go about checking and validating their models and best practices.I know this can be case-specific and information can be found in the various user manuals, but I’m still curious,

# Checking

At a minimum, a good model should produce posterior estimates with \hat{R}\leq 1.01 and large effective sample size (ESS). SE_mean should also ideally be near zero, ensuring that the simulation has been run long enough. If not, then the number of chain iterations can be increased from the default of 2000, to 2500, say.

Chain mixing is checked visually via traceplots of parameter estimates. Plots should look like “fuzzy caterpillars”.

In this 2017 case, study, Michael Betancourt, walks through a robust statistical workflow using Stan:

https://mc-stan.org/users/documentation/case-studies/rstan_workflow.html

which also includes examining aspects of the HMC sampler itself such as the tree depth, E-BFMI (Energy Bayesian Fraction of Missing Information), and divergence.

Finally, model reparameterization may be needed to fix common issues.

# Validating

Model validation typically consists of drawing simulated values from the model via posterior predictive checks. Adequacy of priors is analogously done via prior predictive checking.

Simulation-based calibration (SBC) can further shed light on model performance.

# Question

This is only a summary, but I’m wondering what others think. Care to weigh in?