How can i validate a fit in Stan? Exists any package or function to check the fit?
1 Like
You might be especially interested in the R packages loo and bayesplot, both of which are well integrated with the Stan ecosystem in R. See the three links below.
https://cran.r-project.org/web/packages/loo/vignettes/loo2-with-rstan.html
https://cran.r-project.org/web/packages/bayesplot/vignettes/graphical-ppcs.html
Sorry, i forgot epecify that i’m working with PyStan.
1 Like
In Python ArviZ includes functionality for posterior predictive checking and loo.
1 Like
“Validating a fit” is something of an ambiguous term that many people use to mean different tasks.
The first and most well-defined interpretation is checking if the computational output is an accurate quantification of the exact posterior distribution, i.e. if the Stan accurately “fits” the posterior distribution. For Stan that means validating the conditions required for Markov chain Monte Carlo estimators to be well-behaved, which is unfortunately a messy process. See for example Markov Chain Monte Carlo in Practice and An Introduction to Stan for some more discussion.
Once the posterior quantification is trustworthy one can then consider checking whether or not the assumed model captures the relevant structure of the observed data, i.e. if the model “fits” the observed data. In practice we can never confirm that model assumptions are correct, but we can identify deviations that indicate missing structure. For much more see for example Towards A Principled Bayesian Workflow.
There are numerous packages that implement various prebuilt checks that can sometimes be applied to both of these steps but it’s up to the user to verify that those checks are relevant.
1 Like