I am trying to specify the situation where SBC is needed to complement predictive checks; can anyone share their thoughts or recommend literature on the following, please?
-
Example situations where self-inconsistent model that false positively give decent predictive results can be a problem?
-
For cases where only good predictive inference is needed, what could go wrong if the model is self-inconsistent?
-
SBC’s role in model validation? Can it detect model misspecification?
As the example in Talts, 2018 is titled ‘6.1 misspecified prior’, I thought SBC could detect model misspecification. However, model misspecification refers to the situation when the model can’t match the data generating process. As data is generated from the model in SBC, the model is always well specified.
\pi(\theta) \simeq \int \mathrm{d} \tilde{y} \mathrm{~d} \tilde{\theta} \pi_{\text {approx }}(\theta \mid \tilde{y}) \pi(\tilde{y} \mid \tilde{\theta}) \pi(\tilde{\theta})
To be precise, inconsistency is observed when approximate joint or conditional distribution of computation algorithm is not close enough to that of original model ie. \pi(\theta |y) and \pi_{approx}(\theta ,y) are largely different. However, the identity holds true for any \pi(\tilde{y} \mid \tilde{\theta}) \pi(\tilde{\theta}). Then what word can we use to describe the work SBC is believed to do in terms of model validation? Could the difference of true distribution and finite sample distribution be one candidate?
For 1 or 2, comments from authors of the following two papers which has high focus on predictive beharviors of the model (if I understood correctly) would be very helpful!
Projection Predictive Inference for Generalized Linear and Additive Multilevel Models @AlejandroCatalina @paul.buerkner @avehtari
A Decision-Theoretic Approach for Model Interpretability in Bayesian Framework @to-mi @jpiironen @avehtari
Note) a validation methodology comparison concentrates on parameter distribution in Yao, 2018. It explains VSBC diagnostics could complement PSIS diagnostic when VI posterior produce good point estimates even though the underlying distribution has stark difference with the true posterior.
Thank you.