Conn et al published a very relevant paper on Bayesian model checking for ecologists.

They focus on three different models that are pretty popular these days: spatiotemporal count data, N-mixture models for abundance estimation and hidden Markov models for estimating residence time. After simulating data from these different models they compare the ability of different model checking tools to detect model discrepancy such as bayesian p-values based on posterior predictive distribution or cross‐validated probability integral transform.

I’d be interested to hear what people from the Stan community have to say on this topic.

3 Likes

Annoyingly the published paper is not freely available. I did find some version with the same title, so I comment that one. It’s a reasonable review, although they repeatedly mention the computational issue with cross-validation as they have missed the possibility of fast cross-validation computation using PSIS-LOO (loo package can be used with JAGS, too). Examples are nice and conclusions are sensible (except for missing the fast cross-validation…).

Thanks for posting this. For those who don’t have access and don’t want to use alternative means of getting the published paper, you can access the preprint here: https://peerj.com/preprints/3390/

Yeah, that’s the version I read, but since it has big red letters stating “NOT PEER-REVIEWED”, I assumed that it can be much different than the published version,

It’s important to remember that there is no bias toward 0.5 if the test statistic is ancillary. Thus it is important to choose test statistics which are close to ancillary (and testing some interesting property). See demos 1 and 3 at https://github.com/avehtari/BDA_R_demos/tree/master/demos_ch6 for good and bad statistic and how much these test statistics depend on the parameters of the model. A good test statistic is helpful to detect problems even with the mentioned conservatism. Furthermore demo 4 shows how LOO can be used to avoid the double use of data (using probability integral transofrmation (PIT) test statistic).

Sounds like the journal’s FUD is working.

I’m editing out that link as the Stan forums are not the place to pirate copyrighted papers.

Authors should just not publish in those venues or be willing to have everyone comment on their “NOT PEER REVIEWED” versions.

1 Like