I’ve released a short blog about performing posterior predictive checks for models with observation-level random effects (OLREs) (e.g., for over-dispersed counts). This has come up a few times for me while consulting, so I figured I should write it up. The basic idea is that one should do mixed replication, generating new levels of the observation-level random effect using the posterior draws of the hyperparameter but not re-using the fitted OLRE parameters.
I give examples in both brms and PyMC. Critical feedback is welcome!
Thanks for posting. I always recommend Gelman et al. (1996) on this topic, which you cite. It talks about the alternative setup where there are groups and you can either replicate new draws from the same group or new groups (e.g., another subject in an existing clinical trial, or a whole new trial arm; another sporting event for the same player, or a whole new player, etc.). The former’s not going to be relevant for the varying effects you’re talking about.
Also, I think for a while @avehtari (or was it someone else?) was posting on Mastodon (Stan (@mcmc_stan@bayes.club) - Bayes Club) but I don’t think there’s been activity for a while and maybe now bluesky is more active than mastodon? I don’t have personal accounts on any of these (twitter/x, mastodon, bluesky), so I don’t know much about them at all or what the latest trends are. I just sometimes monitor the Stan twitter account since it’s existed for a while, has 11K followers, and people tag it frequently.