LOO-PIT algorithm applicability

I have just implemented LOO-PIT checks in ArviZ, and I have some questions about the implemented algorithm that I have not been able to find in any of the references. Mainly, I think I have understood the concept of the algorithm and I feel like it could be used with any Bayesian test quantity, but I am not sure about it.

I am uploading 2 pages with my attempt at getting there alone, and where I try to explain a bit better my question because most of the text are actually equations. LOO_PIT_test_function.pdf (123.5 KB)

I would be really grateful if you could explain to me whether or not I am on the right track and why or ig you could point to some literature I may have missed.

Thanks!

I would write

\text{pit}_i \approx \sum_s w_i^s I(\hat{y}_i^s \leq y_i),

where i is the indicator function. Then this has the usual form of self-normalized importance sampling where w^s are normalized weights.

E[g(\theta)] \approx \sum_s w^s g(\theta^s),

You can change the test functions by changing the function g

1 Like

I assume you found the vignette and its links to the papers:

  • Vehtari, A., Gelman, A., and Gabry, J. (2017a). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing . 27(5), 1413–1432. doi:10.1007/s11222-016-9696-4. ( journal, preprint arXiv:1507.04544).

  • Vehtari, A., Gelman, A., and Gabry, J. (2017b). Pareto smoothed importance sampling. arXiv preprint: http://arxiv.org/abs/1507.02646/

I’m looking for an explanation (online) of why the distribution of LOO-PIT should be uniform if the model is calibrated. Gelman (BAD) p153 says “For continuous data, cross-validation predictive
p-values have uniform distribution if the model is calibrated” but doesn’t seem to explain it. I can’t find an explanation in the vignettes or Gabry (2019). The original Gelfand (1992) is in a book but I was hoping for something online and not too difficult. apols if I’ve just missed it.

thanks

2 Likes

I think these two blog posts will help you understand LOO-PIT more clearly: Advanced Bayesian Model Checking: Calibration by Eduardo Coronado written in R which is in progress of being added as a new vignette of bayesplot and LOO-PIT tutorial written by me using python.

4 Likes