LOO-PIT algorithm applicability

I have just implemented LOO-PIT checks in ArviZ, and I have some questions about the implemented algorithm that I have not been able to find in any of the references. Mainly, I think I have understood the concept of the algorithm and I feel like it could be used with any Bayesian test quantity, but I am not sure about it.

I am uploading 2 pages with my attempt at getting there alone, and where I try to explain a bit better my question because most of the text are actually equations. LOO_PIT_test_function.pdf (123.5 KB)

I would be really grateful if you could explain to me whether or not I am on the right track and why or ig you could point to some literature I may have missed.

Thanks!

I would write

\text{pit}_i \approx \sum_s w_i^s I(\hat{y}_i^s \leq y_i),

where i is the indicator function. Then this has the usual form of self-normalized importance sampling where w^s are normalized weights.

E[g(\theta)] \approx \sum_s w^s g(\theta^s),

You can change the test functions by changing the function g

1 Like

I assume you found the vignette and its links to the papers:

  • Vehtari, A., Gelman, A., and Gabry, J. (2017a). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing . 27(5), 1413–1432. doi:10.1007/s11222-016-9696-4. ( journal, preprint arXiv:1507.04544).

  • Vehtari, A., Gelman, A., and Gabry, J. (2017b). Pareto smoothed importance sampling. arXiv preprint: http://arxiv.org/abs/1507.02646/