LOO-PIT algorithm applicability

Algorithms
1

I have just implemented LOO-PIT checks in ArviZ, and I have some questions about the implemented algorithm that I have not been able to find in any of the references. Mainly, I think I have understood the concept of the algorithm and I feel like it could be used with any Bayesian test quantity, but I am not sure about it.

I am uploading 2 pages with my attempt at getting there alone, and where I try to explain a bit better my question because most of the text are actually equations. LOO_PIT_test_function.pdf (123.5 KB)

I would be really grateful if you could explain to me whether or not I am on the right track and why or ig you could point to some literature I may have missed.

Thanks!

2

I would write

\text{pit}_i \approx \sum_s w_i^s I(\hat{y}_i^s \leq y_i),

where i is the indicator function. Then this has the usual form of self-normalized importance sampling where w^s are normalized weights.

E[g(\theta)] \approx \sum_s w^s g(\theta^s),

You can change the test functions by changing the function g

1 Like
3

I assume you found the vignette and its links to the papers:

4

I’m looking for an explanation (online) of why the distribution of LOO-PIT should be uniform if the model is calibrated. Gelman (BAD) p153 says “For continuous data, cross-validation predictive
p-values have uniform distribution if the model is calibrated” but doesn’t seem to explain it. I can’t find an explanation in the vignettes or Gabry (2019). The original Gelfand (1992) is in a book but I was hoping for something online and not too difficult. apols if I’ve just missed it.

thanks

2 Likes
5

I think these two blog posts will help you understand LOO-PIT more clearly: Advanced Bayesian Model Checking: Calibration by Eduardo Coronado written in R which is in progress of being added as a new vignette of bayesplot and LOO-PIT tutorial written by me using python.

5 Likes