How do you summarize posterior predictive p-values analyses of a hierarchical model when the summary statistic you want to test is at the lowest level and there are ~10000 groups? I was thinking I would just make a histogram of all the p-values. Would you find that acceptable? For clarity, I’m trying to do something along the lines of chapter 6.3 here http://www.stat.columbia.edu/~gelman/book/BDA3.pdf
Somewhat related, how do you deal with posterior distributions that have most of their density at 0? If, say, the posterior predictive distribution has 99% of its density at 0 and y=0, the {p_B} = {\rm{Pr}}(T({y^{{\rm{rep}}}},\theta ) \ge T(y,\theta )|y)=0.01 and the flipped tail-comparison {p_B} = {\rm{Pr}}(T({y^{{\rm{rep}}}},\theta ) \le T(y,\theta )|y)=0. That is, the p-values look horrible even though the model prediction is good. Is it acceptable to count half the density at 0 as \le and half as \ge, giving instead p-values \approx 0.5?