Distribution of Goodness of Fit Measures (WAIC, ELPD, LOO)

Is there a way to get prior predictive or posterior predictive distributions of the WAIC and LOO? My thought is that since these measure goodness-of-fit over the full distribution, they should be good statistics for calculating Bayesian p-values and u-values.

I am not really an expert on this, but my understanding is that WAIC and LOO are single quantities computed from the full posterior given observed data, so while I can imagine what their prior predictive distribution would look like (simulate data from the prior, fit the model to it, calculate WAIC/LOO, repeat) I don’t see what would posterior predictive distribution of them look like. Maybe I am genuinely missing something, so I’d be happy to hear that :-)

Thinking about evaluating model fit via predictions from data, it feels like this stuff could be connected to Simulation-Based calibration - does that look like it is addressing a similar problem?

Hope that helps at least a little.

1 Like