@OriolAbril, you’re a hero !
Some follow-up to your excellent points:
Our goal is actually to calculate the
elppd(Equation 7.2 in BDA3, calledelpdin the paper I linked, Equation 1) because it is better suited in practice than what is defined aselpdin BDA3 (Equation 7.1) which is generally impossible to calculate.
My understanding is that anything with the “e” (both elpd and elppd) is impossible to calculate without knowing the true distribution (which BDA3 calls f), we can only hope to estimate it.
This estimation seems to involve two sources of error:
- Replacing \text{elpd}_i = E_f[\log p(\tilde{y}_i|y)] with \text{lpd}_{\text{loo},i} = \log p(y_i | y_{-i})
- Estimating \text{lpd}_{\text{loo},i} = \log p(y_i | y_{-i}) with the PSIS estimator
The error from 2 is the Monte Carlo error. The error from 1 I’m not sure what to call, maybe cross-validation error ?