ArviZ loo_i

[shira]

Thank you again !

I always learn so much from reading your work, the least I can do is catch the occasional typo. :)

More questions:

@avehtari

Even if the inference itself is not exact, the cross-validation given that inference can be exact. If you want to know the predictive performance given MCMC for full posterior then the exact LOO will also use MCMC. If you want to know what would be the predictive performance given an exact inference for full posterior predictive density then MCMC is not enough. So “exact cross-validation” and “exact posterior inference” are different things.

I don’t yet understand. Are you describing the two sources of error I describe here:

• cross-validation error: replacing \text{elpd} = E_{p_{\text{true}}(\tilde{y})}[\log p(\tilde{y}|y)] \overset{\text{assuming our model is exchangeable}}{=} \sum_{i=1}^n E_{p_{\text{true}}(\tilde{y}_i)}[\log p(\tilde{y}_i|y)] with \text{lpd}_{\text{loo}} = \sum_{i=1}^n \log p(y_i | y_{-i})

EDITED (based on @avehtari’s correction below) from “iid” (independent and identically distributed) to “exchangeable”, which implies “id” (identically distributed) given covariates, see BDA3 p.5:

It follows from the assumption of exchangeability that the distribution of y, given x, is the same for all units in the study in the sense that if two units have the same value of x, then their distributions of y are the same.

• Monte Carlo error: estimating \text{lpd}_{\text{loo}}, which can be done using exact LOO/cross-validation with samples from p(\theta|y_{-i}), or using (PS)IS using samples from p(\theta|y) as I describe here.

2. In Sivula, Magnusson, Vehtari (2020) “Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison”, p.3 equation (1), do you assume that the true data generating process p_{\text{true}}(y) is i.i.d. ? That the model p_k(y|y^{\text{obs}}) is i.i.d. ? I’m wondering whether equation (1) for the elpd can be written more succinctly as E[\log p(y|y^{\text{obs}}) | y^{\text{obs}}] ?

3. I’m also looking at Bates, Hastie, Tibshirani (2021), where I think their \text{Err}_{XY} is analogous to your \text{elpd}(M_k | y^{\text{obs}}) = E[\log p(y|y^{\text{obs}}) | y^{\text{obs}}] in equation (1), and their \text{Err} = E[\text{Err}_{XY}] is analogous to your \text{e-elpd}(M_k) = E[\text{elpd}(M_k | y^{\text{obs}})] = E[\log p(y|y^{\text{obs}})] in equation (2) ? If I’m understanding the iterated expectation correctly, maybe equation (2) might be clearer if written with y^{\text{obs}} instead of y, for consistency with equation (1) ?