loo_R2: documentation and comparison to bayes_R2()

[avehtari]

LOO-R^2 is described in the online appendix of Gelman, Goodrich, Gabry, and Vehtari (2018). R-squared for Bayesian regression models. The American Statistician.

@Ilan_Strauss

When I compare two different regression models, the ordering of preference between these two models (i.e. which is ‘better’) changes, depending on which of these statistics I calculate (the bayesR2 or the looR2).

That is possible, because Bayesian R^2 is over-optimistic as it is using the same data to compute posterior and R^2. LOO-R^2 uses LOO-CV to estimate what would R^2 be for new independent data coming from the same data generating process.

@Ilan_Strauss

I also note that the loo_R2() does not produce any Bayesian certainty interval estimates for me

It’s more difficult as there are N different leave-one-out posteriors. That online appendix uses Bayesian bootstrap to give alternative non-parametric uncertainties.

@Ilan_Strauss

and complains about the “Pareto k diagnostic values are too high.”

See loo-glossary() and Vehtari, Gelman, and Gabry (2017a). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing . 27(5), 1413–1432, for more information on Pareto k diagnostic.