loo_R2: documentation and comparison to bayes_R2()

Hi,

Do you happen to have any documentation explaining the loo_R2() function? Trying to compare it to the bayes_R2() function based on Gelman et al (2017, and 2018)? When I compare two different regression models, the ordering of preference between these two models (i.e. which is ‘better’) changes, depending on which of these statistics I calculate (the bayesR2 or the looR2). I also note that the loo_R2() does not produce any Bayesian certainty interval estimates for me and complains about the “Pareto k diagnostic values are too high.”

Many thanks,

LOO-R^2 is described in the online appendix of Gelman, Goodrich, Gabry, and Vehtari (2018). R-squared for Bayesian regression models. The American Statistician.

That is possible, because Bayesian R^2 is over-optimistic as it is using the same data to compute posterior and R^2. LOO-R^2 uses LOO-CV to estimate what would R^2 be for new independent data coming from the same data generating process.

It’s more difficult as there are N different leave-one-out posteriors. That online appendix uses Bayesian bootstrap to give alternative non-parametric uncertainties.

See loo-glossary() and Vehtari, Gelman, and Gabry (2017a). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing . 27(5), 1413–1432, for more information on Pareto k diagnostic.

1 Like

Thanks! Very helpful.