loo_R2: documentation and comparison to bayes_R2()


Do you happen to have any documentation explaining the loo_R2() function? Trying to compare it to the bayes_R2() function based on Gelman et al (2017, and 2018)? When I compare two different regression models, the ordering of preference between these two models (i.e. which is ‘better’) changes, depending on which of these statistics I calculate (the bayesR2 or the looR2). I also note that the loo_R2() does not produce any Bayesian certainty interval estimates for me and complains about the “Pareto k diagnostic values are too high.”

Many thanks,

LOO-R^2 is described in the online appendix of Gelman, Goodrich, Gabry, and Vehtari (2018). R-squared for Bayesian regression models. The American Statistician.

That is possible, because Bayesian R^2 is over-optimistic as it is using the same data to compute posterior and R^2. LOO-R^2 uses LOO-CV to estimate what would R^2 be for new independent data coming from the same data generating process.

It’s more difficult as there are N different leave-one-out posteriors. That online appendix uses Bayesian bootstrap to give alternative non-parametric uncertainties.

See loo-glossary() and Vehtari, Gelman, and Gabry (2017a). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing . 27(5), 1413–1432, for more information on Pareto k diagnostic.

1 Like

Thanks! Very helpful.

Apologies if this is the wrong thread, but it was the closest I found to my question.
Is it possible to use the bayes_r2 (or similar) within the brms model for multinomial model using the ‘categorical’ family? The function gives an error code when I run it.

It’s fine to start a new thread, and in case of very old threads it would be better to start a new thread and possibly link to a old thread if that old thread has useful related content.

The error you should see is
'bayes_R2' is not defined for unordered categorical models.
We’re not aware of a useful definition of R2 for unordered categorical models (including multinomial). If you know something useful, can you provide a pointer to it?

Thanks - I’ll report back if I find something useful.