In “Uncertainty in Bayesian leave-one-out cross-validation based model comparison” https://doi.org/10.1214/25-BA1569 we showed when LOO-CV elpd_diff and se_diff normal approximation uncertainty quantification in model comparison is well calibrated. We (with @Florence_Bockting) have now merged a related PR to loo package.
This PR adds new columns to the LOO comparison. p_worse is the probability that a model has worse performance than the best model. diag_diff tells whether that probability can be trusted. diag_elpd is reminding if the single model LOO computation diagnostics are indicating issues.
It took quite long time to merge the PR as we wanted to be careful with the design choices as we had to also change from matrix output to dataframe to support text in diagnostic columns. Now the model names are also columns, making it easier for further processing and pretty printing. You can see examples of how it looks like in LOO uncertainty case study by me and @Florence_Bockting
You can get this new feature by installing loo from github.
As the loo_compare output changed from matrix to dataframe, there were few other packages that needed fixes, so if some package errors try installing their latest version. We are planning to make the next CRAN release before mid-June and then all dependent CRAN packages should work