Revised Uncertainty in Bayesian leave-one-out cross-validation based model comparison

Publicity
1

We have made a major revision of Uncertainty in Bayesian leave-one-out cross-validation based model comparison with @tuomassivula, @mans_magnusson, and Matamoros. This paper provides a theoretical justification for the normal approximation of the elpd difference used in loo and ArviZ packages (elpd_diff, diff_se).

We have clarified the goal of the paper, made more clear that the uncertainty is described by the posterior of unknown elpd difference, that posterior arises from a model for the future data distribution, a minimal assumption model for the future data is a flat Dirichlet process, the posterior mean and variance of that flat Dirichlet process have analytic solution, and the normal approximation is based on these.

We have added three case studies illustrating the model comparison and computation of the probability of model A being better than B.

We have added to the discussion more references to related papers and papers using the results of this paper.

2

And here’s the link to the code for the new case studies for Uncertainty in Bayesian LOO-CV Model Comparison

3

Uncertainty in Bayesian leave-one-out cross-validation based model comparison has now been published online in Bayesian Analysis, doi:10.1214/25-BA1569 (with minor edits compared to the June version).

We have a PR for loo package, adding new columns to loo_compare() output. Check out how the new output looks in a variant of the paper case study

Please, check out the case study and possibly try also that PR, and provide feedback!