We have made a major revision of Uncertainty in Bayesian leave-one-out cross-validation based model comparison with @tuomassivula, @mans_magnusson, and Matamoros. This paper provides a theoretical justification for the normal approximation of the elpd difference used in loo and ArviZ packages (elpd_diff, diff_se).
We have clarified the goal of the paper, made more clear that the uncertainty is described by the posterior of unknown elpd difference, that posterior arises from a model for the future data distribution, a minimal assumption model for the future data is a flat Dirichlet process, the posterior mean and variance of that flat Dirichlet process have analytic solution, and the normal approximation is based on these.
We have added three case studies illustrating the model comparison and computation of the probability of model A being better than B.
We have added to the discussion more references to related papers and papers using the results of this paper.