Presenting small elpd_diffs based on multiple fits of the same models

[blokeman]

@avehtari

I don’t see any reason to multiply elpd_loo’s by -2, and there is no benefit of doing that, why not just report on the original scale.

I fully agree that it’s nicer to have the utility/loss statistic and p_loo on the same scale. But I’m stuck with the Deviance scale for my present project. Therefore I’d like to know how to calculate mcse_looicdiff. I’m currently using a home-made function for my model comparisons which calculates this mcse_looicdiff as

MCSE_LOOICDIFF = \sqrt{(2*\text{MCSE_LOO}_a)^2 + (2*\text{MCSE_LOO}_b)^2}

and I sure hope that someone more skilled at math/stats can confirm whether this is correct, lest I suffer public humiliation after the study goes to print.

EDIT: I’d also like to know, regardless which scale we use, whether the Central Limit Theorem can reasonably be assumed to apply to the variability of MCSE_DIFF/MCSE_LOOICDIFF.

[avehtari]

@blokeman

I’m stuck with the Deviance scale for my present project. Therefore I’d like to know how to calculate mcse_looicdiff.

First compute everything in log score scale, and in the end multiply all estimates by -2 and all SE’s by 2.

@blokeman

I’d also like to know, regardless which scale we use, whether the Central Limit Theorem can reasonably be assumed to apply to the variability of MCSE_DIFF/MCSE_LOOICDIFF.

If you are using loo package and Pareto smoothed importance sampling then
Pareto smoothed importance sampling paper provides the conditions when MCSE of elpd_loo is valid and can be assumed to be accurate. Assuming independent posterior draws are used for the compared models, then taking the difference doesn’t change the conditions.

[blokeman]

Great, thanks!