If elpd_diff/se_diff > |2|, is this noteworthy?

Good question. I think @avehtari could help in clarifying this.

Nowadays, I personally feel comfortable, if I have a large N, and if \mathrm{elpd}_{\mathrm{diff}} / \mathrm{SE}_{\mathrm{diff}} \geq 4. It would be wonderful if Aki could add a references to his FAQ which one can cite in papers to solidify this claim. In short, Aki’s post here:

https://statmodeling.stat.columbia.edu/2020/08/27/more-limitations-of-cross-validation-and-actionable-recommendations/

is what we have to rely on at the moment :)

1 Like