I am conducting a forward iterative model comparison analysis of longitudinal data, similar to that discussed by Singer and Willett in Applied Longitudinal Data Analysis but using loo-cv methods in stan/brms. Excuse the mixed-effects model nomenclature, but I start with a null model (with no fixed p…

I’ll take a step back from loo specifically. The standard error (SE) of a parameter refers to the standard deviation of the parameter itself. We can use the parameter value and its standard error to approximate the distribution of the parameter as a normal distribution. When Aki says “When the norma…

I recommend to read and cite Tuomas Sivula, Måns Magnusson, and Aki Vehtari (2020). Uncertainty in Bayesian leave-one-out cross-validation based model comparison. arXiv preprint arXiv:2008.10296 . which is also listed as a reference in CV-FAQ 15: How to interpret Standard error (SE) of elpd differe…

Excellent advice @avehtari . Thanks!

[image] avehtari: It is common to use arbitrary 2SE for reporting and illustration, but there is nothing magical in 2SE Thanks @avehtari and @torkar , much appreciated. Aki is the “SE” in the “2SE” quoted above not the same as “se_diff”? And, if they are not the same, what “SE” are you referr…

[image] llewmills: Aki is the “SE” in the “2SE” quoted above not the same as “se_diff”? And, if they are not the same, what “SE” are you referring to? SE is standard error in any normal approximation. Normal approximations and SE are common in other contexts than elpd, too. se_diff is the SE …

[image] avehtari: SE is standard error in any normal approximation. Apologies for one more dumb question @avehtari , but is there an easy way get this SE of the normal approximation from a loo or loo_compare object?

Hi Llew, The standard error of the loo estimates are given in the loo output: Computed from 4000 by 262 log-likelihood matrix Estimate SE elpd_loo -6247.8 728.0 p_loo 292.4 73.3 looic 12495.5 1455.9 And the standard error of the differences in estimates is given in the…

Thank you @andrjohns I’m just still confused how to apply Aki’s rules. He says above “When the normal approximation is assumed to be well calibrated you can use that approximated distribution to make inference and decisions as with any well calibrated distribution describing uncertainty” But how?…

To make this more concrete, I have one model (model 1) whose loo output is Estimate SE elpd_loo -1600.4 17.5 p_loo 91.1 6.0 looic 3200.8 34.9 A second model, Model 2, where I added a single predictor to Model 1 Estimate SE elpd_loo -1581.0 17.7 p_loo 94…

Thank you so much @andrjohns I really appreciate you spelling it out like that. Super clear. forgive my pedantry, but isn’t what you’ve done simply reframing the elpd_diff/se_diff > 2 rule into a credibility interval, Where 19.4/6.7 [1] 2.895522 Because 19.4/6.7 is > 2 we can conclude the models …

Pretty much (as confidence intervals, to be specific). They both provide similar information about the magnitude of the difference relative to its standard error. The elpd_diff/se_diff approach is analogous to the p-value approach (which would compare elpd_diff/se_diff to the quantiles of a standard…

If elpd_diff/se_diff > |2|, is this noteworthy? - #23 by martinmodrak - brms

Topic		Replies	Views	Activity
Loo comparison in reference to standard error General loo	10	3369	May 1, 2018
Quick examples of loo() interpretation Modeling loo	11	2342	July 3, 2020
Reporting the need for a parameter using loo? Modeling	16	443	June 23, 2025
Presenting small elpd_diffs based on multiple fits of the same models Modeling loo	22	1350	October 9, 2023
Interpreting output from compare() of loo Modeling loo , interpret-results	7	4424	March 27, 2024

Loo comparison in reference to standard error

General

loo

10

3369

May 1, 2018

Quick examples of loo() interpretation

Modeling

loo

11

2342

July 3, 2020

Reporting the need for a parameter using loo?

Modeling

16

443

June 23, 2025

Presenting small elpd_diffs based on multiple fits of the same models

Modeling

loo

22

1350

October 9, 2023

Interpreting output from compare() of loo

Modeling

loo , interpret-results

7

4424

March 27, 2024

If elpd_diff/se_diff > |2|, is this noteworthy?

Related topics