Calculating standard error of difference in LOOIC between models

Interfaces brms
1

On page 19 of this tutorial on ordinal regression in brms https://psyarxiv.com/x8swp/ there is a table where the authors compare LOOIC in four different ordinal regression models.

The LOOIC and standard error of each model are: M1: 2040.3 (31.0), M2: 2042.8 (31.5), M3: 2043.7 (30.9), M4: 2039.0 (31.2).

Now getting the difference in LOOIC between each pair of models is very straightforward, it’s just LOOIC_{model1} - LOOIC_{model2}.

What I can’t work out is where the figures listed as athe standard error of the difference come. from?

Is there some way to extract chains of LOOIC for each model from each model object and then subtract one from the other to yeild a chain of LOOIC differences then calculate the standard error of those differences in LOOIC scores? Or am I way off?

2

You’re not way off. The pointwise component of the object returned by loo() has ELPD values for each data point. We subtract those and then compute the standard error of the difference. The loo_compare() function does this internally using this helper function:

2 Likes
3

Oh I just noticed I used ELPD whereas you were referring to LOOIC. LOOIC is just -2 * ELPD. We like to focus on ELPD directly but we report LOOIC because of the tradition of scaling by -2 for the information criteria. Both ELPD and LOOIC contain the same information though since they only differ by a constant factor.

4

Brilliant thank you @jonah