Enquiry LOO output in brms

Modeling
, ,
1

Dear everyone,

This is the first time I post something on this forum, so please let me know in case more information is needed. I’m working with the brms package (version 2.16.1) in R (version 4.1.2).
I have a problem regarding the LOO function that I use to find the best predictive model for my data. I use this function on several models that differ in their interaction terms. So for example, I have a base model with two predictors to predict the reaction times on a mathematical learning task:

*ModBase <- brm(medianRT ~ centre_Block * Task + (1 + centre_Block|Subject), family=lognormal, iter=10000, data=df_all, save_pars = save_pars(all = TRUE))*

Then I add the predictors stimulation (sham or real brain stimulation) and a electrophysiological measure called aperiodic exponent in my next models:

ModStim3 <- brm(medianRT ~ Stimulation + centre_Block * Task + centre_Aperiodic_Baseline + (1 + centre_Block|Subject), family=lognormal, iter=5000, data=df_all, save_pars = save_pars(all = TRUE)) *
ModStim4 <- brm(medianRT ~ centre_Block + Task * Stimulation * centre_Aperiodic_Baseline + (1 + centre_Block|Subject), family=lognormal, iter=5000, data=df_all, save_pars = save_pars(all = TRUE))*
ModStim5 <- brm(medianRT ~ centre_Block * Task * Stimulation * centre_Aperiodic_Baseline + (1 + centre_Block|Subject), family=lognormal, iter=5000, data=df_all, save_pars = save_pars(all = TRUE))*
ModStim6 <- brm(medianRT ~ centre_Block + Task + Stimulation + centre_Aperiodic_Baseline + (1 + centre_Block|Subject), family=lognormal, iter=5000, data=df_all, save_pars = save_pars(all = TRUE))*
ModStim7 <- brm(medianRT ~ centre_Block * centre_Aperiodic_Baseline * Stimulation + Task + (1 + centre_Block|Subject), family=lognormal, iter=5000, data=df_all, save_pars = save_pars(all = TRUE))*
ModStim8 <- brm(medianRT ~ centre_Block * Stimulation + centre_Aperiodic_Baseline + Task + (1 + centre_Block|Subject), family=lognormal, iter=5000, data=df_all, save_pars = save_pars(all = TRUE))*
ModStim9 <- brm(medianRT ~ centre_Block + Task + Stimulation * centre_Aperiodic_Baseline + (1 + centre_Block|Subject), family=lognormal, iter=5000, data=df_all, save_pars = save_pars(all = TRUE))*
ModStim10 <- brm(medianRT ~ centre_Block * centre_Aperiodic_Baseline + Task + Stimulation + (1 + centre_Block|Subject), family=lognormal, iter=5000, data=df_all, save_pars = save_pars(all = TRUE))*
ModStim11 <- brm(medianRT ~ centre_Block + Task * Stimulation + centre_Aperiodic_Baseline + (1 + centre_Block|Subject), family=lognormal, iter=5000, data=df_all, save_pars = save_pars(all = TRUE))*
ModStim12 <- brm(medianRT ~ centre_Block * Task * Stimulation + centre_Aperiodic_Baseline  + (1 + centre_Block|Subject), family=lognormal, iter=5000, data=df_all, save_pars = save_pars(all = TRUE))*

Then, I use the loo function to compare them: loo(ModBase, ModStim3,

ModStim4, ModStim5, ModStim6, ModStim7, ModStim8, ModStim9, ModStim10, ModStim11, ModStim12, moment_match = TRUE)*

However, I noticed that when I rerun the brms models and compute the LOO output, the best fitted model differs sometimes. At first I was running these models on a 4 core computer, but when I checked the LOO output on a 16 core computer and it still differs. I should note that the output usually differs between 4 model (the Base model, ModStim3, ModStim5, and ModStim 12) and I have used set.seed() in R. I find it strange that the best model differs when running the LOO function, and therefore I’m not sure how to proceed further since I don’t want to make wrong conclusions about my data.

The output might differ every time since the EPLD between the first three models is really small and that by recalculating the ELPD with another test point in the LOO procedure causes the difference in best predictive model. However, it surprises me that the ELPD of my base model and two other model including extra predictors are so close to each other.

Here is an example of my last LOO output:

Model comparisons:
                     elpd_diff se_diff
ModStim12  0.0       0.0   
ModBase   -0.1       0.8   
ModStim3  -0.1       0.8   
ModStim5  -1.0       1.1   
ModStim6  -1.3       1.7   
ModStim4  -1.4       1.8   
ModStim10 -1.5       1.9   
ModStim11 -1.6       1.7   
ModStim7  -1.7       2.0   
ModStim9  -1.9       1.8   
ModStim8  -1.9       1.8

I really hope someone can help me with this issue since I’ve been struggling with this for some time. Perhaps I am doing something wrong? Or if I’m not, and the LOO is indeed not reliable in this case, how should I move forward with finding the best predictive model?

Many thanks in advance,
Nienke

EDIT by Aki: added ticks to show code in code blocks

1 Like
2

LOO is reliable in this case, and the differences between the predictive performances of models are negligible (as also shown by se_diff).With finite MCMC sample size there will be some variation between runs. You can also see the estimated MCSE for elpd_loo in the loo output (not shown in the comparison).

2 Likes
3

Thank you for the confirmation that the predictive performances of the models are negligible, and that the LOO function is reliable in this case!