I fitted a stan_glmer model to my data and wanted to use loo to compare models with or w/o a fixed effect of interest. This is my full model:
post_rep_full <- stan_glmer(Resp13 ~ (Distance * Context ) + (Distance * Context || Subject), data = myData, family = binomial(), iter=4000, chains=6, prior_intercept = normal(0,7.5), prior = normal(0,2.5), QR = TRUE) # i removed the int term. the model does it.
and then I dropped my strongest fixed effect (this fixed effect had a coefficient of 0.56 [95% HDI = [0.44-0.68]]), keeping the random-effects structure untouched:
post_rep_no_dist <-stan_glmer(Resp13 ~ (Context + Distance:Context ) + (Distance * Context || Subject), data = myData, family = binomial(), iter=4000, chains=6, prior_intercept = normal(0,7.5), prior = normal(0,2.5), QR = TRUE)
Then, I calculated loo for each model
loo_full<- loo(post_rep_full) loo_no_distance = loo(post_rep_no_dist)
and comparing the models, I got
elpd_diff se -0.4 1.1
suggesting (if I understand correctly) that we are indifferent between the models, with an unreliable preference to the full model. How come that the full model isn’t preferred in this case? I received a similar result when I dropped another fixed effect with a coefficient around zero.
this is what happens when I drop each single fixed effect and compare all models. All the values seem very similar, I don’t know why…
looic se_looic elpd_loo se_elpd_loo post_rep_full 20498.0 117.7 -10249.0 58.8 post_rep_no_dist 20498.7 117.8 -10249.3 58.9 post_rep_no_context 20498.9 117.6 -10249.4 58.8 post_rep_no_interaction 20499.3 117.7 -10249.7 58.8 p_loo se_p_loo post_rep_full 71.7 0.8 post_rep_no_dist 73.1 0.8 post_rep_no_context 72.2 0.8 post_rep_no_interaction 71.8 0.8