Please also provide the following information in addition to your question:
- Operating System: Windows 10
- brms Version:
I have managed to fit several multilevel nonlinear models and now I am trying to compare them, based on the loo criterion.
However, pretty much all my models except for 1 have 3-5 problematic observations each. So R suggested I add the ‘reloo’ option, which would refit the models.
I did that and now for each model all observations have k < 0.7, but I am still not sure I understand how this works. I looked up the R documentation, where it is said that
reloowill refit the original model J times, each time leaving out one of the J problematic observations. The pointwise contributions of these observations to the total ELPD are then computed directly and substituted for the previous estimates from these J observations that are stored in the original
I find this part confusing
The pointwise contributions of these observations to the total ELPD are then computed directly and substituted for the previous estimates from these J observations that are stored in the original
What is it meant by ‘computed directly’. Do we just see how ELPD changes when the model is fitted without an observation i, compared to when the model based on all data points is fitted, and take the difference between the two to be obs i’s pointwise contribution?
Also, I don’t think I understand what exactly the purpose of the reloo option is.
Even if we refit the model and find out what the pointwise contributions of the problematic observations are, and compare these refitted models, how does this change the fact that the original models we had have some ‘influential’ observations?
This is, say we have model1 and model2, with model1 having 5 problematic observations and model2 having 7 problematic observations (out of 590).
After refitting the models using the reloo option, model2 turns out to be the prefered one (higher ELPD). But even if loo_compare says it is the prefered model, how does this change the fact that the original model2 has more influential observations?
Doesn’t this mean the original model2 would still do more poorly in predicting, compared to model1?
I hope my question makes sense.