I have 2 models fitted on few datasets that I would like to compare. They are nested so that model 2 is a special case of model 3.
I expect that in some datasets, the added parameters in model 3 will be relevant but not in all of them, and I am unsure on how to test this.
Should I simply use PSIS-LOO? In which case, I should only consider differences in elpd to be significant when they are above 5 SE, is that correct?
In this case, given that my models are nested, can model 2 outperform model 3? Or should I interpret a small difference in elpd relative to the SE as there being no advantage of adding parameters to my model?
Would computing stacking weights make sense here? I’m still struggling a bit to understand how they work.
Should I rather only fit my full model and look at the estimated parameters? Or do something else entirely maybe?
It depends on the model and priors, but if they are sensible then it’s unlikely that model 2 would outperfom model 3. Small difference as mentioned in 11 and 15 in CV FAQ indicates that there is no advantage of using the more complex model measured with log score. The more complex may still be better in extreme tails of the predictive distributions, so it depends on your application. See 5 in CV FAQ
Stacking weights provide complementary information (sorry I need stop now, and don’t have ready answer in FAQ and I’ll get back to this)