I am comparing 3 (quite different) models of cognitive mechanisms underlying the performance of clinical patients (schizophrenia) in an experimental task.
I have a few options as to how to perform model comparison. Here the pros and cons as I understand them, but I’d appreciate some feedback.
Ideally I would do a mixture multilevel model where the theta is conditioned on participant. This is in practice impossible to properly fit.
I can run loo and stacking weights on the 3 multilevel models. Pros: pooling. Cos: many in the field would object to participants being assumed as similar.
I can run the models on each individual separately and do loo and stacking weights at the individual level. Pros: individual weights, so different models can be better for different individuals. no Pooling. Cons. No pooling.
I could extract pointwise loo scores, consider them by participant and do a post-hoc individual level model comparison. Pros: pooling and individual weights. Cons: Pooling. A bit convoluted.
At the moment I have implemented 2 and 3, with partially complementary results.