Hi! I have a question on model selection across same model structures across different response variables.
Say I have 3 response variables. Each response variable has different sample sizes. However, for each response variable, I fit 2 different models: a null model that only includes the intercept (s) and a full model that includes a set of covariates. I fit the same null and full model structure to each response variable, and for each response variable, I perform leave-one-out cross validation to know whether for that specific response variable the null or the full model is preferred.
What I want to know is how I go about deciding on the best model to use across all response variables. Can I sum the looicâs across responses that shared the same structure? (i.e., looic_null_response1+looic_null_response2+looic_null_response3=summedlooic_null; and looice_full_response1+looic_full_response2+looic_full_response3=summedlooic_full)
or is there a better alternative? Any guidance and references will be greatly appreciated :)
Yes. You can think of this as having one model for all response variables, but the model parts are just independent. Then summing pointwise log scores is the right thing to do, and you can do this also by summing then the sums of pointwise log scores.
See also in CV-FAQ why I think the IC form obtained by multiplying by -2 does not make sense
Thank you Aki, this is very useful!
Do you know of any manuscripts that specify this?
I have read yours (i.e., Vehtari, A., Gelman, A. and Gabry, J. (2017) âPractical Bayesian model evaluation using leave-one-out cross-validation and WAICâ, Statistics and Computing , 27(5), pp. 1413â1432. doi: 10.1007/s11222-016-9696-4.) and others related to LOO (and LOOIC) but canât see this being implemented. Thank you very much in advance!
Can you clarify what do you mean by âthisâ?
Sorry for the lack of clarity! This= that âsumming pointwise log scores or the sums of pointwise log scores is the right thing to doâ when we have different response variables with the same model structure. Thank you!
If the response variables are assumed to be conditional independent (depending on the case assuming conditional exchangeability can be enough) then this follows from the basic probability calculus. The model structures donât need to be even the same for different response variables. It is enough to assume that you are making a joint model, but it just happens that the model parts are independent and thus you can do the posterior inference for each part separately and combine the results in the end. Thus, Iâm certain that modellers have used that, but itâs also likely that there is no good reference which explicitly explains that. Iâll add to my TODO list to add this to my CV-FAQ.