Hello,
I have 4 linear regression models and their only difference is one column of the design matrix (the other 20 columns are all the same). The task is to set a model selection strategy to select only one model among 4. I first considered loo_compare to select those who have considerably better elpd_loo (elpd_diff > 2*se_diff), then from the “surviving” ones, I prioritized Bayesian R2 to select one(s) that have similar elpd_loo but probably better fitting. In the end, if there is still more than one model left, I plan to use the absolute z score of the parameter corresponding to the column that is different across 4 models. I have two questions could you please help?
-
elpd has se_diff to indicate whether the difference is considerable, but R2 and z scores do not. There will be always a numerical difference such as mode1 R2 = 0.86123, model2 R2 = 0.86133, but I am not sure the difference has any practical meaning. Is there a way to tell if the difference between R2s is big enough? I guess I shouldn’t compute the distribution of R2 difference by doing R2(model1) - R2(model2) since the samples are not paired. Then, should I do one-way ANOVA? or maybe just set an arbitrary R2 difference like larger than 0.01 is big enough? Same issue for z score.
-
Does it make sense to select one model using zscore of the parameter corresponding to the column that is different across 4 models after loo and R2 can not tell a difference? Note that the only difference across 4 models is one column in the design matrix
Thanks a lot