I have two models - (i) a linear fixed effect model, and (ii) a linear mixed effect model. The mixed effect model contains additional random effect parameters. Clearly, the number of overall parameters in the mixed effect model is higher. Also, I am providing the same data to both the models.
Which method should I choose for model comparison? I am considering methods such as PSIS-LOO or WAIC as they are invariant to parameterizations (I think). Or should I choose methods that penalize for the number of parameters such as DIC (or some other).
Sorry, I still haven’t learned what fixed, mixed and random effects are. Since these terms can have different definitions, I’m always uncertain what is meant by them (I know there are others in this forum who understand these better than me, For me fixed effect sounds like having fixed parameters, but I guess you man something else). See
Looic and elpd_diff (rstanarm model) - #2 by avehtari reasons for not to use those terms and what would be better terminology.
So now I answer in general, and not specific to your models but assuming that they are related to hierarchical models.
I recommend to first read
Andrew Gelman, Jessica Hwang and Aki Vehtari (2014). Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24(6):997-1016. http://www.stat.columbia.edu/~gelman/research/published/waic_understand3.pdf
which explains how LOO, WAIC and DIC are connected and their differences. Based on that paper you notiice that you can forget DIC. At that time we didn’t yet have PSIS-LOO, so then it’s good to read
Aki Vehtari, Andrew Gelman and Jonah Gabry (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. In Statistics and Computing, 27(5):1413–1432. doi:10.1007/s11222-016-9696-4. [1507.04544] Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC which shows why we now days prefer PSIS-LOO instead of WAIC.
Then you can think whether you would like your model to able to generalize ie predict for new individuals in old groups, or for new individuals in new groups, and then you know whether PSIS-LOO or K-fold-CV would be good for you.
Thanks a lot for your response. I will surely read the papers suggested by you. Based on your recommendation, I would go ahead with PSIS-LOO.