Hello,
I’m working on simulations with hierarchical models and I’m comparing models fitted with stan / nlme and other approaches. Besides other things, I’m interested in comparing different models (fitted with the same package) using information criteria. I read and really enjoyed the paper on LOO-CV and I’m definitely gonna use that (for stan models at least). However, there is no implementation for non-bayesian models, for example, those fitted with nlme. I thought that computing matrix of pointwise loglikelihood from the nlme fit and feeding it into loo function could work. Nevertheless, the nlme provides only mean estimates for variance components, therefore, the pointwise log likelihood is not complete (a lot of the variability from variance components is missing). Do you think, that computing loo using this approach is complete nonsense, or it still can lead to a better result than AIC for those models?
loo function in loo package uses importance sampling and requires draws from the posterior and it doesn’t produce valid results if the draws are from approximation.
I don’t know, but if you are comparing models some of which are fitted with stan and some with nlme, then you need to use something better than either of these for nlme. It’s likely that you need to do exact cross-validation, e.g. with k-fold-CV.
Thank you for your answer, I thought it couldn’t be such an easy solution.
No, I’m not planning on comparing models across packages.