Compare models with k-fold cv

[avehtari]

@kangchonsara

When comparing models using k-fold cv, what is the good way to compare models?

I read this article

Thanks for asking! I only now realized that there is an error in that article!

@kangchonsara

and here the log likelihoods of held out data points after k-fold refitting are saved and the models are compared by: compare( loo (loglik_model1), loo(loglik_model2), loo(loglik_model3) ).

In the article loo is called also for logliks from k-folf-cross-validation which doesn’t make sense.

@kangchonsara

Do I need to put log likelihood to loo(loglik_model1) instead of just doing kfold(loglik_model1) so that the effective number of parameters is taken into account?

Neither loo nor kfold compute effective number of parameters directly and p_eff shown in loo output is just something which is computed afterwards for additional diagnostic value.

@kangchonsara

doing kfold(loglik_model1), kfold(loglik_model2), kfold(loglik_model3) ?

Using the function you included, this is the correct thing to do.

@kangchonsara

kfold function basically sum across likelihood,

For comparison, it is better to use pointwise values. If you form the structure correctly you could use loo_compare function to compute elpd_diff’s and SE of the pointwise difference