Hi,
What did you mean to write?
I meant to write elpd estimate.
How did you estimate these? The thread is long and you have reported different results, so I’m uncertain what computations you have used for these. You can post the code lines you are using so it’s easier to check what you’ve done.
I ran k-fold with 10 folds for each of my three models in rstan. Then I extracted log-lik and computed pointwise elpd for the k-fold, based on the functions in this link - K-fold cross-validation in Stan | DataScience+. So this was the function for extracting elpd estimate and its se:
kfold ← function(log_lik_heldout) {
library(matrixStats)
logColMeansExp ← function(x) {
# should be more stable than log(colMeans(exp(x)))
S ← nrow(x)
colLogSumExps(x) - log(S)
}
See equation (20) of @VehtariEtAl2016
pointwise ← matrix(logColMeansExp(log_lik_heldout), ncol= 1)
colnames(pointwise) ← “elpd”
See equation (21) of @VehtariEtAl2016
elpd_kfold ← sum(pointwise)
se_elpd_kfold ← sqrt(ncol(log_lik_heldout) * var(pointwise))
out ← list(
pointwise = pointwise,
elpd_kfold = elpd_kfold,
se_elpd_kfold = se_elpd_kfold)
#structure(out, class = “loo”)
return(out)
}
And what are these model_x objects?
Based on what you wrote in this thread - Compare models with k-fold cv - #2 by avehtari, for each model I created a structure consisting of the pointwise elpd estimate and its se derived from the kfold. The structure was defined according to the guidelines in the loo package manual regarding kfold (pp. 12, under “kfold generic”, so it will be compatible with other loo functions - hope I got it right.
Added where?
In the loo_comapre function, I thought I’m supposed to define this if i’m performing it on k-fold.
Are these based on K-fold-CV? And is the table from loo_compare?
Yes, based on th kcv, and this is the table yielded by the loo-compare.
Thanks,
Ofir