I want to use leave-one-out (LOO) cross-validation (CV) for a hierarchical model. My intuition is not to use the information on the group assignment of the left-out data point when I compute the likelihood of that point given the model fitted to the rest of the data.

Since, if I would use the group assignment of the left-out data point for LOO-CV, I would treat the group asignment as a explanatory variable (with corresponding parameters having a special, informative prior – the hierarchical prior including information of the data itself). And this is not what I want. I use the information of the group asignement only to account for the fact that the data points aren’t independent given the non-hierarchical model for a known reason (but hopefully are independent given the group assignment), finally to get the variance correct(er).

Questions:

- Is this view correct?
- Is LOO-PSIS an approximation for the LOO-CV without using the assignment to the hiararchical group of the left-out data point?
- Should I use or not use the assignment to the hiararchical group of the left-out data point as a feature when I do exact LOO-CV for the data point where the pareto-k value was too high in the LOO-PSIS (I use the loo package in R.)?

From my understanding of “Vehtari/Gelman/Gabry. 2017. ‘Practical Bayesian Model Evaluation Using Leave-One-out Cross-Validation and WAIC’.” the answer to question 2 is ‘No’. (I understand that \tilde{y}_{i} is the random variable of an unknown data point given features of data point y_{i} in equation (7).) But maybe there is an argument why this is not a problem if I then sum up over all data points.