I’m trying to wrap my head around something. I’m used to using the loocv in the regression situation where I am interested in predicting a single outcome variable, say y which is N x 1, conditional on X’s arranged in an N x p matrix with p predictors. This is straightforward to understand as the elpd is calculated based on each y_(-i). But in the case of a method like factor analysis, there is no “outcome” per say and one has a matrix of variables y, say to be factor analyzed. In this case, I am not sure what is being “left out”. How is the elpd being calculated in this case?
Factor analysis is essentially just a regression model, where the observed variable is the outcome, the latent variable is the covariate, and the factor loading is the regression coefficient:
y_i = \lambda\eta_i + \epsilon_i^2
In the LOO context, it’s easiest to work with the parameterisation that marginalises out the latent variable for each individual, so that no individual-specific parameters are needed.
Given a model with p outcomes and k latent variables, having:
p \text{ x } 1 Intercept vector \nu
p \text{ x } k Loading matrix \Lambda
k \text{ x } k Latent covariance matrix \Psi
p \text{ x }p Residual covariance matrix \Theta
The likelihood is then given by:
\Sigma = \Lambda\Psi\Lambda^T + \Theta
\\
y \sim MVN(\nu, \Sigma)
Now that the likelihood is not dependent on individual latent variable parameters, you can more easily hold-out observations to be predicted.
In concrete terms, the LOO-CV in this context is assessing the extent to which the latent variable model is generalisable to new observations - it provides some indication of the extent of model overfitting
Thanks. I understand the factor model, but I think my question is simpler. Is it the persons entire p-vector
of responses that is being held out at once?
You can specify different thing that can be left out. But the most common use would leave out an entire row/person data. So leaving out all responsesn from each person.
In blavaan we use the LOOIC, which approximates this leave one subject out approach
Hi Andrew, if my goal is to use the Bayesian Stacking to deal with the multimodality problem, which LOOCV do you think is more appropriate in this factor analysis? Say leave one element out or leave one column out? Many thanks!
Thanks, @avehtari and @andrjohns. I think I understand this depends on if I want to predict one observation or all observations per individual, and interestingly, both predictions make sense to my data. Here are the two pipelines.
The first is to use Leave-one-element-out + PSIS_LOO + Bayesian stacking.
The second is to use the leave-one-column-out, but PSIS_LOO might not be a good approximation because more data points are left out, so it would be better to refit the model N times using n-fold-CV?
For the first one, I want to make sure this is correct for the multivariate case because I didn’t see multivariate examples in the Bayesian Stacking paper.
For the second one, I want to know if I don’t want to refit the model N times, how good is the PSIS_LOO approximation? I work on very large datasets and can’t afford to refit the model multiple times.
If I understood your model correctly, it is correct. It’s also possible that the difference in stacking weights is negligible between leave-one-out and leave-one-column-out.
Good thing is that PSIS-LOO has built-in diagnostic, so you will know whether it’s useable or not (see, e.g., Sections 4 and 5 in Roaches cross-validation demo)