I’ve got a decently large RI-CLPM (N = 7796) that has Rhats < 1.01, does not have pareto k warnings, but has a p_loo >> p (see below). This concerns me since the loo glossary indicates that this can be a sign of severe misspecification. Incidentally, this abnormally high p_loo is also leading to an inability to generate fit indices using p_loo due to p_loo > sample statistics (here 171; see warning below). The model uses blavaan’s default priors, but my understanding is that the large amount of data should overwhelm the prior anyway. Is there another diagnostic check that I’m missing that would help me explain and shrink the very large p_loo? Relatedly, is a p_loo > p not an issue if pareto k diagnostics aren’t a problem? I’ve attached the model syntax in case that would be helpful. Thanks!
P.S. In case the output isn’t readable, npar = 109 and p_loo = 214.461.
Edit: Since I can’t use p_loo (or p_waic) to generate fit indices, I have to use p_dic. However, those indicate very good fit ((BGammaHat = .996, BCFI = .993), so I’m left with a bit of an unclear picture.
3: blavaan WARNING: The effective number of parameters exceeds the number of sample statistics (covariances, etc.), so fit index calculations may lead to uninterpretable results.
See below. Please let me know if this isn’t what you meant or if you need more info.
For some extra context, this issue replicates for some other traditional CLPMs I’m running, so I’m trying to figure out if there is something universal about how the model is set up that is causing these p_loo >> p (and p_loo > sample statistics) issues, and if they’re even issues in the first place.
Thanks for everything you do for the stats community!
As the person who wrote a large part of that Stan file, I expect it will take a good deal of time for others to parse. (I tried to see how far I could go with a precompiled Stan file, and the answer is “far” but at the sacrifice of readability.)
I noticed a similar issue to what you describe in a two-level SEM, when the model assumes that between-person variability = 0 for a certain variable and that assumption conflicts with the observed data. It seems that a similar thing may be happening here. I have a draft of a paper describing the issue that I need to get online, and I could send it off-list if you are interested.
The issue also seems to go away when you generate data from this model (say, using lavaan::simulateData()) and fit the model back to the data. I think this supports the idea of a model-data conflict, as opposed to a software bug.
First, thank you for all the work you do. I’d also definitely like to give that paper a read, feel free to send off-discussion.
As for the model-data mismatch, I am certainly open to that possibility, just hoping to get some clarification on how to identify the misspecification issue if this p_loo situation indeed indicates severe misspec despite Pareto ks < 0.7. However, the BRMSEA/BCFI/BGammaHats I have are indicative of acceptable to excellent fit. If data-model mismatch were indeed quite severe, wouldn’t that show up in the form of a high BRMSEA and a low BCFI/BGammaHat?
Ok, I will send the paper soon. I believe the model-data mismatch may not be severe from a practical point of view, but it is a specific mismatch that causes problems for the WAIC and PSIS-LOO computations.
Here is a simple setup to illustrate the issue (though I have not spent enough time on your model to guarantee that it is the same problem!). Say that each student supplies 4 test scores, and students are nested in schools. For each student i, you could specify a multivariate normal model like
y_i \sim \text{N}(\mu, \Sigma_w + \Sigma_b)
that decomposes the covariance matrix into a “within student” part and a “between student” part that captures covariance between students in the same school. Now say that the first row and column of \Sigma_b are fixed to 0, perhaps to say that the first test score is exclusively attributed to individuals and that the school plays no role in that variable. When you compute a loo metric treating schools as units (so a log-likelihood is calculated for each school as opposed to for each student), then you can see behavior like what you describe. I think it is related to the “scaled 8 schools example” from Aki’s paper “Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC”, which shows that WAIC and PSIS-LOO break down when there is no pooling across schools.
Thanks for the example! Some context I forgot to include but which might be helpful is that this sample has 542 missing patterns and missingness increases in later waves. Could this be in some way the cause of the high p_loo despite very good fit indices? In the broadest sense, I’m really trying to understand if p_loo >> p is still suggestive of “severe misspecification” (or even concerning misspecification) when there are no concerning pareto k values present and other evidence (e.g., fit indices) indicate no serious model concerns. Do you (or Aki) have an indication if the p_loo > p interpretation changes in such a case?
I cannot yet find the specific issue, but I suspect this is a situation where the default blavaan computations for the information criteria do not work well for your particular model. Especially if other model checks look good, I would not be overly concerned about this behavior. But I agree that it is unusual and could use further explanation.