It it possible to do VI in Stan such that some variables are approximated by an mvn with a dense covariance and others with a diagonal covariance?
The use case I have in mind is a sparse gaussian process wherein the inducing covariance is dense but the covariance on hyperparameters and possibly other latent variables is diagonal.
I don’t think it’s possible, but see Intermediate between mean-field and full-rank ADVI for a somewhat similar case (the low rank ADVI is implemented in a PR that has however not yet been merged into Stan main branch, so you can use it, although with some hassle)
Low rank + diagonal advi hasn’t been merged as the current implementation is failing for more than one posteriordb posterior for which full rank and diagonal advi don’t fail. I’m not aware that someone would be actively working on fixing the issue.