In a few posts over the past few years I’ve hinted at some developing work on a prior model for latent factor models that incorporates meaningful domain expertise and facilities computation without having to deal with orthogonal matrices directly. I am happy to share that the work has just been released publicly, [2208.07831] Structured prior distributions for the covariance matrix in latent factor models.
In this paper Sarah Heaps constructs a prior model for latent factor loadings that provides interpretable constraints on the implied observation component covariances. The prior model is structured around a covariance matrix which can be chosen to impose symmetry amongst the components or based on meaningful structure in a given application, such as phylogenetic distances or the Gram matrix from a gaussian process correlation function. There are also some clever implementation strategies that make its easier to work with the residual degeneracies that remain even with this prior model.
If you’re working with latent factor models definitely give the paper a read and experiment with this new prior modeling strategy which can substantially improve performance.