# Latent factor loadings
**URL:** https://discourse.mc-stan.org/t/latent-factor-loadings/1483
**Category:** Modeling
**Tags:** specification, fitting-issues
**Created:** [August 4, 2017, 5:57pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483 "2017-08-04T17:57:12Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@jpritikin](https://discourse.mc-stan.org/u/jpritikin)
#### Post date: [August 4, 2017, 5:57pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/1 "2017-08-04T17:57:12Z")
Greetings!
Latent factor models are popular in research. A single factor model with 5 indicators is shown on the OpenMx [front page](http://openmx.ssri.psu.edu/). Due to rotational indeterminacy, it is necessary to constrain the sign of one of the factor loadings. For background, see [this article](https://link.springer.com/article/10.1007/s11336-012-9259-3). Typically, the first loading is constrained positive.
Edgar Merkle (blavaan) and I have put together code to try to fit this kind of model using Stan 2.16. The problem we’re running into is that, sometimes, some of the chains get stuck with the first loading close to zero and the remaining loadings with the opposite of the expected sign. I’ve even tried to specify starting values for the first loading, but it doesn’t help consistently. Either the loading starts near 0 or 1 and stays there during sampling. When this happens, of course the Rhats are terrible in the 20s and 30s.
I’ve tried to make the code as simple as possible to make it easy to understand. Looking forward to your suggestions.
Thank you.
[test2.R](https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/original/1X/74c7bec05948162290534cb9383e3912ba07a267.R) (3.1 KB)
[test1.stan](https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/original/1X/101c1b12e18bb941a65225ea0e1f1b6bce37d11e.stan) (1004 Bytes)
---
### Author:  [@Trung\_Dung\_Tran](https://discourse.mc-stan.org/u/Trung_Dung_Tran)
#### Post date: [August 5, 2017, 8:52am UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/2 "2017-08-05T08:52:31Z")
Hi,
If we have prior information that all loadings are positive then we can constrain that and it is fine for me.
> [@jpritikin](#):
>
> some of the chains get stuck with the first loading close to zero and the remaining loadings with the opposite of the expected sign
I had encountered the same problem unless the positive value for the first loading is far from 0.
If the signs for the loadings can be both positive and negative, what I have done so far is that running model without constraining loadings. At the end, for the summary, just take a chains that has the same signs.
If you see in the plot I attach here, then the chains are symmetric about 0. For final estimate, I take chains has the same signs as 1, 3, and 4 in the plot, or if you do not want to loose chain 2 then you could change the sign of that chain in R before getting the final estimates.
I hope this is useful for you!
Tran.
[Rplot\_lambda.pdf](https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/original/1X/1c90676ce106164b465843aa7db8ab44546b2918.pdf) (915.2 KB)
---
### Author:  [@jpritikin](https://discourse.mc-stan.org/u/jpritikin)
#### Post date: [August 5, 2017, 2:46pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/3 "2017-08-05T14:46:53Z")
Yes, of course we can guess the sign of the loadings after running some chains. When I write up my paper, are reviewers going to accept that? I’d much prefer it if Stan could figure out the signs based on the data.
---
### Author:  [@Trung\_Dung\_Tran](https://discourse.mc-stan.org/u/Trung_Dung_Tran)
#### Post date: [August 5, 2017, 3:05pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/4 "2017-08-05T15:05:49Z")
Because they are solutions to the equation that
> Cov(Y)=Lambda\*Lambda^T + Sigma
so when we switch the signs it does not change the product. I think it is reasonable that Stan recognizes the maximum from the posterior distribution ans it samples from those maximum.
When I set lambda1\>0 I mean to take a half of the parameter space, however, in case that some loadings close to 0, the output does not give estimates as expected. That is why I have to follow what I have explained above.
I would like to hear more about this from others’ experience!
Tran.
---
### Author:  [@jpritikin](https://discourse.mc-stan.org/u/jpritikin)
#### Post date: [August 5, 2017, 5:53pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/5 "2017-08-05T17:53:24Z")
Yes, it is maybe possible to guess the signs of the loadings for a single factor. However, if there are 2 or more factors then I think it would be very difficult to guess the signs of the loadings.
---
### Author:  [@bgoodri](https://discourse.mc-stan.org/u/bgoodri)
#### Post date: [August 5, 2017, 6:10pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/6 "2017-08-05T18:10:53Z")
For a one factor model, I would usually declare the loadings to be unconstrained in the `parameters` block and in the `generated quantities` block, multiply stuff by -1 as necessary to force the sign of one loading to be positive.
For a multi-factor model with an identity matrix for the correlations among the factors, a `cholesky_factor_cov` can work well as a loadings matrix since it is lower trapezoidal with positive diagonal entries. But that can run into problems like the one you encountered, if for example, the first manifest variable has little signal.
For a multi-factor model with correlated factors, it is harder. One option would be to use a `positive_ordered` type for the factor variances and restrict some loading on each factor to be 1. But that too can run into problems if two of the factor variances are close together.
This problem is quite similar to the one discussed in the mixture case study
[http://mc-stan.org/users/documentation/case-studies/identifying\_mixture\_models.html](http://mc-stan.org/users/documentation/case-studies/identifying_mixture_models.html)
---
### Author:  [@jpritikin](https://discourse.mc-stan.org/u/jpritikin)
#### Post date: [August 5, 2017, 7:36pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/7 "2017-08-05T19:36:23Z")
Oh, I see. Here’s [test1.stan](https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/original/1X/03823f3396fe8bc972af21ac0fee5becf5f368a9.stan) (927 Bytes)
modified with your idea. It seems to work great.
---
### Author:  [@bgoodri](https://discourse.mc-stan.org/u/bgoodri)
#### Post date: [August 5, 2017, 9:20pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/8 "2017-08-05T21:20:20Z")
If you multiply the raw loadings by -1, you need to also multiply the raw latents by -1.
---
### Author:  [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)
#### Post date: [January 20, 2021, 4:04pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/9 "2021-01-20T16:04:06Z")
Sorry to dredge up an old thread, but arrived here from the [Arxiv paper describing the Stan update for blavaan](https://arxiv.org/abs/2008.07733) where @bgoodri 's suggestion is mentioned. I’m a little confused how the generated-quantities-transform helps things in the single-latent-trait model relative to having one loading be explicitly positive constrained. Say there’s two loadings and the data are very consistent with values of 1 for both; with the generated-quantities-transform approach, the sampler is still only “seeing” the bimodal version of the parameters, so won’t that risk exploration difficulties? With an explicit positive constraint on one, the sampler will then only see a single mode.
---
### Author:  [@saudiwin](https://discourse.mc-stan.org/u/saudiwin)
#### Post date: [January 20, 2021, 4:39pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/10 "2021-01-20T16:39:26Z")
A better way to do this is to fix a parameter to +1 instead of simply constrain it to be positive.
---
### Author:  [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)
#### Post date: [January 20, 2021, 4:45pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/11 "2021-01-20T16:45:09Z")
Thanks, but I’m more trying to understand the suggestion by @bgoodri, which is what blavaan is using internally so presumably it’s an idea that has some consensus as making sense and I’m just not following it.
---
### Author:  [@saudiwin](https://discourse.mc-stan.org/u/saudiwin)
#### Post date: [January 20, 2021, 4:53pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/12 "2021-01-20T16:53:35Z")
What they are saying is that having no constrained parameters is better because your sampling efficiency improves. I agree, though it’s not usually that big of a performance difference as you’re only constraining a single parameter.
---
### Author:  [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)
#### Post date: [January 20, 2021, 5:09pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/13 "2021-01-20T17:09:08Z")
> [@saudiwin](#):
>
> sampling efficiency improves
Yeah, I get that that’s the claim, I just don’t understand precisely how; indeed, given the unconstrained approach leads to a multimodal posterior in the eyes of the sampler, I would have expected less efficient exploration. But maybe I’m under-appreciating the efficiency of HMC and it has no issues with multimodality?
---
### Author:  [@saudiwin](https://discourse.mc-stan.org/u/saudiwin)
#### Post date: [January 20, 2021, 5:22pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/14 "2021-01-20T17:22:57Z")
HMC isn’t aware of other modes if they are too far away from each other. So it will select a mode randomly based on initialization values and whatever random trajectories take it to the closest mode.
---
### Author:  [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)
#### Post date: [January 20, 2021, 5:24pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/15 "2021-01-20T17:24:28Z")
Yeah, that’s my intuition as well, hence my confusion as to why they explicitly assert that imposing the positive constraint in GQ is _better_ than doing so in the model proper.
---
### Author:  [@saudiwin](https://discourse.mc-stan.org/u/saudiwin)
#### Post date: [January 20, 2021, 5:42pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/16 "2021-01-20T17:42:21Z")
There’s a minor sampling improvement without a constraint. Personally I think the need to do a transformation in the GQ actually makes things more complicated, especially if you have hierarchical parameters.
---
### Author:  [@Joshua\_Pritikin](https://discourse.mc-stan.org/u/Joshua_Pritikin)
#### Post date: [January 20, 2021, 7:05pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/17 "2021-01-20T19:05:26Z")
> [@saudiwin](#):
>
> Personally I think the need to do a transformation in the GQ actually makes things more complicated
My experience is that any improvement is sampling efficiency is worth the additional programming complexity.
---
### Author:  [@Stephen\_Martin](https://discourse.mc-stan.org/u/Stephen_Martin)
#### Post date: [January 21, 2021, 1:30am UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/18 "2021-01-21T01:30:06Z")
Personally, I just find it easier, and better, to ensure all indicators are in the same direction, then just impose positivity constraints for all non-crossloadings. This can get tricky for multilevel models (because then you either need a multiplicative random effect, or to do some fixed effect constraint to ensure all group-specific loadings remain positive after adding the RE… ).
The next best would be to post-process via the GQ method (flipping loadings accordingly).
The worst, imo, is to constraint _one_ loading to a sign. What happens, is that there is a _target_ mode where that sign is positive, and others are in the correct direction; but there is also a set of local modes where that loading is effectively zero, and the others are directionally unidentified. So you either get oscillation between (at least) three solutions, or you get some bad between-chain convergence. This is made worse if the fixed-sign loading is not particularly strong, because it can be set to zero without much difference in likelihood.
I’ve tried all three; I still prefer the first approach, though it can be messier with cross-loadings. Second approach is fine, but is irritating to implement especially with multilevel structure. Third is a no-go in my experience - It can and will fail if the fixed loading is too weak, or if there are a lot of indicators.
---
### Author:  [@Joshua\_Pritikin](https://discourse.mc-stan.org/u/Joshua_Pritikin)
#### Post date: [January 21, 2021, 1:59pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/19 "2021-01-21T13:59:07Z")
> [@Stephen\_Martin](#):
>
> Personally, I just find it easier, and better, to ensure all indicators are in the same direction, then just impose positivity constraints for all non-crossloadings.
Sure, but it’s not always feasible to make that assumption. Sometimes you don’t want to assume the sign of the loading.
---
### Author:  [@Stephen\_Martin](https://discourse.mc-stan.org/u/Stephen_Martin)
#### Post date: [January 21, 2021, 5:36pm UTC](https://discourse.mc-stan.org/t/latent-factor-loadings/1483/20 "2021-01-21T17:36:59Z")
That seems like a rare case, personally. If I a-priori know the sign of one loading, and I know what direction I want the latent variable in, then I generally know the rest too [except in the case of cross-loadings].
[Next page](https://discourse.mc-stan.org/t/latent-factor-loadings/1483.md?page=2)