After spending some time playing around with implementing a modification of the LKJ prior, I believe there are errors in the statement of the “manageable expression” for W provided in the stan manual at the following link:
The definition provided seems to assume that z has ones along the diagonal, instead of the zeros shown in the definition above. Otherwise, all diagonal entries other than the first row will be forced to 0.
Also, I believe the final line in the piecewise statement is missing a factor of 1/Z_{i-1, j}. After making these two changes, I think the manageable expression will now agree with the “long version” above it.
I think I even said I would get to the bottom of it in the first link (I definitely did not and then forgot about it, whoops). I am more confident in the formulae in 10.12: 10.12 Cholesky factors of correlation matrices | Stan Reference Manual, and would then take the crossproduct of the result to get a correlation matrix.
Since this error is apparently already known, I submitted a pull request making the changes that I suggested in my above post. I didn’t dig into how it is actually implemented in stan, I just want to prevent others from falling down the same hole I did.
This is actually kind of funny - I just encountered this on Friday (3 days ago). I was planning to submit a bug report today, and voila - it’s known already.