Thank you for your responses.
Following your suggestions, I found this: Hierarchical prior (for partial pooling) on correlation matrices? - #9 by bgoodri
I believe that this is the best way forward, even though it can be quite time consuming ( I am sampling a 5x5 covariance matrix per unit, and I have 10-200 units).