Although I’m far from a linear algebra buff, I was interested by your question and did some quick research online to see how one might generate an orthogonal matrix from simple elements. Wikipedia to the rescue:
“To generate an (n + 1) × (n + 1) orthogonal matrix, take an n × n one and a uniformly distributed unit vector of dimension n + 1. Construct a Householder reflection from the vector, then apply it to the smaller matrix (embedded in the larger size with a 1 at the bottom right corner)”
As you want to generate a 3x3 matrix, you could start to generate with generating a 2x2 based on the properties described here in the section “elementary construction”: https://en.m.wikipedia.org/wiki/Orthogonal_matrix
If you want to define priors on the elements of the orthogonal matrix, you’ll have to correct the posterior for the Jacobian of the inverse transform (which or magics words to me, so sorry can’t help there). The Stan reference manual or other people here may provide more advice on that.