There is nothing about unrooted trees that precludes computing an expected variance-covariance matrix from them. I think it’s more likely that there’s a numerical issue that is causing the numerical representation of the variance-covariance matrix to not be positive definite, or alternatively your matrix is merely positive semidefinite while brms requires a VCV matrix that is strictly positive definite. These issues can often be solved by adding a tiny positive term to each diagonal element, something like new_matrix = old_matrix + o * I where I is an appropriately sized identity matrix and o is a scalar that is chosen to be as small as possible while still yielding a positive-definite output.
If you can’t fix the issue with a small value of o (like less than 1e-5 or 1e-6) then it might be worth checking back in with a reproducible example.
See also here