There could be a lot of stuff going on - my current best advice is at Divergent transitions - a primer. I would definitely try running the no-intercept model with wider priors (at least by factor of sqrt(2)). It might also be useful to use different coding of the factor variable (e.g. as outlined at Contrast coding with brms).