I think the default behaviour of brms::residuals(), as you’ve called on ‘mod1’, will return an array containing the posterior mean residuals, posterior standard deviation, and credible interval, unlike lme4::residuals() for example which returns a vector. Probably you only want to plot the first column. This might explain some of your concerns in the residuals plots.
Is the response variable reasonably continuous? If so and there are no zero observations, then I’d say a log-transformed response, or a gamma response distribution as you’ve tried, should be good starting choices for it. The rest I think depends on what you’re trying to get out of the model. If you’re primarily after the coefficients, I’d probably go with just the subject-level intercept term first, and work from there.