I’d like to model a monotonic increasing effect in brms (dose-response model so don’t believe that it would be decreasing). Without any constraint the estimated effect is currently decreasing. I tried to specify a prior with a lower bound on for the parameter but got the error “Error: Prior argument ‘coef’ may not be specified when using boundaries.”

If your full model has no other coefficients, then drop the coef argument to your set_prior call. Otherwise, unfortunately last I knew this isn’t straightforwardly possible with brms. You could try updating to the latest development version just in case it’s changed recently, but here’s where things stand to my knowledge

However, it’s possible that you might be able to achieve this in a sneaky way using the nonlinear syntax of brms. I haven’t tested or verified this at all, but something like:

bf(
y ~ a + b,
a ~ other covariates,
b ~ 0 + mo(x),
nl = TRUE
)

with

prior1 <- prior(some prior on the other terms, nlpar = "a") +
prior("normal(0,5)", class="b", lb=0, nlpar = "b")