Anecdotally, I’ve noticed much faster performance with GAMs in brms by specifying a smaller k. Unlike mgcv which “finds” a reasonable k, I don’t believe we expect brms to do the same. So it’s somewhat akin to passing a lot of linear predictors (in your case at least 40).
So one approach, though I can’t speak to it’s “principledness” might be to use mgcv to recommend a k and then use brms and LOO to compare models bracketing that k.
Not sure what your data are but 20 knots seems like a very tortuous relationship!