Hi @maurice_goodman!
This is not possible using Stan, because the number of knots ist a discrete parameter. Any HMC scheme relies on gradients of the parameters and discrete parameters don’t have gradients. If I’m not mistaken, brms is taking a default number of 10 knots (basis functions are built by the mgcv package). You can change this number.
If you’re using 'optimize' then I guess this would result in a maximum penalized likelihood estimator. But in this case you’d probably be better off using mgcv straight away. When you’re using MCMC, the prior acts as the penalty or smoothing parameter. (I think you may have not noticed it in the code, but it should be there.) I hope this also answers your second question.
There’s actually some discussion about GAMs on this board. For example, here.
Cheers! :)