Hmm this data set is really hard to tackle… Did you also fit the mgcv model with the same penalization?
We can try to look at a reproducible example, but unless someone else jumps in with new ideas I am not sure we are getting closer to the reason. Perhaps, at this point, it is worth asking Simon Wood about this issue. If someone knows, then he does.
Hey, no, I did not fit the GAM with the same penalty. I thought your update was to make the penalty default to the same setting in brms as in gamm4. Either way, I get different smooths than in the model with the small subsample excluded. I’ll try for a reproducible example, it takes too long to iterate with my real data.