I am working with some spatial location data from a localization experiment where sounds are presented from discrete sources (loudspeakers) at fixed combinations of horizontal and vertical coordinates.

Currently, I am estimating population-level effects (betas) and varying effects (correlations) for each discrete location of which there are 40.

I’m wondering if there is anything I can do that would represent/estimate continuous effects. For example, if I have target horizontal locations at -90, -60, -30, 0, 30, 60, 90 and I expect errors to be larger as the location moves away from zero, can I estimate a “Gaussian-like” effect of target on error?

the monotonic effects in brms do this for ordinal, increasing, categorical predictors. Would one monotonic effect of -90, -60, -30, 0 and another for 30, 60, 90 be a valid approach?

There are also ways to force splines to be monotonic, but I don’t think I’ve seen anyone do 2D and monotonic (and in the particular way you need). It seems like it should be possible to build a spline basis that does what you need, but it will likely be a tiny research project on its own.

Yes, that sounds like a reasonable way to get something without much hassle. Note that mo effects are 0 for the reference category, so you can setup that when the actual value is positive the predictor for the negative side is at its reference category and this should work.