Priors for psychometric curve parameters

I’m trying to estimate a 3-parameter (bias, noise, lapse) psychometric curve with brms. The observer gives a response left or right depending on the signal strength x with the equation:

Pr(left|x) = \lambda/2+(1-\lambda)\Phi(\frac{x-\mu}{\sigma})

where different parameters might depend on a condition (for example, manipulation of the stimulus contrast). I wonder what would be the good priors to use for the uncertainty (\sigma) and lapses (\lambda)?

Here is my model assuming that only \sigma is affected by a condition (contrast manipulation):


model_formula <- bf(
  resp_left ~ lambda/2 + (1-lambda)*Phi(eta),
  lambda ~ 1,
  family = bernoulli(link="identity"),
  nl = T
      criterion ~ 1,
      sigma ~ 1+contrast)

fit_contrast <- brm(model_formula, data = data, prior = c(
  prior(normal(0, 3), class = "b", nlpar = "criterion"),
  prior(normal(0, 3), class = "b", lb = 0, nlpar = "sigma"),
  prior(beta(1, 1), nlpar = "lambda", lb = 0, ub = 0.5)
), control = list(adapt_delta = 0.95), cores = 4)

This works fine if there is a lot of data, but if there is not that many data points, the estimates in the baseline condition are lower compared to the other condition even if there is no effect.

Some random data:

data <- data.table(expand.grid(delay = seq(-2,2,0.2), contrast = c('high','low')))

mu <- -0.5 # response criterion is shifted from zero
lambda <- 0.05
n_trials <- 10

# assume that there is no effect
noise_high_contrast <- 0.3
noise_low_contrast <- 0.3

data[,noise:=ifelse(contrast == 'high', noise_high_contrast, noise_low_contrast)]
data <- data[,.(x = rnorm(n_trials, mean = delay, sd = noise)), by = .(delay, noise, contrast)]
data[,is_lapse:=runif(.N, 0, 1)<=lambda]
data[is_lapse==T, response := sample(c('left','right'), .N, replace = T)] # for trials with lapses, the response is chosen randomly
data[is_lapse==F, response := ifelse(x > mu,'left','right')] # for other trials, it depends on the measurements
data[,resp_left := as.numeric(response == 'left')]

If the model above is fitted on this data, the means for sigma are lower for the baseline condition:


 contrast emmean lower.HPD upper.HPD
 high      0.263     0.165     0.377
 low       0.350     0.232     0.479

The same happens for lambda. I think that the bias arises because of the way I set the priors but I don’t understand why and how to fix it.