We have been trying out IRT modelling as perfectly demonstrated by @paul.buerkner, but are having some issues with interpreting the estimates resulting from the fitted model. In particular, we are finding a discrepancy between estimates resulting from hypothesis() and from the conditional effects plots, which we don’t understand.
We are not sure whether these discrepancies come from a bug or whether there is a conceptual misunderstanding on our part.
Hopefully someone can help with this.
This is the model:
formula <- bf( Acc ~ 0.5 + 0.5 * inv_logit(eta), eta ~ 1 + pairType * (MDT + MPT) + (1 | item) + (1 | participant), nl = TRUE ) family <- brmsfamily("bernoulli", link = "identity") priors <- prior("normal(0, 5)", class = "b", nlpar = "eta") + prior("constant(1)", class = "sd", group = "participant", nlpar = "eta") + prior("normal(0, 3)", class = "sd", group = "item", nlpar = "eta") fit <- brm(formula = formula, data = data, family = family, prior = priors, iter = 10000, warmup = 1000, cores = 4, control = list(adapt_delta = 0.99, max_treedepth =12))
Note that we removed the discrimination parameter, as all items are very similar and adding this parameter did not improve the model. Pairtype is a factor consisting of three levels, MPT and MDT are two continuous variables, both scaled with possible values ranging from -4 to 4 and centred at zero. The response variable is Acc (accuracy with possible values of 0 and 1).
The problem occurs when we want to calculate evidence ratios or posterior probabilities and compare these with visualisation of the effects from conditional effects.
For example, if we want to check the posterior probability of the Intercept (which is pairtype 1 with MPT and MDT centred at zero), we would run this hypothesis:
hypothesis(fit, “eta_Intercept = 0”)
The estimate we get is -2.45, which we then run through the inv_logit transform in order to be able to compare it with the estimates in the conditional effects plots.
So that will be:
(inv_logit(-2.45) * 0.5) + 0.5 = 0.5397193
If we then check the estimate used to create the Intercept in the conditional effects plots, this value is 0.544.
If we repeat the same procedure with the other pair types, the values are also off, although each time by different amounts. To our understanding, these estimates should be the same.
Does anyone know where these discrepancies come from?