Ordered item response theory model with flipped scales

I have data on several respondents placement of parties on a left-right political scale, and the idea is to estimate the latent party placements by a model that looks like this (This is also called an Aldrich-McKelvey scaling model):

y* = \alpha_i + \beta_i \theta_j

where y* is the placement on the latent scale (let’s say I am assuming a latent normal distribution), \alpha_i is the individual respondent’s shift of the scale, \beta_i the respondent’s scaling of the scale and \theta_j the latent party placements. The model is not identifiable if \beta_i is not restricted by sign.

By following @paul.buerkner s paper on item response theory in brms I realized this is identical (at least in concept) to an ordinal item response theory model, so in brms I set up:

bf(value ~ 1+ (1|id|rowid)+ (1|party),
disc ~ 1+ (1|id|rowid))
with cumulative(link=“probit”) as family.

The problem is that some (albeit a few) respondents have answered as if the scale were flipped, put left-wing parties to the right and vice versa. In this parametrization those get very high scaling values, which of course is reasonable, since the variance/scaling would be seen as high when the scaling is restricted to positive. It could be interpreted as if the model regards them as very unprecise.

See this plot of correlation of each respondent’s placements’ correlation with the mean party placement of the latent scale versus the estimated mean scaling (I think I inversed the link correctly to make it interpretable as variance).

One alternative would be to formulate the model as an non-linear one (which was where I began before realizing the conceptual equivalence to ordinal IRT models), but I cannot figure out on how to add some kind of term that would model this scale flip.

This is how I formulated the model as a non-linear formula initially:

bf(value ~ 1 + exp(scalepar) * partylat + shiftpar,
shiftpar ~ 0+(1|id|rowid),
scalepar ~ 0+(1|id|rowid),
partylat ~ 0+(1|party),
nl = TRUE)

(exp(scalepar) is to make the model identifiable. This model also took ridiculously long time to estimate in comparison with the one above.)

The literature says that one fixed scaling parameter will make the model identifiable, but since brms does not support priors (also not constant priors) for individual random effects it would seem I am out of options. The pooling would also make this difficult to estimate properly I guess.

Do you have any input on how to incorporate the scale flip for some respondents?

1 Like

I don’t know about the models at hand but since this has gone unanswered I’ll give it a go. This sounds tough though.

Like, if you can easily identify the people that flipped the scales, then it seems like you could just flip them back?

But if you aren’t sure who interpreted the question correctly, that’d make your inference much much harder (you’d presumably need to model whether or not you though the person was interpreting the questions correctly).

Are you in the first or second situation?

1 Like

More the second situation, or rather, it is rather ambiguous how a respondent whose answer more weakly correlates negatively with the (properly defined) latent placements should be treated. What should the cutoff between a clear flip and noise (or a totally different view on the scale) be. It is a hard problem!

One way might be to model the parties as fixed effects and put some priors on the signs.

Oof, yeah, I’ve got no real clear idea of what to do there. I guess you could do some fake data stuff and try to figure out what the effect of a few people getting the scale backwards would be, but I don’t think there’s information here to do inferences on whether or not people flipped the scales.

Cause couldn’t they correctly interpret the scales and just have unusual beliefs or something?

1 Like

I have zero experience with anything like this, but one quick idea comes to mind (but would IMHO require moving to pure Stan). Add a parameter for each participant representing the probability that they flip the scales, let’s call it p_i. You would then marginalize over the two possibilities as

P(Y_i = d) = p_i(P(Y_i = d | y* = \alpha_i - \bar\beta_i\theta_j)) + (1 - p_i)(P(Y_i = d | y* = \alpha_i + \beta_i\theta_j)

(this would be implemented via log_mix in Stan). Also note that we have separate parameters for the flipped and not-flipped scenario (\beta, \bar\beta), possibly having separate \alpha and \bar\alpha could be needed (depending on whether flipping the scale should change the shift of not).

In it’s raw form this is identical to letting \beta_i change signs and would thus make your model not identifiable. However, you could put strong prior on p_i, say p_i \sim Beta(1, 100) to express that you consider flipping very unlikely and this could keep the model identified.

Just an idea. Not sure it would work (I would bet at least 2:1 that it would not work) and it would likely be computationally quite inefficient.

Also the intepretation of such model would be complicated, as I am with Ben that

is always a valid explanation.

1 Like

For whatever it’s worth, I’ve recently been working on a similar(ish) problem, and have done pretty much exactly what @martinmodrak suggested — except with a single parameter p_i \equiv p — and it has worked pretty well so far in recovering the true p in simulations.

Mine was a bit different because there was only one question out of four that we thought respondents could’ve mistakenly reversed, because it was the only one that was negatively worded. So we supposed that the remaining questions provide information about the individual’s ‘intended’ response, because all of the questions were geared towards the same idea (although @bbbales2’s alternative explanation could also hold here too, in that some individuals maybe just had a very different feeling about that particular question).

In your case, where you think every response from an individual might be the opposite of what they intended, I agree that a reasonably strong prior would be needed to make it identifiable. Try simulating data under your model and see how well you can recover the known p_i.

I believe what you need is an ideal point model. Check out my package idealstan. It includes ordinal outcomes.