Item characteristic curves and item information curves from item response models

Interfaces brms
1

I’m using brms for item response models. I’d like to create item characteristic curves (ICCs) and item information curves (IICs). How can I do that? Item characteristic curves plot the predicted score (y-axis) for each item as a function of trait level (x-axis). Item information curves plot the amount of information (y-axis) for each item as a function of trait level (x-axis). Information is the reciprocal of the standard error of estimation of trait level (Item response theory - Wikipedia). You can see examples of ICCs in section 2.2 and examples of IICs in section 2.3 here: Logistic IRT Models

Here’s a reproducible example of a 2PL IRT model with ordinal data from Paul Burkner’s white paper, that I’d like to create ICCs and IICs for:

data("VerbAgg", package = "lme4")

formula_va_ord_2pl <- bf(
  resp ~ 1 + (1 |i| item) + (1 | id),
  disc ~ 1 + (1 |i| item)
)

# some weakly informative priors
prior_va_ord_2pl <-
  prior("constant(1)", class = "sd", group = "id") +
  prior("normal(0, 3)", class = "sd", group = "item") +
  prior("normal(0, 1)", class = "sd", group = "item", dpar = "disc")

# fit the model
# this models throws some convergence warnings which are false
# positives and can be safely ignored
fit_va_ord_2pl <- brm(
  formula = formula_va_ord_2pl,
  data = VerbAgg,
  family = brmsfamily("cumulative", "logit"),
  prior = prior_va_ord_2pl,
  seed = 1234,
)
summary(fit_va_ord_2pl)

# extract item and person parameters
(ranef_va_ord_2pl <- ranef(fit_va_ord_2pl))

# plot person parameters
# item easinesses (deviations from thresholds)
eta <- ranef_va_ord_2pl$item[, , "Intercept"] %>%
  as_tibble() %>%
  rownames_to_column()

# discriminations
alpha <- ranef_va_ord_2pl$item[, , "disc_Intercept"] %>%
  exp() %>%
  as_tibble() %>%
  rownames_to_column()

# put easinesses and discriminations together
bind_rows(eta, alpha, .id = "nlpar") %>%
  rename(item = "rowname") %>%
  mutate(item = as.numeric(item)) %>%
  mutate(nlpar = factor(nlpar, labels = c("Easiness", "Discrimination"))) %>%
  ggplot(aes(item, Estimate, ymin = Q2.5, ymax = Q97.5)) +
  facet_wrap("nlpar", scales = "free_x") +
  geom_pointrange() +
  coord_flip() +
  labs(x = "Item Number")

Thanks in advance!

2

Just following up on this. Does anyone know how to create item characteristic curves and item information curves from item response models fit in brms? Thanks in advance!

In case it’s helpful, here are examples of ICCs and IICs:

ICCs
ICCs1344×960 17.2 KB

IICs
IICs1344×960 20.3 KB

3

I believe that the birtr package has a convience function for generating several IRT plots. From your initial post, it seems like you already have your estimates of the item parameters, so you can give those parameters to the birtr functions to generate the plots.

For illustration purposes, the icc function from the birtr package is the following:

icc <- function(b, a, c) {
  if (missing(c)) c <- 0
  if (missing(a)) a <- 1
  par(lab=c(7,3,3))
  theta <- seq(-3, 3, .1)
  P <- c + (1 - c) / (1 + exp(-a * (theta - b)))
  plot(theta, P, type="l", xlim=c(-3,3), ylim=c(0,1),
       xlab="Ability", ylab="Probability of Correct Response")
  thetai <- b
  pthetai <- c + (1 - c) / (1 + exp(-a * (thetai - b)))
  vliney <- seq(0, pthetai, .01)
  vlinex <- b + vliney * 0
  lines(vlinex, vliney, lty=2)
}

The plots are just functions of the estimated item parameters, so the key part is basically estimating the probability of a correct response over a range of theta values and plotting that function. Information curves are a little different since you need both the probability correct (P) and probability incorrect (1-P), but that’s not a hard computation. Again, the birtr package has an information plotting function, though it can be misleading since it’s labeled as a test information function (instead of listing a vector of the items’ parameters, just specify a single item’s parameters and the result is the item information curve)

—EDIT—
I just realized that you’re working with ordinal responses. I don’t think that the functions from birtr will work in that case. I believe that the same basic ideas apply to generating the ICC/IICs with polytomous data, but I’ve never personally had to create these plots from their parameters. You may try inspecting the plotting functions in the ltm or mirt packages to see how they compute the plots.

An alternative option would be to plot the empirical probabilities by response level. You can bin individuals’ latent traits and compute the proportion of persons with that ability level who endorsed the item at each level. The resulting proportions are the empirical probabilities, which in a well-fitting model ought to mirror the theoretical curves generated by standard plotting methods.

I believe, that in brms, you can get what you’re looking for via a conditional_effects(..., categorical = TRUE) plot. I’m not sure exactly, though, if that’s the case when you have no covariates in the model – I’ve never run this on my own before

4

Thanks very much for the response. I tried using conditional_effects(fit_va_ord_2pl, categorical = TRUE) with my minimal example above, and I received the following error:

Error: No valid effects detected.

In my actual data and model, I have predictors in the model and can generate conditional effects plots, but none of the plots have the latent level/ability on the x-axis; rather, they have levels of various predictors (e.g., age) on the x-axis.

The icc() formula you provided works well with traditional IRT models where the latent trait (theta) has a mean of zero and a standard deviation of 1. However, I’m using a different response distribution, and the easiness (not difficulty) parameters are not on the typical metric. The (absolute value of the) items’ easiness parameters are much larger than typical–the items’ easiness parameters (i.e., eta intercepts) are mostly greater than |10|, despite not being excessively easy or difficult. So, it’s not clear to me that this formula applies in this case.

It’s a good suggestion to bin ability levels to determine the empirical scores at each (binned) level of ability. I will plan to do that. It would also be nice to be able to generate the theoretical model-implied ICCs and item information curves, if possible. Any suggestions would be much appreciated.

5

Yes: the ICC function I linked to assumes a logistic model (e.g., 1PL, 2PL, etc), so it doesn’t work in the case of ordinal data. Still, the basic idea of using the model formula to generate the expected probability of endorsement should work. So long as we are willing to treat the posterior parameter estimates as “known”, the theoretical item characteristic curves still ought to be able to be estimated over a range of theta values by plugging in various theta estimates across some range of ability estimates and solving for the expected probability.

My understanding of this particular model is that the intention is to fit the graded response model, so the equation would be:

P(X_{ij} \ge k | \theta_j) = \frac{e^{a_i(\theta_j-b_{ik})}}{1 + e^{a_i(\theta_j-b_{ik})}},

so for every ordinal level k, you’d basically solve the equation using the posterior’s estimates of item discriminations (a_i) and locations (b_{ik}) at varying levels of theta. The function I sent before from the birtr package might actually be fairly easy to tweak since this model is just a polytomous generalization of the 2PL model, so most everything you’d need is already in that function. I think the only challenge is really adding in the indexing for which level of the ordinal response you’re in, passing a vector of the location parameters to correspond to those ordinal levels, and then figuring out how to layer in the expected probability trace lines for each response category onto the same plot. I do believe that you’d need to change the sign of your location parameters to make the plots consistent with the standard item “difficulty” rather than “easiness” parameterization. Alternatively, you could just warn the reader that the location parameter was computed differently than the “standard” IRT parameterization.

That said, I’m not entirely clear as to why the cumulative likelihood is being used here. I can’t say that I’m very familiar with the Verbal Aggression data, but I don’t recall the responses being anything other than dichotomous. Is this just for the sake of a reproducible example or am I mistaken about the data?

6

Good question–the example I gave was just for the sake of a reproducible example. My actual data are continuous proportion data from 0-1. We are using zero-one-inflated beta as the response distribution. We are estimating an easiness and discrimination parameter, in addition to precision (phi), zero-or-one inflation probability (zoi), and conditional-one probability (coi). So, the model most closely aligns with a 2-parameter IRT model (but with a different response distribution). Should I use your formula above, while changing the sign of the easiness parameters to convert them to difficulty? Thanks again.

7

The formula for the graded response model won’t work in your case. I’m not aware of any IRT models that use the ZOIB distribution. Theoretically, the same logic applies to your case, though: use the models’ item parameters to produce estimated probabilities given the model’s formula over a range of thetas. There’s an extra wrench in the ease of doing this since you have to account for the various components of the model, which is not something IRT models generally have to worry about since there isn’t that zero-or-one inflation component in most models.

I suppose that there’s really nothing stopping an IRT model from having alternative response distributions than the typical culprits (e.g., logistic, cumulative, adjacent category, etc), but I just haven’t seen the ZOIB used before. My suggestion would be to find the formula in whatever literature exists for your model and then use that as your function for generating the ICCs, if you really need to report those

8

Hey y’all, I just uploaded a blog post on this topic: https://twitter.com/SolomonKurz/status/1409951540228628482. I hope it helps.

3 Likes
9

This is great @Solomon! I don’t know why it never occurred to me to streamline things with the inv_logit_scaled() function from brms. I’m unfortunately not very tidyverse saavy, so I’m going to need more time to parse out what all the calls are doing. Regardless, bookmarking the blogpost for future reference!

1 Like
10

@Solomon Thanks so much for this very helpful post! I greatly appreciate it and look forward to reviewing it closely. It looks like the approach you took applies to the 1/2-parameter logistic model. I’m using a different response distribution (zero-one inflated beta). Do you know how to adapt your approach to generate ICCs and IICs with the zero-one inflated beta distribution (or other response distributions, more generally)?

We are estimating an easiness and discrimination parameter, in addition to precision (phi), zero-or-one inflation probability (zoi), and conditional-one probability (coi). So, the model most closely aligns with a 2-parameter IRT model (but with a different response distribution). Either way, thank you so much for this incredibly helpful tutorial!

11

Unfortunately, I’m not fluent enough with the ZOIB approach to IRT to give you a good answer. You or someone else will have to play around with the underlying principles in my post and find the right way to apply them to the ZOIB model. But if you do figure it out, this would probably be a great place to post the answer. I’d love to see it.