Scale mismatch in the 'disc' Parameter between prior and data for Ordinal IRT Models in brms

Modeling
,
1

Hello everyone,

I am currently fitting a generalized partial credit model using brms, and I think there is some mismatch in scale with my prior and the disc parameter estimated from data.

Here is my formula:

gpcm_formula <- bf(
  R2 | thres(nthres, gr = Item) ~ 1 + ( 1 | Person),  # Thresholds and latent trait
  disc ~ 0 + Item                    # Varying discrimination    
)

I am imposing different prior distributions for every item because I have discrimination parameter estimates that I obtained from my previous prediction. The correlation between the true (or banked/calibrated values) and my previous prediction was 0.3527. Although the correlation is not superduper high, I thought this would be a good starting point.

Below is how I specified prior means the discrimination parameter from each item:

# Set priors for discrimination parameters
for (item in unique(prior_disc_list2$newID)) {
  tmp <- prior_disc_list2[which(prior_disc_list2$newID == item), ]
  rows <- prior_list %>% filter(dpar == "disc" & coef == paste0("Item", tmp$newID))
  if (nrow(tmp) != nrow(rows)) {
    stop(paste("Error: Mismatch in discrimination rows for item:", item))
  }
  my_priors[[index]] <- set_prior(
    paste0("normal(", log(tmp$predictions2), ",", sdv_disc, ")"),
    class = rows[, "class"],
    coef = rows[, "coef"],
    dpar = rows[, "dpar"]
  )
  index <- index + 1
}

The tmp$predictions2 above is the vector that contains the predicted values from my previous prediction. You can see that I put log() because disc parameter is internally modeled on the log scale—so I put the log-transformed predictions as the prior mean. The prior standard deviation sdv_disc is my experimental variable —which sdv_disc yields posterior estimates that are closer to banked values.

The weird thing that I observed is that it seems that there is mismatch in scale from the estimates from data and my prior means.

image
image3232Ă—1700 357 KB

So here you can see the plots where I plotted “true vs posterior” and “prior vs posterior”. As you can see I plotted both N=200 and N=500, where N=200 means that I used 200 responses for each item and N=500 for 500 responses. As one may see, as I put more responses, the posterior became further from the expected range for the discrimination parameter.

The above made me think that there is some mismatch going on between my prior and the estimates from the data. I think I was correct in using log() in specifying the prior mean because the b_disc parameter (the disc parameter on the log-scale) is internally exponentiated in Stan. Since the true value is really the calibrated values from the data, the estimates should get closer as I put more data into the model, but opposite is happening…

image

Would my perception on this issue be correct? From my experimentation, it seems clear to me that there is some mismatch going on between data and prior, but I am not sure how to address this issue. Should I modify my formula to be more sophisticated to solve this issue?

FYI, difficulty parameter estimates behave as expected — scales match between prior and posterior and more data resulted in better results…

Any insights will be greatly helpful… Thank you!!!

@martinmodrak Would you have any insights on this?

2

“disc ~ 0 + Item” says the discrimination is modeled by items contrasted to a reference item. In particular, since disc is specified in log scale (you are right on that), the formula specifies multiplicative contrast between items. Is that consistent with the established prior?

3

Hi,

thank you so much for your reply!!

I used “disc ~ 0 + Item” formulization because I have specific prior for the discrimination parameter for each item. I did not know about the notion of discrimination modeled in contrast to a reference item; I just followed what was described in the Burkner (2020, JSS) to be able to specify different priors across items.

For the priors, I just put the log-transformed values of my priors (I have some ML predicted values for each of the item discrimination) as the prior mean.

I guess I am not following what you mean by “multiplicative contrast between items” obviously because my understanding of “disc ~ 0 + Item” formula is that item discriminations are estimated without considering correlation across items (i.e., no partial pooling).

You are saying that “disc ~ 0 + Item” assumes that there is a reference item? Then should my formula or prior specification change? For example, currently for identification, I constrained the standard deviation of theta to 1. But should I use a different constraint to be in line with that formula such as constrain one of the discrimination parameters to 0?

@paul.buerkner Should you have some insights on this problem, it will be really really appreciated!!

4

disc ~ 0 + item has not reference item (0 + implies cell mean coding). we would have a reference item only if we had disc ~ 1 + item or simply disc ~ item.

1 Like