Hello everyone,
I am currently fitting a generalized partial credit model using brms, and I think there is some mismatch in scale with my prior and the disc
parameter estimated from data.
Here is my formula:
gpcm_formula <- bf(
R2 | thres(nthres, gr = Item) ~ 1 + ( 1 | Person), # Thresholds and latent trait
disc ~ 0 + Item # Varying discrimination
)
I am imposing different prior distributions for every item because I have discrimination parameter estimates that I obtained from my previous prediction. The correlation between the true (or banked/calibrated values) and my previous prediction was 0.3527. Although the correlation is not superduper high, I thought this would be a good starting point.
Below is how I specified prior means the discrimination parameter from each item:
# Set priors for discrimination parameters
for (item in unique(prior_disc_list2$newID)) {
tmp <- prior_disc_list2[which(prior_disc_list2$newID == item), ]
rows <- prior_list %>% filter(dpar == "disc" & coef == paste0("Item", tmp$newID))
if (nrow(tmp) != nrow(rows)) {
stop(paste("Error: Mismatch in discrimination rows for item:", item))
}
my_priors[[index]] <- set_prior(
paste0("normal(", log(tmp$predictions2), ",", sdv_disc, ")"),
class = rows[, "class"],
coef = rows[, "coef"],
dpar = rows[, "dpar"]
)
index <- index + 1
}
The tmp$predictions2
above is the vector that contains the predicted values from my previous prediction. You can see that I put log()
because disc
parameter is internally modeled on the log scale—so I put the log-transformed predictions as the prior mean. The prior standard deviation sdv_disc
is my experimental variable —which sdv_disc
yields posterior estimates that are closer to banked values.
The weird thing that I observed is that it seems that there is mismatch in scale from the estimates from data and my prior means.
So here you can see the plots where I plotted “true vs posterior” and “prior vs posterior”. As you can see I plotted both N=200 and N=500, where N=200 means that I used 200 responses for each item and N=500 for 500 responses. As one may see, as I put more responses, the posterior became further from the expected range for the discrimination parameter.
The above made me think that there is some mismatch going on between my prior and the estimates from the data. I think I was correct in using log()
in specifying the prior mean because the b_disc parameter (the disc parameter on the log-scale) is internally exponentiated in Stan. Since the true value is really the calibrated values from the data, the estimates should get closer as I put more data into the model, but opposite is happening…
Would my perception on this issue be correct? From my experimentation, it seems clear to me that there is some mismatch going on between data and prior, but I am not sure how to address this issue. Should I modify my formula to be more sophisticated to solve this issue?
FYI, difficulty parameter estimates behave as expected — scales match between prior and posterior and more data resulted in better results…
Any insights will be greatly helpful… Thank you!!!
@martinmodrak Would you have any insights on this?