Reducing SD on Theta in count IRT

Modeling
1

Hello,

I’m fitting the below IRT model designed for count data. My data is very highly dispersed across all items (mean = ~100, variance = ~100,000) and I currently have 6 items.


stancode <- '
data {
  int<lower=1> J; // number of people
  int<lower=1> K; // number of items
  int<lower=1> N; // number of observations
  int<lower=1,upper=J> jj[N]; // user for observation n
  int<lower=1,upper=K> kk[N]; // item for observation n
  real X[N]; // average attempts for person J on log scale
  int<lower=0> y[N]; // count of successful attempts for observation n
  
}

parameters {
  vector[K] delta;
  vector[K] alpha; // discrimination parameter
  real<lower=0> sigma_alpha;
  vector[J] theta;
  real gamma; // person covariate coefficient
  real<lower=0> sigma_gamma;
  real<lower=0> phi; // dispersion parameter.
}

model {
  delta ~ cauchy(0, 2); // item intercept, cauchy for weakly informed prior
  alpha ~ cauchy(0, 2); // discrimination/slope parameter
  sigma_alpha ~ normal(0,1);
  theta ~ normal(0, 1); // ideology/ability
  gamma ~ cauchy(0, 2); // person covariate coefficient
  sigma_gamma ~ normal(0,1);  
  phi ~ normal(0,1); // dispersion parameter
  
  for (n in 1:N)
    y[n] ~ neg_binomial(exp(delta[kk[n]] + alpha[kk[n]] * (theta[jj[n]] + gamma * X[n])), phi);
}
'

The model is working as expected and chains are converging nicely, however the standard deviation on theta is very high for most estimates with counts fewer than ~1,000 which constitutes roughly 80% of my data. I want to get more precise estimates.

I assume the best way to do this is with either more or more informative items. I’m currently working on this but was wondering if there are other approaches I could use as well.

I’m currently running 4 chains with 20k iterations, 5k warmup. The ESS for most of my theta > 100,000 with occasional odd observations < 10,000. The ESS on alpha and beta are all around 5-8,000. Will running more chains and more iterations get me more precise estimates, or are there diminishing returns? Are there any other approaches that might be helpful?

Thanks!

2

I am doubtful that more chains/iterations will help you. Based on those ESS values, you probably could have gotten by with fewer iterations. I think your intuition about more items/information is on the right track.

I am not sure it helps, but I wondered about the possibility of a binomial model here. The code comments seem to say that y is number of successful attempts, and X is possibly related to total number of attempts. If X were total number of attempts, you could set up a model like

y ~ Binomial(X, p)

and then put your IRT model on logit(p), which would be closer to traditional IRT.

3

You are correct that X is the total number of attempts. I think a Binomial model is an excellent idea and worth a try. I’ll fit a model and see how it goes.

Thank you, very insightful!

1 Like