Multilevel IRT

fitting-issues

#1

I am interested in fitting multilevel rating scale IRT models with rater effects. I saw Bob Carpenter’s Youtube presentation at MLSS Sydney 2015 where he mentioned a multilevel IRT model with rater effects to show the efficiency of Stan relative to JAGS. I was wondering if the code for this series of models is available anywhere.

Also as suggested by the talk, these models should be challenging to fit. What procedures are used to address these challenges - i.e. guidance on implementing non-centered parameterizations in this context or possibly using Riemannian-manifold HMC.

As a side question, when is Riemannian-manifold HMC likely to be available in Stan. Is it available in a developer version?

Any comments would be appreciated!


#2

I believe the code for the 2015 IRT models is in the Stan example-models repo here:


#3

I found the example-models helpful. One paper which is also useful in going through the challenges facing IRT / ideal point fitting is this one:

Bafumi, Joseph, Andrew Gelman, David K. Park, and Noah Kaplan. “Practical Issues in Implementing and Understanding Bayesian Ideal Point Estimation.” Political Analysis 13, no. 2 (April 1, 2005): 171–87.


#4

Thanks for your reply. I will check out the examples and read the Bafumi paper.

Steve Schilling


#5

you should also check out the work done on the edstan package and
these two case studies on IRT models:

http://mc-stan.org/documentation/case-studies/hierarchical_2pl.html

http://mc-stan.org/documentation/case-studies/rsm_and_grsm.html

cheers,
Mitzi


#6

Hi Mitzi:

Thank you.

I did some more exploring and saw the edstan package. These case studies should be useful.

By the way, do you have any idea about when the Riemannian-manifold HMC algorithm might be available in Stan? From what I have seen from Michael Betancourt’s presentations and papers, this would solve many of the conditioning problems presented by multilevel IRT models.

Thanks again for your help.

Steve Schilling


#7

Principled priors and appropriate parameterizations will solve IRT issues far easier than resorting to something like RHMC.


#8

My next project is whipping higher-order autodiff into shape. When we’ve tested that more thoroughly and figured out how to deploy it without killing other build times, we’ll expose RHMC in the interfaces. Right now, @betanalpha’s (Michael Betancourt’s) softabs version is already in the C++ for Stan.


#9

HI Michael:

I certainly defer to your expertise as most of what I know concerning this topic comes from your excellent talks at the MLSS Iceland conference and and at Tokyo Stan.

In the MLSS talk at about 1:20 you show how RHMC provides nice trajectories that cover the entirety of the typical funnel set encountered in multilevel logistic regression.

I thought that since multilevel IRT models are simply multilevel nonlinear regression models, that this would have great applicability for more complicated multilevel IRT models with many levels of nesting - raters, occasions, classrooms, teachers, etc.

In such situations it might be difficult to determine appropriate transformations at multiple levels.

Steve


#10

I am also thinking not only for a specific application but for the future of fitting these models.


#11

Bob:

Thanks for the info.

Steve