Hi all,
I am trying to fit a 2-parameter IRT model to estimate the ideological positioning of elected officials using survey responses from some non-elected political elites. The non-elected political elites indicated the perceived ideological positions of the elected officials on a continuous 1-5 scale (1 as conservative, 5 as liberal; the survey used a slider so values are truly continuous).
I’ve been following @paul.buerkner’s helpful forthcoming JSS on implementing IRTs using brms, and I think I’ve gotten most of the way there but am stuck on some model fit issues. Even after tweaking some estimation options, I am getting the following warnings post-estimation:
1: The largest R-hat is NA, indicating chains have not mixed.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#r-hat
2: Bulk Effective Samples Size (ESS) is too low, indicating posterior means and medians may be unreliable.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#bulk-ess
3: Tail Effective Samples Size (ESS) is too low, indicating posterior variances and tail quantiles may be unreliable.
Running the chains for more iterations may help. See
http://mc-stan.org/misc/warnings.html#tail-ess
For warning 1, the only parameter with an NA R-hat is the standard deviation for the rater intercept in the eta term, which I fix as a constant for identifiability (as per @paul.buerkner). This might be a silly question, but is this R-hat coming back as NA because the parameter is fixed as 1? If so, can I safely ignore this warning?
For warnings 2 and 3, some parameters have stubbornly low bulk and tail ESS despite increasing the number of iterations. How should I adjust the model to try to address this issue?
The code I am currently using is as follows:
model_formula <- bf(rating ~ exp(logalpha)*eta,
eta ~ 1 + (1|d|rater) + (1|elected_official),
logalpha ~ 1 + (1|d|rater),
nl = TRUE)
model_priors <- prior("normal(0, 3)", class = "b", nlpar = "eta") +
prior("normal(0, 1)", class = "b", nlpar = "logalpha") +
prior("constant(1)", class = "sd", group = "elected_official", nlpar = "eta") +
prior("normal(0, 3)", class = "sd", group = "rater", nlpar = "eta") +
prior("normal(0, 1)", class = "sd", group = "rater", nlpar = "logalpha")
rating_model <- brm(formula = model_formula,
data = data,
family = brmsfamily(family = "gaussian"),
prior = model_priors,
control=list(adapt_delta=0.99,
max_treedepth = 20),
refresh = 10,
iter = 10000, warmup = 8000,
chains = 4, cores = 4, seed = 1989)
summary(rating_model)
- Operating System: Windows 10 Pro
- brms Version: 2.14.5