Thanks for the input, strengejacke. I tried the package you’re referring to. And indeed, its author defines ICC identically to what I’ve understood it to mean: “the proportion of the variance explained by the grouping structure in the population”. But the output of the icc()
function is cryptic to me:
icc(mod, ppd = TRUE)
# Random Effect Variances and ICC
Family: categorical (logit)
Conditioned on: all random effects
## Variance Ratio (comparable to ICC)
Ratio: 0.01 HDI 89%: [-0.10 0.13]
## Variances of Posterior Predicted Distribution
Conditioned on fixed effects: 1.10 HDI 89%: [0.97 1.22]
Conditioned on rand. effects: 1.11 HDI 89%: [1.07 1.14]
## Difference in Variances
Difference: 0.01 HDI 89%: [-0.11 0.14]
A ratio comparable to ICC is 0.01? That would mean next to no ICC, entailing that the random effect has little explanatory power. But this is not true – even just comparing the Deviances of the two models yields a difference of over 600 points – on three degrees of freedom! Also, the random-effects model predicts 62% of the quaternary responses correctly, compared to 57% for the fixed-effects model. It doesn’t seem an inconsequential group-level effect to me.
Also, running icc(mod, ppd = FALSE)
yields:
# Random Effect Variances and ICC
Family: categorical (logit)
## trigger
ICC: 0.50 HDI 89%: [0.36 0.65]
Between-group: 1.07 HDI 89%: [0.46 1.66]
## Residuals
Within-group: 1.00 HDI 89%: [1.00 1.00]
This says ICC is 0.5, i.e. the random effect accounts for HALF the explained variance. This, on the other hand, sounds very extreme – none of the three component logistic regressions constituting this multinomial model seems to have an ICC that high – calculating them individually using Hosmer et al’s formula (see first post) yields .24, .25, and .47, respectively. Their average (weighted by sample size) yields 0.3.
Which of the two values seen above is the estimated ICC? The 0.01 or the 0.50 – or neither?
Hmm. If I had to guess, I’d guess it’s the second one, calculated with ppd = FALSE
. That’s where the output correctly names the random effect (‘trigger’) whereas the output of ppd = TRUE
just uses the generic phrase “all random effects”. But still – 50 percent?