I have fitted a lognormal model in brms with two binary factors b_1, b_2, and a random effect on the intercept (like A ~ b_1 + b_2 + (1|id)), where id is a participant id. Now I would like to assess/interpret the effects of the random effect of the participants. Intra-class correlation seems to me like an intuitive way to analyze the consistency among participants.
Using sjstats package (@strengejacke), I have computed the intra-class correlation coefficient (ICC) using the median (parameter typical), and got a result which is estimated to 0.26, 95%-HDI [-0.55, 0.76].
- Is this a reliable ICC estimate, i.e. can I report the estimate (and interpret it) and can I compare it to ICCs of other models?
- Is there a way to interpret the large HDI which is even spread to negative values?
- Operating System: Ubuntu 16.04 LTS
- brms Version: 2.6.0
Have you set the ppd-argument to TRUE? Except for Gaussian models, the “classical” ICC is not useful, and you should rather focus on a different way of variance decomposition (as stated here: Rstanarm: extracting variance components). This is what you get when ppd = TRUE, however, note that this “ICC” will probably differ notably from ICCs you get when you fit the same model in the frequentist framework.
Just a footnote: functions that are related to model quality or model performance from sjstats have been re-implemented in the performance-package (which is part of the new “easystats”-project).
2 Likes
Thank you for the prompt answer.
Yes, I have. I also found the implementation of the underlying ICC definition as ratio of the variances between all ppd and the ppd without considering the random effects. But I struggle to understand how reliable the estimate is, considering the large spread of ICC values.
Good to know about the re-implementation, will check the new package as well.
I’d like to ask again about my initial questions, how can I interpret the larger uncertainty/credibility in the 95%-HDI of my ICC? Is the estimate reliable to infer consistency (or rather no consistency) between participants? Do I need more data or a better model to arrive at a less widely spread distribution?
The ICC should not be negative. I would thus question the initial ICC results as the credible intervals goes into the negative territory. I would conclude that whatever ICC you are using there should not be used with lognormal models.
1 Like