Low effective sample size for random effects only

Modeling
1

I’m using brms to model repeated measures data with random slopes and intercepts across participants. When I originally ran the model, it converged fairly quickly (< 1 min running parallel) but I got warnings about divergent transitions and low ESS.

I increased adapt_delta and number of iterations. Now the model still runs quickly (~ 1 min) & I get no divergent transitions. However, I still get the warning about low ESS. When I inspect the model summary, only the random effects display low ESS, all of the fixed effects are fine with large ESS.

My question is: if I’m only interested in accurately estimating the fixed effects, and don’t really care about accurate estimation of the variance of random effects (or rather, I’m not doing inference on the random effects), should the low ESS on the random effects bother me? I vaguely recall reading something about this being fine in Kruschke’s book, but I can’t find it now.

2

Yes, they should still bother you since it’s an indication that the chains have, perhaps, not converged. That could still affect all params.

  • What happens if you run the chain for a longer time, i.e., iter=10000?
  • You write that you have repeated measures, how many times do they repeat per “subject” and is it the same for all subjects?
3 Likes
3

Specifically ESS should grow linearly with the number of iterations. You can check ESS with different number of iterations to see whether this holds. See some examples in [1903.08008] Rank-normalization, folding, and localization: An improved $\widehat{R}$ for assessing convergence of MCMC and Rank-normalization, folding, and localization: An improved \widehat{R} for assessing convergence of MCMC (online appendix)

3 Likes
4

Thank you both for replying!

When I run the model with 10,000 iterations, I get warnings about divergent transitions. I don’t get warnings about low ESS anymore, however, when I check the summary, the ESS for some of the random effects are still very low (< 1,000).

I have four measurements per subject, from a 2 x 2 design. All subjects have 4 responses. Is that the problem? That each of the fixed effects has only two levels? Sorry if it’s something obvious that I’m missing.

5

Wait. So for each unique condition in the experiment you only have 1 observation per subject? You can’t fit a hierarchical model to that kind of data (well, you can, but the measurement noise term will be completely determined by the prior). I don’t know how to set up non-hierarchical repeated measures in brms, but for plain old Stan see my lectures and code.

(Edit: and just to clarify, I guess I’m making the assumption that you asked brms to model the data hierarchically from your terminology; usually when folks talk about random intercepts and slopes, they’re talking about a hierarchical model)

3 Likes
6

Yeah you’re right, I see, dumb mistake, thanks for helping me out.

2 Likes
7

Not dumb at all! With hierarchical models being all the rage now (for good reason), folks often aren’t taught about when they shouldn’t be used.

3 Likes
8

My ego would prefer to think I just didn’t think it through properly lol…I’ve read a few text books & played around with a bunch of longitudinal data, haven’t actually tried to model ANOVA-like data like this before though.

2 Likes