Identifiability problems with mixed effects models

Hi,

I’m attempting to build a nested random effects model in Stan. (Note: This model works fine in lme4, produces recovers parameters nicely, etc.)

The general structure - We are looking at student test scores. The random effects:

  • 4 grade levels (freshman, sophomore, junior, senior)
  • 100 classes (actual “class” in school)
  • 20 teachers
  • 2 past educations levels (public/private)
  • 1000 students

The fixed effect:

  • Hours of study (continuous)

The concept is that students are nested in classes, which are then nested in grade-level. The teacher and past education are random interactions that cross all the other groups.

In lme4: test_score ~ hours_study + (class|grade) + (1|teacher) + (1|pub_priv)

In Stan, no matter how I code this, I run into identifiability problems. conceptually the teacher and pub_priv variables should have small effects. However, Stan may assign the highest values to those, then small values to hours_study (or other similar problems)

I’ve read on past posts here, and in the Satan manual how identifiability can be an issue with multiple random effects. (Add a value one effect, subtract same value from second effect, and you get the same likelihood) .

How have others here approached this? Any suggestions or best practices?

Thanks!

If you want help on a Stan program, it helps us to see the program.

First, you could try rstanarm, which lets you use exactly the lme4 syntax to fit using a Bayesian model in Stan. Models that work in lme4 should work in rstanarm.

To get a max marginal likelihood fit like in lme4, you need an identifiability one way or another. To get a Bayesian posterior, you also need identifiability. With Bayes, you get a choice to do it softly with parameters or in a constrained way like lme4 must be doing in some way.

If you have redundant intercepts, you need to put priors on them to weakly identify. You have a choice with things like 1 | teacher to include an intercept (aka “random effect”) for all of the teachers or all but one of them and think of the other values as relative to the fixed teacher.

Next, you should make sure you’re not getting the right answer and just don’t realize it. You want to evaluate a Bayesian model by coverage, not by its mean estimates, which may be very uncertain.

It’s long, but it’s not that bad!

1 Like

Ha, oops. Freudian slip?