Brms formula for partially nested random effects

I am trying to model random effects that are nested, but only for a subset of all observations. Essentially, I have observations from 5 species coming from 3 clades (see attached image). However, 2 of the clades only have one species in them. For the clade with multiple species, I want to partially pool at the clade and species level.

I would find this straightforward to implement directly Stan, but am a bit stumped on the brms syntax.


y ~ 1 + (1|genus) + (1|genus:species) # explicit nesting


y ~ 1 + (1|genus/species) # implicit nesting

These formulations will estimate parameters for Genus2:Species4 and Genus3:Species5, which are poorly identified given their redundancy with Genus2 and Genus3, and the high uncertainty in variance across Genus. I wanted to know if there’s a way to not estimate species random effects for Genus 2 and 3.

I think a practical solution would be to dummy-code “Genus 1 (0/1)” and then have random effects only for Species.