I wonder if multi-membership terms could be used to account for compositional similarities between different communities, assigning “membership” of a community to each of the species that comprise it. Individual species can have an idiosyncratic influence on community-level properties and I think mm terms may be used to account for this.
Yes, I think you can do this, but I think you’re assuming that there is no overyielding/non-additivity in the community. The model will decompose the effects of each species additively. I don’t think brms has support for more complicated interactions between multimembership terms (e.g., here).
My half-baked work-around idea is to measure the non-additivity with the tree species composition of the plots as an additional random intercept (... + (1 | composition)). Some care would need to be taken to ensure that plots with, say, species 7 10 7 10 is coded the same way as 10 7 10 7. If I’m thinking about it correctly, the multimembership term would allow for partial pooling of information across plots to measure the additive contribution that each species makes to the ecosystem function response. I think this may only work for experimental designs where each community composition is well-replicated.
Alternatively, random slopes for the multimembership group might be a way to explore mechanisms underlying the differences in species contributions. Here, a lot of care and domain knowledge would be important to ensure the causal map would allow for the inference.