I would like to ask an advice regarding whether I drew correct conclusions from a Bayesian mixed effects model.
I had an experiment where participants performed a task in two conditions. Overall, I saw no significant difference in their performance between the two conditions. However, I noticed that some of the participants performed a lot stronger in one condition, and others - in a different one. That made me wondering if this variability was above mere chance. I looked at the 95% credible interval for variance of random slopes for participants with respect to condition, and found that it doesn’t overlap with zero. From that, I concluded that it was indeed above chance and there were both some “condition-A-lovers” and some “condition-B-lovers”. Further, I used sampling from the posterior distribution to extrapolate what fraction of the population prefers each condition (along with credible intervals) and saw that a sizable fraction of participants appears to prefer each condition.
I wanted to ask whether such a method is legitimate. Although it appears to make sense, I haven’t seen it being used in publications. Typically, some analysis of interaction effects is used to arrive at similar conclusions. Unfortunately, I haven’t measured the parameters which I suspect might differentiate “condition-A-lovers” and some “condition-B-lovers”.
The general terminology for what you’d like to model is “mixture modelling” where your population of participants is a mixture of different categories and you don’t have explicit labels for which category a given participant falls into a priori. You see this kind of model used in the growth mixture modelling literature, where they often are doing intervention studies and want to model the existence of participants that for whatever reason do and don’t respond to the intervention.
The model you’ve already run, a standard hierarchical/mixed effects model, can, as you have done, be inspected a posteriori for hints of mixture behavior, but the partial pooling implied by the model may lead to different inferences relative to running a full mixture model
Note that the 95% credible interval for the variance will never overlap zero because the variance is constrained to be >= 0. So to inspect the mixed effects model for hints that there’s meaningful variation here, you’d want to check that your model is putting minimal posterior mass in a sufficiently wide neighborhood near zero, where “minimal” is measured relative to the prior mass in the neighborhood (unless you’ve intentionally chosen a well-considered informative prior) and “sufficiently wide” is wide enough so that the estimation of the posterior mass in the neighborhood is reliable.