I would like to ask an advice regarding whether I drew correct conclusions from a Bayesian mixed effects model.

I had an experiment where participants performed a task in two conditions. Overall, I saw no significant difference in their performance between the two conditions. However, I noticed that some of the participants performed a lot stronger in one condition, and others - in a different one. That made me wondering if this variability was above mere chance. I looked at the 95% credible interval for variance of random slopes for participants with respect to condition, and found that it doesnâ€™t overlap with zero. From that, I concluded that it was indeed above chance and there were both some â€ścondition-A-loversâ€ť and some â€ścondition-B-loversâ€ť. Further, I used sampling from the posterior distribution to extrapolate what fraction of the population prefers each condition (along with credible intervals) and saw that a sizable fraction of participants appears to prefer each condition.

I wanted to ask whether such a method is legitimate. Although it appears to make sense, I havenâ€™t seen it being used in publications. Typically, some analysis of interaction effects is used to arrive at similar conclusions. Unfortunately, I havenâ€™t measured the parameters which I suspect might differentiate â€ścondition-A-loversâ€ť and some â€ścondition-B-loversâ€ť.

The general terminology for what youâ€™d like to model is â€śmixture modellingâ€ť where your population of participants is a mixture of different categories and you donâ€™t have explicit labels for which category a given participant falls into a priori. You see this kind of model used in the growth mixture modelling literature, where they often are doing intervention studies and want to model the existence of participants that for whatever reason do and donâ€™t respond to the intervention.

The model youâ€™ve already run, a standard hierarchical/mixed effects model, can, as you have done, be inspected a posteriori for hints of mixture behavior, but the partial pooling implied by the model may lead to different inferences relative to running a full mixture model

Note that the 95% credible interval for the variance will never overlap zero because the variance is constrained to be >= 0. So to inspect the mixed effects model for hints that thereâ€™s meaningful variation here, youâ€™d want to check that your model is putting minimal posterior mass in a sufficiently wide neighborhood near zero, where â€śminimalâ€ť is measured relative to the prior mass in the neighborhood (unless youâ€™ve intentionally chosen a well-considered informative prior) and â€śsufficiently wideâ€ť is wide enough so that the estimation of the posterior mass in the neighborhood is reliable.