Bayesian modeling of ordinal (Likert) response data

[Ana_Clemente]

We have liking ratings on a 1–5 Likert scale of images, each in one of two conditions (asymmetrical vs. symmetrical) and belonging to one of six categories. We are interested in the effect of stimulus category on symmetry preference (whether liking more symmetrical or asymmetrical images). This is the head and structure of our dataset:

Here is our model configuration:

prior_m1 ←
#fixed
set_prior(“normal(0,1)”, class = “Intercept”) +
set_prior(“normal(0,1)”, class = “b”, coef = “symmetry1”) +
#error
set_prior(“gamma(1,2)”, class = “sd”)

m1 ← brm(liking ~ symmetry + (1|participant) + (symmetry|set),
data = Data1,
family = cumulative(link = “probit”, threshold = “flexible”),
iter=20000, warmup=2000,
prior=prior_m1,
chains=4, cores=4,
init=‘0’, control=list(adapt_delta=0.99, max_treedepth = 10),
seed = 111)

Then, we draw some data and run equivalence tests as follows:

post1 ← as_draws(m1)
post1 ← do.call(rbind.data.frame, post1)

hyp_Sym ← equivalence_test(post1$b_symmetry1, ci=(.89), range=c(-0.1sd(Data$liking), 0.1sd(Data$liking)))

Importantly, we are interested in the effects of this relationship per set because we want to compare them later, which is why our model considers them a grouping factor with independent intercepts and slopes. To build the different posterior distributions per set, we include the main effects beta “b_symmetry1” and the effects by set. We would normally also include the intercept in any other model. However, the intercept in a cumulative is dependent on the particular tau by level. Therefore, we think it should not be included.
The probit link determines the likelihood that an item or event will fall into one of a range of categories by estimating the probability that an observation with specific features will belong to a particular category. We are interested in all these estimations at the same time. Therefore, we think the intercepts are useless for our contrast of interest. Please correct us if we are wrong. Anyway, following this logic, we calculate the equivalence tests as follows:

hyp_A ← equivalence_test((post1$b_symmetry1 + post1$r_set.Artworks.Intercept. + post1$r_set.Artworks.symmetry1.), ci=(.89), range=c(-0.1sd(subset(Data$liking, Data$set==“Artworks”)), 0.1sd(subset(Data$liking, Data$set==“Artworks”))))
hyp_B ← equivalence_test((post1$b_symmetry1 + post1$r_set.Bertamini.Intercept. + post1$r_set.Bertamini.symmetry1.), ci=(.89), range=c(-0.1sd(subset(Data$liking, Data$set==“Bertamini”)), 0.1sd(subset(Data$liking, Data$set==“Bertamini”))))
hyp_D ← equivalence_test((post1$b_symmetry1 + post1$r_set.Designs.Intercept. + post1$r_set.Designs.symmetry1.), ci=(.89), range=c(-0.1sd(subset(Data$liking, Data$set==“Designs”)), 0.1sd(subset(Data$liking, Data$set==“Designs”))))
hyp_J ← equivalence_test((post1$b_symmetry1 + post1$r_set.Jacobsen.Intercept. + post1$r_set.Jacobsen.symmetry1.), ci=(.89), range=c(-0.1sd(subset(Data$liking, Data$set==“Jacobsen”)), 0.1sd(subset(Data$liking, Data$set==“Jacobsen”))))
hyp_P ← equivalence_test((post1$b_symmetry1 + post1$r_set.Pepperell.Intercept. + post1$r_set.Pepperell.symmetry1.), ci=(.89), range=c(-0.1sd(subset(Data$liking, Data$set==“Pepperell”)), 0.1sd(subset(Data$liking, Data$set==“Pepperell”))))
hyp_S ← equivalence_test((post1$b_symmetry1 + post1$r_set.Sasaki.Intercept. + post1$r_set.Sasaki.symmetry1.), ci=(.89), range=c(-0.1sd(subset(Data$liking, Data$set==“Sasaki”)), 0.1sd(subset(Data$liking, Data$set==“Sasaki”))))

Is this approach correct? How should we treat the main intercepts and the intercepts per set? Is there a better approach to address our question, such as: liking~ symmetryset + (symmetryset|participant)? And (last question, we promise) should we use a probit in this case or should we use another link function as it is difficult to mathematically estimate?

We greatly value your expertise and look forward to learning from your suggestions. Thank you in advance for your time and insights.

[AWoodward]

Hello @Ana_Clemente, here’s a few thoughts on your modelling:

• the cumulative link model does seem to be reasonable selection for this, in principle. The other major choice for the link function is the logit. I doubt that it will make much practical difference to the probit link, at least in terms of the overall interpretation.
• I’m confused by your comments about the use of the intercepts. The intercepts in this model correspond to the thresholds that map the latent variable to a conditional probability distribution of the category levels. So the model brm(liking ~ symmetry + (1|participant) + (symmetry|set) specifies a location shift of the latent variable by participant, or equivalently a location shift in all thresholds (by the same location and magnitude) for each participant. Note that by using this (symmetry|set) term to capture the effect of set you are assuming that the thresholds are the same by set and symmetry except for location shifts, or in other words, that sets may differ from each other in position on the latent scale, but not in how the latent variable maps to the observable category levels. I don’t have any specific knowledge about your application, but this strikes me as a potentially strong assumption. Have you done some goodness-of-fit evaluation?
• regarding the equivalence questions and hypotheses I think this depends strongly on how you want to conceptualize the question being asked. In this case I think that you’ve asked all these on the latent scale, which is quite direct but may lack clear interpretability (in particular, this doesn’t mean anything directly about equivalence in terms of the original Likert-type scale). In principle there are many possible ways that effects from this model could be summarized.
• in most applications there’s no need for so many samples (probably 2000-4000 is a reasonable starting point for this model).

[Ana_Clemente]

Hello @AWoodward!

Many thanks for your comments. They are very helpful to us and have made us reflect on several different assumptions, which is always very useful. However, our questions are:

1. Assuming that the model is correct and all the assumptions are correct, how should we build the posterior distributions for each set?
2. Is this equivalence test the most appropriate to investigate how each set modulates the relationship between symmetry and liking? post1$b_symmetry1 + post1$r_set.Artworks.Intercept. + post1$r_set.Artworks.symmetry1.

Again, thank you very much for all your comments and suggestions.