I have a data-set with a highly-skewed response that has distinct ceiling and floor effects. So, I thought a possible way to account for these features could be to transform them to ranked categories and analyse them with an ordinal model.
I fitted two (identical) ordinal models to the same data. The first model used the cumulative family and the second used the cratio. Both models have the default logit link for the mean and the log link for the discrimination parameters.
The first goal of the models is to tested if “expert” raters (rtype “expert“) have (a) lower ratings and (b) better discrimination than untrained raters (rtype “anaive”). The second goal is to gain insight into the mental mechanisms by which raters decide on their ratings. That is, are they (i) swayed by the intensity of a single feature of the “stimulus”, or do they (ii) sum effects of several features? I thought it possible that cumulative family may represent (i) and the cratio family may represent (ii).
I first compared two models that did not include the ‘rtype’ factor and found that the cratio family fitted slightly better (Bayes factor 2.65). I then added the ‘rtype’ term and re-estimated the models.
When I examined the results of the two analyses, I found that the signs of the coefficients for only the crucial ‘rtype’ effect are reversed in the two models. Here are the the fixed effects of the two models:-
cratio cumulative
Estimate Q2.5 Q97.5 Estimate Q2.5 Q97.5
Intercept[1] -6.89 -14.93 -2.66 -9.82 -22.09 -3.67
Intercept[2] -5.32 -11.33 -2.04 -6.80 -14.97 -2.57
Intercept[3] -2.10 -4.58 -0.77 -2.73 -6.01 -0.98
Intercept[4] -1.20 -2.83 -0.36 -0.65 -1.78 -0.05
disc_Intercept -0.01 -0.89 0.87 -0.26 -1.18 0.65
age -0.14 -0.70 0.40 0.10 -0.30 0.60
sexM -0.21 -1.37 0.81 -0.53 -1.94 0.42
rtypeexpert 3.40 0.08 9.99 -0.83 -1.99 -0.03 <== opposite signs
disc_age 0.11 -0.14 0.33 0.00 -0.15 0.16
disc_sexM -0.18 -0.68 0.31 -0.25 -0.65 0.14
disc_rtypeexpert -1.27 -2.14 -0.50 -0.77 -1.26 -0.28 <== same effects
The raw means and medians of the raw responses for rtype “train” are lower:-
anaive train
17.30 14.68
$anaive
0% 25% 50% 75% 100%
3 15 20 20 20
$expert
0% 25% 50% 75% 100%
3 10 15 20 20
and the corresponding means and quantiles for the rank-transformed data are also smaller for rtype expert:
tapply(dat$resp5,dat$rtype,mean)
anaive expert
4.375 3.840
tapply(dat$resp5,dat$rtype,quantile)
$anaive
0% 25% 50% 75% 100%
1 4 5 5 5
$expert
0% 25% 50% 75% 100%
1 3 4 5 5
Also, in both models, the discrimination parameter is lower in experts.
The conditional effects of rtype look similar for the two models. It is clear that experts have lower discrimination with both the cratio and cumulative familes, but the probabilities of each category appear similar in both models
cratio:
cumulative:-
The cratio model fits slightly better than the cumulative model:-
Estimated Bayes factor in favor of model_crat over model_cumul: 2.18583
The vignette about the parameterisation of the cumulative and cratio families says:
Note that cumulative and sratio models use τ−η, whereas cratio and acat use η−τ. This is done to ensure that larger values of η increase the probability of higher response categories.
So, the results of the cumulative model seem to fit the idea that larger values of η increase the probability of higher response categories, but the results of the cratio model seem to contradict it.
I’d be very grateful for an explanation of what is happening and guidance concerning which model’s results I should believe