Ordinal regression: opposite signs of a coefficient with families cumulative and cratio

Modeling
1

I have a data-set with a highly-skewed response that has distinct ceiling and floor effects. So, I thought a possible way to account for these features could be to transform them to ranked categories and analyse them with an ordinal model.

I fitted two (identical) ordinal models to the same data. The first model used the cumulative family and the second used the cratio. Both models have the default logit link for the mean and the log link for the discrimination parameters.

The first goal of the models is to tested if “expert” raters (rtype “expert“) have (a) lower ratings and (b) better discrimination than untrained raters (rtype “anaive”). The second goal is to gain insight into the mental mechanisms by which raters decide on their ratings. That is, are they (i) swayed by the intensity of a single feature of the “stimulus”, or do they (ii) sum effects of several features? I thought it possible that cumulative family may represent (i) and the cratio family may represent (ii).

I first compared two models that did not include the ‘rtype’ factor and found that the cratio family fitted slightly better (Bayes factor 2.65). I then added the ‘rtype’ term and re-estimated the models.

When I examined the results of the two analyses, I found that the signs of the coefficients for only the crucial ‘rtype’ effect are reversed in the two models. Here are the the fixed effects of the two models:-

                         cratio             cumulative  

                Estimate   Q2.5 Q97.5 Estimate   Q2.5 Q97.5
Intercept[1]       -6.89 -14.93 -2.66    -9.82 -22.09 -3.67
Intercept[2]       -5.32 -11.33 -2.04    -6.80 -14.97 -2.57
Intercept[3]       -2.10  -4.58 -0.77    -2.73  -6.01 -0.98
Intercept[4]       -1.20  -2.83 -0.36    -0.65  -1.78 -0.05
disc_Intercept     -0.01  -0.89  0.87    -0.26  -1.18  0.65
age                -0.14  -0.70  0.40     0.10  -0.30  0.60 
sexM               -0.21  -1.37  0.81    -0.53  -1.94  0.42
rtypeexpert         3.40   0.08  9.99    -0.83  -1.99 -0.03 <== opposite signs
disc_age            0.11  -0.14  0.33     0.00  -0.15  0.16
disc_sexM          -0.18  -0.68  0.31    -0.25  -0.65  0.14
disc_rtypeexpert   -1.27  -2.14 -0.50    -0.77  -1.26 -0.28 <== same effects

The raw means and medians of the raw responses for rtype “train” are lower:-

anaive train 
 17.30 14.68 

$anaive
  0%  25%  50%  75% 100% 
   3   15   20   20   20 

$expert
  0%  25%  50%  75% 100% 
   3   10   15   20   20 
and the corresponding means and quantiles for the rank-transformed data are also smaller for rtype expert:

tapply(dat$resp5,dat$rtype,mean)
anaive expert 
 4.375  3.840 
tapply(dat$resp5,dat$rtype,quantile)
$anaive
  0%  25%  50%  75% 100% 
   1    4    5    5    5 

$expert
  0%  25%  50%  75% 100% 
   1    3    4    5    5

Also, in both models, the discrimination parameter is lower in experts.

The conditional effects of rtype look similar for the two models. It is clear that experts have lower discrimination with both the cratio and cumulative familes, but the probabilities of each category appear similar in both models

cratio:

image
image977×1122 33.6 KB

cumulative:-

image
image977×1122 33.6 KB

The cratio model fits slightly better than the cumulative model:-

Estimated Bayes factor in favor of model_crat over model_cumul: 2.18583

The vignette about the parameterisation of the cumulative and cratio families says:

Note that cumulative and sratio models use τ−η, whereas cratio and acat use η−τ. This is done to ensure that larger values of η increase the probability of higher response categories.

So, the results of the cumulative model seem to fit the idea that larger values of η increase the probability of higher response categories, but the results of the cratio model seem to contradict it.

I’d be very grateful for an explanation of what is happening and guidance concerning which model’s results I should believe

3

Hi there. Welcome to Stan discourse, and I’m sorry this question seems to have fallen through the cracks. I don’t have time to get all the way to the bottom of this, but I do note that the models are giving fundamentally very similar predictions, including similar predictions of the effect of expertise on the probability of each response. Therefore, the answer to your question does not lie in the fact that the models are saying wildly different things about the effect of expertise on ratings; rather the interpretation of the effect size differs between models.

I’m reasonably sure that the behavior is subtler than what I’m about to write below, but to give a general intuition for the sort of difference that I suspect could be at play, imagine that in one model class the fixed effect of expertise gave the difference for response 1, whereas in the other class of model it gave the difference for response 5. Then we would straightforwardly expect the effects to be opposite in the two model classes. To make progress, I would suggest digging into the code here to understand what is the interpretation of the quantity that is given by the predictor in each model, and what intuition do we have about how that quantity should be shifted due to the effect of expertise?

Alternatively, I might suggest interpreting these models in terms of the predictions that they make, rather than the meaning of the parameter estimate for expertise.

4

Thanks for getting back to me.

I am retired, now - but I was a clinician. The last formal maths teaching that I had was 53 years ago and that never included any statistics. But, I have accumulated an interest in statistics over the years. Unfortunately, that has never gone as far as understanding other people’s code. I need to read a plain English description of what a test is doing and I can “play around” with the test myself (e.g. by creating new data for it, that resemble or my ideas, or their opposite).

Anyway, I will try inventing some data (If I can think of possible reason for what is happening and a way to test it) and also compare the cratio and sratio families. I think the quote from Paul Buerkner’s vignette, that speaks of estimating “τ−η” for the cumulative families, but “η-τ” for the acat and cratio families is the most likely explanation that I’ve encountered, so far. So, I’ll pursue that.

If I succeed, then I’ll let you know.

Thanks again,

Jonathan