Brms: how can I compare differences of variance of marginal_effects() for categories (i.e. category probabilities)

Interfaces brms
1

Hi everyone,

I ran an ordinal mixed model (cumulative(“probit”), model below) on my data and I looked at the probabilites for each category for two factor levels (subject shift = 0 and subject shift = 1) for two separate groups (diagnosis = 0 and diagnosis = 1): subject_shift_diagnosis

I can see a tendency for the probabilites on level “subject shift = 0” to be less diverse in the group “diagnosis = 1” compared to the group “diagnosis = 0”, i.e. the ratings (ratings 1 to 7) are more likely to occur at guessing probability in the “diagnosis=1”-group, whereas the “diagnosis=0”- group seems to make more of a distinction by giving higher ratings more often. However, I want to investigate, to what extent this is statistically reliable. Does somebody have an idea, if it makes sense to run an F-test of variances of these probabilites, and how to access these proabilities? Or does somebody have a different, maybe easier way of investigating this?

Thank you very much for advice.
Juliane

model:

model_1_brm = brm(answer ~ 1 + subject_shift * diagnosis +  # 1 is included for clarity, it will be included anyway
                    (1 + subject_shift|subj_uid) + (1 + subject_shift|item),
                  data = df_all_subjects %>% filter(type %in% c('a', 'b')),
                  family = cumulative("probit"),
                  warmup = 2000, 
                  iter = 12000, # reduce iterations to e.g. 5000 and warmup to 1000, if you want to roughly get an impression. 
                  save_all_pars = TRUE,# this is needed for later calculating Bayes factor
                  set.seed(555)
)
2

Consider taking a distributional modeling approach and allowing the disc parameter to vary across experimental conditions. Paul has an example of this approach in his IRT paper (formula_va_ord_2pl, p. 37) and in his ordinal regression paper with Matti Vourre (fit_sc4, p. 17).

3

Thank you for your answer. I tried the approach in the ordinal regression paper by Bürkner and Vourre. I ran my model, but I do not know how to transform the disc parameter to standard deviation. It should be in their OSF repository, as the paper says, but I cannot find it.

4

With brms, \text{disc} = 1 / \sigma, which means \sigma = 1 / \text{disc}. However, once you start modeling the disc parameter, the brms default is to work with \log (\text{disc}), which means \sigma = 1 / \exp\big(\log (\text{disc})\big). You can get a sense of what that workflow might look like here

5

I ran a new model with the disc parameter according to the paper by Bürkner and Vourre and tried to interpret it. However, I am unsure I can infer from it, what I want to know, namely: Is the variance for diagnosis = 0 different from the variance for diagnosis = 1 for only one of the levels of the other factor (subject_shift = 0). I get the disc parameter information for my two factors and their interaction. But how can I access the information for only one factor level?