Hello everyone,
I’m having some modelling issues and I’m hoping the answer to the question above might help. I am running a multinomial/categorical model in brms and am getting results that do not make sense for my data (e.g. high probabilities for outcomes that actually have close to zero observations in that condition). One thing I was playing with to hopefully find a cause and repair the model are the reference levels. This made me realize that the predicted probabilities change a lot when I change the reference level, when (to my understanding) they should not be changing at all, since the reference level serves as a baseline for only the estimates and not the predicted probabilities. So now I am hoping that finding the cause of this issue might tell me more about what is going wrong with this model.
Further information on the model:
- the categorical outcome variable has six levels
- there are 12 total predictor variables, all of which are either categorical or binomial
- I have run this model successfully with three predictor variables before, but this larger model should be a better fit based on LOO estimations
- as it worked with three predictors, but not 12, I have also tried using six predictors to see if there is a cut-off point that I might be able to identify: the results also did not fit the data
Model calculation and predicted probabilities:
Mdl <- brm (categorical outcome variable ~ predictor variables + (1 | participant),
data, family = categorical(),
prior = priors.weak,
chains = 4, iter = 3500, warmup = 1000,
backend = "cmdstanr",
sample_prior = "yes",
cores = parallel::detectCores())
#calculation of predicted probabilities
condeff.mdl<- conditional_effects(Mdl, categorical = TRUE, re_formula=NULL)
condeff.mdl.predictor1 <- condeff.mdl [["predictor1"]]
With the original reference levels, this gives me the following results (rounded numbers).
The main problem in comparison to the data is that the predictions for outcome level 2 are way too high to make sense for my data.
| Outcome level 1 | Outcome level 2 | Outcome level 3 | Outcome level 4 | Outcome level 5 | Outcome level 6 | |
|---|---|---|---|---|---|---|
| Predictor level 1 | 0.0458 | 0.8773 | 0.0123 | 0.0162 | 0.0026 | 0.0006 |
| Predictor level 2 | 0.0139 | 0.9452 | 0.0093 | 0.0063 | 0.0015 | 0.0002 |
| Predictor level 3 | 0.0309 | 0.9137 | 0.0170 | 0.0077 | 0.0008 | 0.0003 |
| Predictor level 4 | 0.1282 | 0.6346 | 0.0181 | 0.0601 | 0.0024 | 0.0015 |
| Predictor level 5 | 0.0262 | 0.6625 | 0.2398 | 0.0012 | 0.0011 | 0.0003 |
| Predictor level 6 | 0.0379 | 0.9158 | 0.0015 | 0.0091 | 0.0032 | 0.0008 |
| Predictor level 7 | 0.0171 | 0.9319 | 0.0288 | 0.0022 | 0.0003 | 0.0002 |
| Predictor level 8 | 0.0190 | 0.7712 | 0.1688 | 0.0006 | 0.0005 | 0.0004 |
After changing the reference level, I get the following numbers.
When compared to the data, the predictions for outcome level 4 are now way too high.
| Outcome level 1 | Outcome level 2 | Outcome level 3 | Outcome level 4 | Outcome level 5 | Outcome level 6 | |
|---|---|---|---|---|---|---|
| Predictor level 1 | 0.2295 | 0.0270 | 0.0157 | 0.6567 | 0.0100 | 0.0010 |
| Predictor level 2 | 0.1617 | 0.0677 | 0.0274 | 0.6070 | 0.0133 | 0.0009 |
| Predictor level 3 | 0.2691 | 0.0483 | 0.0380 | 0.5509 | 0.0053 | 0.0010 |
| Predictor level 4 | 0.2073 | 0.0055 | 0.0079 | 0.7406 | 0.0031 | 0.0008 |
| Predictor level 5 | 0.2276 | 0.0284 | 0.4850 | 0.0850 | 0.0069 | 0.0008 |
| Predictor level 6 | 0.2726 | 0.0487 | 0.0029 | 0.5499 | 0.0178 | 0.0020 |
| Predictor level 7 | 0.2838 | 0.0971 | 0.1266 | 0.3179 | 0.0035 | 0.0012 |
| Predictor level 8 | 0.2407 | 0.0502 | 0.5070 | 0.0621 | 0.0050 | 0.0018 |
If anyone has an idea what might cause something like this, that would be great! Please let me know if more information is necessary to answer my question.
- Operating System: macOS Tahoe 26.3
- brms Version: 2.23.0