Model comparison using bayesfactor_models() (bayestestR package) produces -Inf/Inf

Interfaces brms
1

Hello everyone,

for model comparison, I calculated Bayes factors using the function “bayesfactor_models” from the bayestestR package. The models I ran with brms are ordinal models. When I calculated the Bayes factors, I got the factors, but also a warning of which I am not sure how bad it is, where it comes from and how to troubleshoot it. I paste the models further below. Maybe somebody has an idea.

I paste here the output of the model comparison for the model and the respective null model:

# Bayes Factors for Model Comparison

Model                                                                                          
[1] 1 + subject_shift * diagnosis + (1 + subject_shift | subj_uid) + (1 + subject_shift | item)
BF
17.17

* Against Denominator: [2] 1 + (1 | subj_uid) + (1 | item)
*   Bayes Factor Type: marginal likelihoods (bridgesampling)
Warning message:
14 of the 32000 log_prob() evaluations on the proposal draws produced -Inf/Inf.

models:

model_subj_shift_brm_more_iterations = brm(answer ~ 1 + subject_shift * diagnosis + (1 + subject_shift|subj_uid) + (1 + subject_shift|item), 
                           data = df_all_subjects %>% filter(type %in% c('c', 'd')), 
                           family = cumulative('probit'),
                           warmup = 2000, 
                           iter = 18000,
                           save_all_pars = TRUE,
                           set.seed(555),
                           control = list(adapt_delta = 0.9),
                           init_r = 0,
                           prior =
                             c(prior(normal(0, 2), class = Intercept),
                               prior(normal(0, 2), class = b),
                               prior(normal(0, 2), class = sd),
                               prior(lkj(1), class = cor)
                             )
)

model_subj_shift_brm_NULL = brm(answer ~ 1 + (1 |subj_uid) + (1 |item), 
                                           data = df_all_subjects %>% filter(type %in% c('c', 'd')), 
                                           family = cumulative('probit'), 
                                           warmup = 2000, 
                                           iter = 18000,
                                           save_all_pars = TRUE, 
                                           set.seed(555),
                                           init_r = 0, 
                                           control = list(adapt_delta = 0.9), 
                                           prior =
                                             # the prior for the intercept, i.e. the mean rating
                                             c(prior(normal(0, 2), class = Intercept),
                                               # taus in our model
                                               prior(normal(0, 2), class = sd)
                                             )
)

ADMIN EDIT: Formatted code for clarity

2

Sorry for note getting to you earlier, your question is relevant. Did you manage to move forward in the meantime?

Anyway, while I have little direct experience with Bayes factors, they tend to be finicky and hard to get right, so it is possible you are just hitting one of the problematic situations. I also noticed you have increased adapt_delta and changed inits, so I presume the models had problems otherwise? If so, it is likely that computing bayes factors has a bit stricter requirements on the model behaving “nicely” than just fitting with brms and you might need to address the underlying problems before being able to compute Bayes factors.

I’ve also previously written about alternatives to Bayes factors for model selection/comparison: Model comparisons and point hypotheses so this might help as well…

3

Thank you for your answer. I have not been able to solve this issue so far. And I am (Bayesian)statistically too unexperienced to actually understand, why this happens in the first place.

4

Unfortunately, I can’t advise much beyond that. I think some potential issues (if you need Bayes factors) are:

  • The NULL model differes from the original model by multiple predictors, maybe comparing when adding one predictor at a time would be stable
  • Formulas like 1 + subject_shift * diagnosis + (1 + subject_shift|subj_uid) + (1 + subject_shift|item) can be problematic to fit because you include the subject_shift term as both fixed and varying effect and unless your data is very rich, it might be impossible for the model to distinguish the two leading to all sorts of problems. I would try either 1 + diagnosis + subject_shift:diagnosis + (1 + subject_shift|subj_uid) + (1 + subject_shift|item) or 1 + subject_shift * diagnosis + (1 |subj_uid) + (1 |item) - would those work without the need to increase adapt_delta? Would they give sensible Bayes factors when compared with the null model?

But I am really just guessing… Best of luck with your model!

5

Hello Martin,

thank you very much for taking the time to get back to me. I have tried your suggestions and found that only the one which involves taking out the random slopes (but leaving in random intercepts) works smoothly without procuding the error message. I originally did not want to leave out the random slopes, but now found that they do not seem to affect the results much. So, I will now work with these simpler models, not including random slopes.

Best wishes,
Juliane

1 Like