I’m using brms to estimate Bayes factors for null effects found when fitting lmer models. To do so, I’ve fitted the full brm model to the data (using a normal distribution so comparable to lmer) and then used bayes_factor() to compare this model to models without the predictor of interest. However, I’m getting very large Bayes factors (over 1000) for effects that are nowhere near significance (t values are < 0.5) and I don’t think they’re quite right. I’ve tried running with more iterations, but it doesn’t seem to make much difference. I’ve also ran pp_check and the normal distribution doesn’t fit particularly well, but I’m not keen on changing it since the models showing null effects were fitted in lmer. Any idea what might be going on, or how I can solve this issue?

I attach the data and script in case they’re helpful. I fit a full model with three predictors, and then two null models (one without a main effect of CC, one without an interaction between CC and PC). I’m running brms version 2.2.0 on macOS High Sierra.

Hi Ruth, I didn’t have time yet to look at and run the R code, but maybe the priors aren’t amenable to this calculation? One problem with Bayes factors is that the priors you want to use for estimation are often not the ones that make sense for Bayes factors:

I’m not sure if that’s the issue here but definitely worth thinking about.

Also, if the PPCs look bad then why not try to find a model that fits better and estimate the “effect” of interest with that model? That would be a better estimate to trust, regardless of what a Bayes factor comparing two other worse fitting models says.

One other thing that could help debug is to try doing it with rstanarm (more about that in the link from my previous post) and compare to what you get with brms. But you’ll probably find that since they have different default priors the Bayes factors will come out differently unless the priors are specified to be the same (if possible).

Anyway, Bayes factors are a lot trickier than they are often made out to be. The bridgesampling package (used by brms and rstanarm) makes it easier than before, but it’s still quite complicated.

The problem with your Bayes factor is that brms uses improper flat priors for regression coefficients by default. These priors should not be used with Bayes factors. Thus, you need to set reasonable and proper priors on these coefficients. You may want to take a look at the this blog post: https://rpubs.com/lindeloev/bayes_factors