When computing Bayes Factors, do the null and alternate hypotheses have to have the same random effects structure?

Interfaces brms
1

Please also provide the following information in addition to your question:

  • Operating System: OSX 10.14
  • brms Version: 3.5

Imagine that I am interested in the Bayes Factor in favour of brightness having an effect on response time to shapes. Do the null and alternate hypotheses have to have the same random effects structures? For instance, is this okay?

m1 <- brm(RT~Brightness + (1+Brightness|Subject) + (1|Item))
m0 <- brm(RT~(1|Subject) + (1|Item))
bf <- bayes_factor(m1,m0)

Or is it a problem that the alternate hypothesis has a random subject slope?

2

They just have to have the same number of observations and the same outcome (and one of the models has to be correct).

3

I actually get substantially different Bayes Factors based on whether I include the random subject slope or not. I wonder if one option is more correct?

4

A Bayes Factor is essentially asking which prior predictive distribution was more consistent with the observed data, so it is entirely possible to get very different answers when including or excluding things that have a noticeable effect on the prior predictive distribution. It is certainly much easier to estimate the most general model and look how far the variance of the group-level parameters is zero in the posterior distribution.

5

This and also the computation gets more difficult when there are more parameters, so you should check the stability of the computation, too.