I am using bayes factor for the first time and have difficulties to interpret the outcome. I want to test whether my predictor effect is “significant” or I should probably say if there is evidence for an effect. My model results give me a point estimate for a regression coefficient of 0.12 with 95% CI [-0.08, 0.41]. I followed this up with hypothesis() to see whether there is more evidence for or against an effect.
When I do hypothesis(model, class = “b”, ‘b = 0’) I get an evid.ratio of 6.81. So this tells me there is more evidence that the effect is 0 than that it is not zero. When I do hypothesis(model, class = “b”, ‘b > 0’) I get an avid.ratio of 10.24, which tells me there is more evidence that there is a positive effect.
What shall I now conclude from this? Is it valid to do a one-sided test if this is in line with my predictions?
Thanks
Hi,
so there is a lot of conflicting statistical advice going around the Internet and real life, I hope I won’t be adding much to the cacophony. I understand if you are a bit overwhelmed.
What I think you are seeing is an instance of Bayes factors being confusing. I have to admit I never took time to get a good grip on Bayes factors because of such confusing properties (which I read about). I think a similar issue is discussed at [78a] If you think p-values are problematic, wait until you understand Bayes Factors - Data Colada (the tone of the post is a bit too agressive and snarky for my taste, but the explanation is nice) - Bayes Factor will integrate over your whole prior and are thus sensitive to your prior choice and to the way you choose your hypothesis.
Due to those confusing properties of Bayes factors most people here on the forums would probably not recommend using them broadly (just my guess - we didn’t have a vote or something).
So unless you have to use Bayes factors (your boss/reviewer/… wants it), I would just interpret the posterior distribution of the coefficient directly (mean, 50% interval, 95% interval at minimum). I.e. I would say something like “Provided that our model is a good approximation, the data cannot rule out negative effect (95% CI [-0.08, 0.41]) and are also consistent with negligible effect (posterior probability of $|b| < 0.X% is Y%), although strong positive effect is possible as well (posterior probability of $b > 0.Z is Q%)”.
If you totally need Bayes factors, then I’ll try to bring somebody with more background on BFs than I do.
I recently wrote an answer to a similar question which might also be of interest:
Hope that clarifies more than confuses and best of luck with your research!
Hi Martin,
Many thanks for your answer. I did read before that the use of bayes factor for hypothesis testing is debated but thought I give it a try anyways because I’m still a little stuck in my frequentist thinking and felt as though something was missing otherwise. But in that case, I will stick to interpreting the CIs. Thanks for giving me an example about how I could write something like this.