I have calculated a Bayesian linear mixed effect model with brms for the first time for my thesis and now I am stuck on how to report the results correctly.
Specifically, I am stuck on the fact that I feel almost helpless not being able to use the term “significant”.
Let’s imagine the following result: in my model, I found a regression weight for my treatment that has a value of -11.69 (median, 95% CrI: [-14.34,-9.04], pd = 99.99%) points on the outcome. In my frequentist framework, I would simply report that the intervention group has significantly improved compared to the control group and report the point estimate, CI and p-value.
So far, however, I have not found a good example of analogous reporting in Bayesian models. What do I need to report? Does the significance result from the CrI or do I have to explicitly write that I reject the H0? I understand the principle and think it is very good, but I notice that I am lacking the concise sentences that I usually read and use?
Can someone perhaps recommend me some good example papers? (Preferably from the field of psychology, sociology, or medicine).
I think the short answer is you can just say the treatment negatively affects the outcome (estimate and CI). And if -9 is a large enough effect size to be important or noteworthy, then you can explain that as well.
Significant/significance has a very specific meaning in null hypothesis significance testing that is quite different from its meaning in ordinary language, and we usually avoid the terms in Bayesian inference just to avoid confusion with their specific and counterintuitive frequentist meanings. You might eventually find that you feel more free, rather than more constrained, by not using the term. You are free to talk about the effect size and your degree of certainty about the effect size, and whether such an effect size is of practical importance, rather than being constrained by convention to report on whether or not you can reject a null hypothesis.
Note that you especially should not write that you reject H0; the idea of rejecting a null is entirely wrapped up in null hypothesis significance testing, which to a first approximation corresponds to frequentist inference.
Thank you very much for your explanation, I think it is incredibly helpful.
I’ve looked at a number of studies so far on https://pubmed.ncbi.nlm.nih.gov/ and just got totally confused. Some describe in their methods that they do the analyses in a Bayesian framework, but still report p-values or use the term significant or confidence interval. In general, the articles are very inconsistent in the way they report.
Therefore, I am very grateful for this clear input.