Bayes factors for intercept (BRMS)

I have data from 24 participants who completed each two conditions. I have computed the difference between the two conditions and I want to know whether that difference is negative (one-tailed t-test). Depending on the method I use, I get quite different results and I’m not sure which one is correct/ where I’ve made a mistake.

Methods I have used:

  • hypothesis function on BRMS model
  • bayes_factor function
  • JASP coded as one-sample t.test

Here is a visualisation of the data from JASP:

model_priorBound=  brm(CAPE.Pos_total_CDMminNDM ~ 0+Intercept,data=dfT,family=gaussian(),prior=set_prior("normal(0,1)",ub=0),iter=10000,warmup=1000,save_pars=save_pars(all=T))
model_noPriorBound=brm(CAPE.Pos_total_CDMminNDM ~ 0+Intercept,data=dfT,family=gaussian(),prior=set_prior("normal(0,1)"),iter=10000,warmup=1000,sample_prior = TRUE)
model_noIntercept= brm(CAPE.Pos_total_CDMminNDM ~ 0,data=dfT,family=gaussian(),iter=10000,warmup=1000,save_pars=save_pars(all=T))

The results are:

summary(summary(model_noPriorBound,prob=0.9)

Family: gaussian
Links: mu = identity; sigma = identity
Formula: CAPE.Pos_total_CDMminNDM ~ 0 + Intercept
Data: dfT (Number of observations: 24)
Draws: 4 chains, each with iter = 10000; warmup = 1000; thin = 1;
total post-warmup draws = 36000

Regression Coefficients:
Estimate Est.Error l-90% CI u-90% CI Rhat Bulk_ESS Tail_ESS
Intercept -0.59 0.50 -1.43 0.24 1.00 24878 21389

Further Distributional Parameters:
Estimate Est.Error l-90% CI u-90% CI Rhat Bulk_ESS Tail_ESS
sigma 2.84 0.42 2.24 3.61 1.00 24367 20714

→ 90% [one-tailed] credible interval includes 0: [-1.43;0.24]

hypothesis(model_noPriorBound,"Intercept<0")

Hypothesis Tests for class b:
Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio Post.Prob Star
1 (Intercept) < 0 -0.59 0.5 -1.43 0.24 7.42 0.88
→ Bayes Factor: 7.42

bayes_factor(model_priorBound,model_noIntercept)

Estimated Bayes factor in favor of model_priorBound over model_noIntercept: 1.83863

And from JASP, I get a figure assessing the Bayes Factor sensitivity to the prior:

→ Bayes Factor: 0.92
When I set in JASP normal(0,1) instead of their default of cauchy(0,707), I get BF:0.938
0

I suspect the source of the discrepancy between the brms and JASP results is a difference in the underlying statistical models.

I believe that under the hood, the JASP t-test is based on the BayesFactor R package, which performs the “JZS” t-test described by Rouder et al. (2009). This test is quite different from the brms models you fit above. For example, the t-test is based on the standardised effect size (mean difference from zero divided by the standard deviation), and assumes this effect size is drawn from a Student’s t-distribution (i.e., not a Gaussian likelihood). There are also some important details regarding the prior. So long story short, it seems like you didn’t perform an apples-to-apples comparison.

The manual of the BayesFactor package has some more details as well, and this stack exchange post explains the JZS prior it in great detail.

If you have a single model and \alpha is the random variable representing the difference in that value (it doesn’t need to be a parameter), you can evaluate \Pr[\alpha < 0 | y] given observed data y. You do this in Stan by coding \alpha as a parameter or generated quantity. This isn’t a t-test, but is probably what’s going on with the “one-tailed credible interval.” Rather than just reporting whether the 90% central interval of \alpha includes zero, reporting the probability here is more useful, especially as you have a direction. It’s like reporting the p-value rather than just the binary value of p < \alpha. And you can’t evaluate \Pr[alpha \neq 0 \mid y] for continuous \alpha, because it will always be zero.

Bayes factors are for comparing the prior predictive performance of pairs of models. So maybe the two models are with and without some effect whose value is the difference? Bayes factors are also not probabilistic, so the only way to evaluate the output is heuristically. Posterior predictive comparisons are usually more relevant unless you really trust your priors.

I don’t know it, but it sounds like JASP is looking at prior sensitivity, which doesn’t seem related to this difference calculation.

1 Like