Above chance in a Bayesian model

Modeling
1

Hello,

As a beginner in bayesian stats, I have the following question. In my model, I test how speakers of RP language discriminate a particular vowel contrast compared to speakers of CG language (reference level). Here’s my model’s formula:

fit <- brm(response ~  0  + Intercept + language + (1 | subject), data = test, family = bernoulli("logit"), prior = prior, sample_prior = "yes")

Family: bernoulli 
  Links: mu = logit 
Formula: response ~ 0 + Intercept + language + (1 | subject) 
   Data: test (Number of observations: 320) 
  Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
         total post-warmup draws = 4000

Group-Level Effects: 
~subject (Number of levels: 20) 
              Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(Intercept)     0.87      0.30     0.42     1.58 1.00     1373     2192

Population-Level Effects: 
           Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept     -0.06      0.32    -0.68     0.58 1.00     1694     1964
languageRP     2.76      0.62     1.59     4.07 1.00     2503     1846

Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).

Now, I want to do a hypothesis testing to find the evidence of above chance performance (i.e., > 50%) in the discrimination of the contrast by the two populations. I would like to know the code for this testing

Your help will be much appreciated!

2

This should do the trick:

hypothesis(
  fit,
  c(
    "Intercept > 0", # for the reference group
    "Intercept + languageRP > 0" # for the RP group
  ) 
)

If anything’s not clear, please ask.

3

Thanks for your reply!

I need to do two things. First, to find the evidence of languageCG < languageRP. For that, I used the following code in hypothesis testing 0 < languageRP. Am I correct?

The other thing is the one I asked, that is, to find if each language group performed above chance. Does the code you provided show evidence for above chance performance, that is, performance of more than 50%? (as the forced choice task had two options). What if the chance score was, for example, 70%. How the code would change?

Many thanks!

4

Yes

Yes

You would now need to be checking if your effects are greater than logit(0.7) = 0.8473 . So hypothesis(fit, "X > 0.8473"). Note that logit(0.5) = 0, which is why you would be checking if the effect exceeds 0 if chance is 0.5.

1 Like
5

Fantastic, many thanks again!