What could have gone wrong if the summary output of a model shows no evidence of an effect, but the conditional_effects graph shows a clear effect?

Interfaces brms
1

My problem is that I can see in the graph of conditional_effects and in the graph of probabilities of getting a response correct, that there is a time effect. But the output of the model says there is no effect.

time: 0, 1, 24 hrs
group: younger and older

model30 <- brm(stringent ~ 1 + time + group
  + time:group
  + ( 1 + time | id )
  + ( 1 + time + group | item ),
  data = data30,
  family = bernoulli(),
  iter = 4000,
  file = "model30 "
)

I ran a model to figure out if older people forget faster than younger people. Here is are the results of the model:

Population-Level Effects: 
                     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept                2.74      0.43     1.94     3.65 1.00     1726     3176
delayhour               -1.40      0.32    -2.05    -0.79 1.00     3141     4444
delayday                -2.21      0.36    -2.93    -1.52 1.00     2574     3900
groupolder              -1.31      0.53    -2.33    -0.27 1.00     1516     2809
delayhour:groupolder     0.15      0.38    -0.58     0.90 1.00     3260     4860
delayday:groupolder     -0.01      0.44    -0.87     0.85 1.00     2423     4031

To find if there was an effect between one hour and one day, I ran this code

my_samples <- posterior_samples(mod30items)

c <- my_samples$b_Intercept + my_samples$b_delayday
d <- my_samples$b_Intercept + my_samples$b_groupolder

younghour.youngday <- c - d
meanhd <- round(mean(younghour.youngday),2)
cihd <- round(quantile(younghour.youngday, probs = c(0.025, 0.975)),2)

This is the result:

> meanhd
[1] -0.89
> cihd
 2.5% 97.5% 
-1.77  0.03

As you can see, there is no evidence of an effect of time between one hour and one day. But now look at the graph from conditional_effects. It´s super obvious that there’s a difference.

One
One1280Ă—720 30.2 KB

Even more, this is the graph of the probabilities of getting each response right, and it’s the same!

Two
Two1280Ă—720 40.2 KB

What did I do wrong?

If possible, add also code to simulate data or attach a (subset of) the dataset you work with.

Please also provide the following information in addition to your question:

  • Operating System: Windows 10
  • brms Version: 2.12.0
1 Like
2

It is not clear how this test

c <- my_samples$b_Intercept + my_samples$b_delayday
d <- my_samples$b_Intercept + my_samples$b_groupolder

relates to the effect of time – it is comparing the effect of being older with the effect of “one day”.

Tests such as these are probably easier to make directly within brms using the hypothesis function. I think you mean to test something like one or all of the following (I am assuming here you are using dummy coding for your factors, which is the default):

h <- c("delayhour < 0", # evidence delayhour is less than immediate for the younger group
       "delayday < 0", # evidence delayday is less than immediate for the younger group
       "delayday - delayhour < 0", # evidence delayday is less than delayhour for the younger group
       "delayhour + delayhour:groupolder < 0", # evidence delayhour is less than immediate for the older group
       "delayday + delayday:groupolder < 0", # evidence delayday is less than immediate for the older group
       "(delayday + delayday:groupolder) - (delayhour + delayhour:groupolder) < 0" # evidence delayday is less than delayhour for the older group
)
hypothesis(model30, h)

General advice is that if your conditional effects plots do not seem to correspond to your hypothesis tests, one of them has been misspecified.

Edit: corrected misplaced bracket

3

Thank you so much. This cleared everything up for me. I made a mistake and wrote $b_groupolder instead of b_delayhour. If nothing else, this proves that I shouldn’t do stats at 2am…

So, following what you taught me, I figured out how to test if there is an interaction. My real research question is whether older adult’s performance decreases with time faster than the performance of younger adults. Following what you wrote I concluded that to test if the decrease between one hour and one day is larger for older adults than for younger adults, I would add the last line:

h <- c("delayhour < 0", # evidence delayhour is less than immediate for the younger group
       "delayday < 0", # evidence delayday is less than immediate for the younger group
       "groupolder < 0", # evidence groupolder is less than groupyounger at immediate
       "delayday - delayhour < 0", # evidence delayday is less than delayhour for the younger group
       "delayhour + delayhour:groupolder < 0", # evidence delayhour is less than immediate for the older group
       "delayday + delayday:groupolder < 0", # evidence delayday is less than immediate for the older group
       "(delayday + delayday:groupolder) - (delayhour + delayhour:groupolder) < 0", # evidence delayday is less than delayhour for the older group
       "(delayday - delayhour) + delayday:groupolder < 0"  # evidence the slope between delayhour and delayday  is steeper for older adults than for younger adults

)

Do you know what could I read to understand the difference between the output of summary(model30) and the hypothesis function? If I wrote that last line correctly ( “(delayday - delayhour) + delayday:groupolder < 0”) in the result of hypothesis() the CI are -1.5 and -0.13 which means substantial evidence of a difference, but the output of summary() says the CIs are -0.87 to 0.85 which means there is no substantial evidence of a difference.

4

If you want

evidence the slope between delayhour and delayday is steeper for older adults than for younger adults

you need to test delayhour:groupolder - delayday:groupolder. You can see why by looking at the two relevant conditions in the last set of tests. Firstly, there is

"delayday - delayhour < 0", # evidence delayday is less than delayhour for the younger group

secondly, there is

"(delayday + delayday:groupolder) - (delayhour + delayhour:groupolder) < 0" # evidence delayday is less than delayhour for the older group

These are the two conditions you want to compare and if you subtract one from the other, the delayhour and delayday terms drop out leaving delayhour:groupolder - delayday:groupolder.

hypothesis in brms does the same as what you did in your original test: it takes linear combinations of draws from the posterior distributions for the predictors and then calculates the probability these combinations are greater than or less than a given value. They can also be used to test against a point value (but that is something I rarely do). I think the best source of info is just the help file within brms.

2 Likes