Interaction is significant, but neither simple effects nor Bayes analysis

General
,
1

I am currently running a 2 x 2 x 2 repeated measures ANCOVA in JASP. I have 2 within factors (animacy, word pairing) and one between factor (group).

I have found a significant interaction of animacy * group. However, when i use the “simple effects” option in JASP, it yields that both simple effects are not significant. I have plotted the interaction using the estimated marginal means using R; it is a slight crossover interaction. However the crossover is not as intense to make the simple effects non significant I believe. I have run additional pairwise post-hoc comparisons for the animacy by group interaction (using Bonferroni correction). The pairwise comparisions yielded two significant differences, one of which was that the two levels of the animacy factor for one of the groups (community-dwellers) showed a significant difference.

I additionally ran a Bayesian analysis for the ANCOVA and found that the BFincl for the animacy * group interaction was only 0.311, hinting towards the null hypothesis.

Could these issues be due to power? (N = 56; n group 1 = 26; group two = 30). What should I do now? Which additional analysis could I run and which analysis do I report? How do I interpet the interaction?

Thanks in advance!

ANCOVA

image
image1691×878 192 KB

image
image689×499 7.02 KB

Post Hoc & Simple Effects

image
image793×808 19.9 KB

Bayesian analysis

image
image1691×898 197 KB

2

Hi, are these somehow related to Stan and mcmc?

3

Sorry, no its not

4

I’d suggest reading the Bayesian Data Analysis, 3rd Edition chapter on model comparison for why Bayes factors are problematic. Here’s a link to the book’s home page, that has a free pdf: Home page for the book, "Bayesian Data Analysis"

I know they’re a key element of JASP, but it’s not something we encourage with Stan—we don’t provide any methods to compute the data marginals. I also wrote a blog post about how Bayes factors measure prior predictive performance, not posterior predictive performance, which is what we usually care about for downstream inference. So I’d urge you to do more posterior predictive checks, which are the Bayesian analogue of goodness of fit tests.