a general question (has coding and statistical aspects):
I have a model that looks like this:
m<-brm(DV~0+IV1IV2+IV3+IV1:IV2:IV3+(1+IV2IV3| subject), family=bernoulli(),…)
So its a logistic regression with:
IV1 - Factor (3 between subject conditions)
IV2 - Factor (2 within subject conditions)
IV3 - Continuous (centered; within subjects)
(currently dummy coded; IV2 and IV3 are fully crossed, and present in each condition of IV1.)
As you see this goes beyond the ‘simple’ main effect formulations which are usually presented here, and this comes with some problems in the Bayesian logistic regression framework I am struggling with.
What I am especially interested in are the slope (IV3) differences between the levels of IV1, by each level of IV2. For instance: on IV2(Level1)-> is the slope steeper in IV1(Level1) than in IV1(Level2); or statistically speaking: is their difference 0? or: are the posteriors of the slope-differences in the region of practical equivalence? or: what is the evidence (BF) for this?
Now the above model formula is based on what Paul once suggested somewhere else, and serves as a neat start, since it provides indeed all the six separate estimates for:
This is nice and convenient, however, this is a logistic regression… and the problem is, that there are no really meaningful priors, since the posterior of each of these Slope estimates depends on ‘what goes into the logit’ from the other model terms. Given that there is an effect of the slope(IV3), then if a lot variance is already explained by IV1 alone, then even large changes of the slope on the log-scale will hardly change the log-odds (or p(1|Model)). But if none of the other factors contribute to the prediction, then small changes on the slope might alter the prediction a lot. This then of course also depend on the type of data one got (e.g. lots of 1’s or lots of 0’s), and makes “a priori” prior scale considerations somewhat futile. Thus, depending on the “variance” in the binary data and other IV’s, the prior range of the IV3 estimates might to broader or narrower, respectively.
Although this might be a question I could ask, this serves as a mere example, of why I do not want to interpret the estimation results with Svage-Dickey density rations for prior vs. posterior, i.e. Bayes Factors with hypothesis(m, “IV1_1:IV2_1:Slope-IV2_1:IV2_1:Slope=0”), which depends on the arbitrary prior scale.
Now… a different approach: the newest GitHub version of brms allows to specify by=IV ‘random effects’ (population variance / hyper parameters). This comes quite handy here, because this actually gives a population variance estimate in each condition of IV1 like this:
m<-brm(DV~0+IV1IV2+IV3+IV1:IV2:IV3+(1+IV2IV3| gr(subject, by=IV1)), family=bernoulli(),…)
What I have done now, is first simplistic (and I think wrong) to be as clear as possible: First, I take a “standard” convention like: a small effect is a difference of d=.2 (Cohens d; standardized mean difference), and the region I do not want the slope-effect differences (IV3) in is d=-.1 to d=.1 (usually called ROPE, see Kruschke’s late work). Now the by-IV1 variance estimates actually can be pooled in a classical sense, like: sigma_pooled=sqrt((sd_Intercept_IV1_1^2+sd_Intercept_IV2_1)/2), which then could be used to make sense of d_low=-.1 and d_high=.1. Namely, by their multiplication (e.g. d_low*sigma_pooled), which scales the “convention” of the Null-region to the log-scale, for a particular effect comparison. Such a region then could be checked for whether the posterior HDI of the slope differences lie within or outside it.
So far so good, I wonder if there are any objections until this point. I guess, the first “wrong” thing about this would be, to use an inappropriate variance term for pooling. This might be related to the question to whether one would take “within” or “between” subject-variance to interpret effect sizes like d. This then further splits up to which type of variance should be included (random intercepts, and which random slopes). For instance, I could use only the variance hyper estimates of the random slopes for IV3; or one could use all other residual terms, including the intercepts. Are there any technical estimation issues with this? In the latter case, however, I can not think about how all of these variance terms should be “added” together, because I simply don’t now how their deeper relation in the model code is, in the first place…
As it looks like, again, the reasoning behind this might become quite complex in a Bayesian sense, although the simple question still breaks down to:
Is there a neat way to “standardize” the slope - difference estimate for a better comparison to some meaningful decision criterion, about how much evidence is there, or whether the effect is “large” or something like this. I wonder, whether everything could be so simple.
A next approach:
So that is why my actual main question is: Is there a way in brms to model the slope effects - by the IV2 levels, directly as a combination of the grandMean ± (d/2)*SD; where the difference estimate is a standardized (Cohen d like) effect (unitized prior), multiplied with a variance-on-the-log-scale estimate. In my imagination, this would solve all my problems :)
Paul previously suggested to me that contrast.coding might be able to give me something similar to this, but I can not see it, because it still would imply estimating/modeling the Slopes (IV3; as in the last model version) independently, but not as a result of d*SD. This would again imply to transform the estimates to the standard scale (as above) for determining a (hopefully) meaningful ROPE, with all the above questions and possible difficulties. It also would appear less elegant to me, because directly modeling d and then transforming it into the actual effects (on the models-logit-scale), might automatically take all the other variance estimates into account by definition (this is actually a question…), which may not lead to the above question of which variance term I can and should use to standardize the slope-difference effects. (This actually almost answers my own question: because the SD term here, of course, should model the random slope variance of IV3…).
OK. I stop here. I hope my goals are understandable and that there is an easy solution I do not see :)