How can I select a ROPE range for a logistic model based on odds ratios

Hello,

I am trying to understand how to specify ROPE when fitting a logistic model.
Let’s say I would like to detect whether the effect of a dummy-coded predictor (e.g., intervention) is larger than 1.20 or smaller than 0.8 on an odds ratio. I want to examine whether the intervention increases/decreases the outcome value by 20% or larger; if not, the intervention does not make a meaningful difference.

In this case,
ln(1.20) = 0.1823215568
ln(0.8) = -0.2231435513

So, I should set my ROPE as -0.2231435513, 0.1823215568.

Am I understanding this right? I feel this makes sense, but I did not encounter such a simple guideline, so I wonder if I might be misunderstanding something.

  • added:

I found this post (ROPE range specification for binary models) and started understanding more clearly, but I still feel I am not 100%.

So, does the log(1.1) in the post refer to ln(1.1)?

Then, I guess my above approach is in the right path, but the interpretation should be:

I want to examine whether the intervention increases/decreases the ODDS of the outcome value (i.e., success/fail rate) by 20% or more; if not, the intervention does not make a meaningful difference.

I may be struggling, as English is not my first language, and I am not keen on math. If somebody can confirm my understanding, I would greatly appreciate it.

Hey, @a_t. I think part of the difficulty here, which is part of the difficulty that often comes up with logistic regression, it there are many ways to talk about the results. For example, at times you are talking about odds ratios, and other times you are referencing percentages. Let’s get specific. How exactly do you want to express your outcome? As an odds ratio? As a difference in probabilities? As one probability expressed as a percent change relative to another probability? Something else?

Me, for example, I always prefer probability contrasts, which is one probability minus another probability. I believe this is sometimes called a risk difference (though the jargon of “risk” is a poor fit for my discipline). Other folks, however, love those odds ratios.

Thank you for your response, @Solomon! I truly appreciate you and this community, as I am learning Bayesian modeling alone without having anybody near me to ask such questions.

My initial idea was as an odds ratio.
By me stating

increases the odds of the outcome value by 20% or more

I intended to express the differences in odds ratio (1.2 times+) compared to a control condition (e.g., no intervention).

I am using the brms code something like as follows:
fit = brm ( outcome ~ intervention + pretest_outcome + (1|participant) + (intervention | item), family = "bernoulli",...)
where intervention was dummy coded as 0 for control and 1 for intervention. outcome is a correct/incorrect response on a cognitive task (e.g., math questions) after the treatment (intervention vs. no intervention). We did the same test for the outcome variable as a pretest beforehand, hence pretest_outcome in the model.

So I maybe should have stated like…?
I want to examine whether the intervention increases/decreases the ODDS of the outcome value (i.e., success/fail rate) 1.2 times or more/less compared to the control condition; if not, the intervention does not make a meaningful difference.

then, setting rope to be:
upper: ln(1.20) = 0.1823215568
lower: ln(0.8) = -0.2231435513
Am I understanding this right?


Me, for example, I always prefer probability contrasts, which is one probability minus another probability. I believe this is sometimes called a risk difference (though the jargon of “risk” is a poor fit for my discipline). Other folks, however, love those odds ratios.

Thank you for the suggestion. I am wondering if this corresponds to the comments made here (ROPE range specification for binary models)?

So, with this approach, we need to know the baseline odds to start with right? In my experiment, I do not know how well the control condition performs on the outcome value beforehand. Even in such a case, can I still use the probability contrasts?

I know it’s been a while, but if I could hear any ideas from @Solomon and/or anybody about my understanding and confusion of setting a ROPE range, it would be really appreciated! :)

I’d suggest following @andrewgelman’s advice and just report the posterior ranges rather than trying to reduce to a binary decision. If you have some downstream decision-making process, then you can compose the two with Bayesian inference without first reducing the posterior to a binary decision of significant/non-significant. That is, is there really that big of a difference between 19.9% and 20.1% that you want to treat them completely differently?

Usually people use log and ln interchangeably, with log being much more popular in programming languages and stats.

Thank you for your insight, @Bob_Carpenter !

I appreciate your confirming my understanding and setting of the ROPE range and suggesting that I avoid testing hypotheses dichotomously. I totally agree with you, and the reason I got so fascinated by Bayesian modeling in the first place was that I thought that I could interpret results in a more nuanced manner.

However, I am having difficulty dealing with the practical aspect of analysis in my research area to take full advantage of the posterior. If I may, I would like to ask a follow-up question about this.

Could you kindly let me know which reference (and the section name and/or page number, if possible) you would recommend I read into this? I have read Bayesian Data Analysis (Third edition) as well as other Bayesian books (Statistical Rethinking and Doing Bayesian Data Analysis), but I feel that I am not establishing the way I want to interpret the posterior distribution. However, I could be missing a lot.

In my research area, most studies still rely on the frequentist approach. So, even if Bayesian modeling is adopted, the “meaningfulness” of a predictor variable is judged simply based on whether or not the 95% Credible Interval includes 0.

Many people recommend setting a credible interval range based on the area’s knowledge, but there is no practical shared idea on the size of credible intervals in the area. Also, if we keep judging whether the credible intervals include zero or not, I feel that we cannot avoid the binary decision… This is probably because we need to (1) set a range that sounds familiar to the majority (i.e., 95%) and (2) judge the meaningfulness of predictors quickly based on a pre-determined range (either 95% or 89% or anything else, but decided beforehand to make ‘subjective’ judgment) because usually there are many predictors to examine in a study.

Could you suggest how to deal with such a situation?

I may be missing, but I feel that I have to have a pre-determined cut-off point to make a judgment. I could make a statement such as “the probability of a participant providing an accurate response on the final test being above 80% was 80%”, based on the posterior… but this seems to be fairly complicated as talking about the probability of the probability, so does not seem so appealing to reviewers.

I could describe the posterior with multiple ranges (e.g., 5%, 10%, 50%, 90%, and 95% CrI), but I am unsure how to make a nuanced judgment without a predetermined cut-off range, such as whether 90% CrI excludes 0…

I have been reading many articles and books on hypothesis testing in Bayesian and learning about the probability of direction and ROPE, and other stuff (e.g., Reporting Guidelines • bayestestR), but I feel I am still having difficulty taking the best advantage of bayesian modeling.

Any suggestions and references that I can learn and read into would be very much appreciated!

Look, for example, at figure 5.4 in BDA3 (you can get it free online from the book’s home page). It just reports the 95% intervals. McElreath is just using 89% to get across the idea that hte choice is arbitrary.

I’m not sure why you feel that a binary decision is inevitable. What’s the downstream task? Why report whether it’s non-zero at some interval width versus just reporting the median and 90% interval.

What kind of judgement are you trying to make and what kind of nuance are you worried about? Let’s say we estimate a regression coefficient and it’s 90% interval is (0.1, 1.8) and it’s 99% interval is (-0.2, 2.9). Report them both. Show a histogram. But why try to reduce to a yes/no decision about significance? For a start, these coefficients only make sense relative to the other coefficients. You’ll get different results about whether the “income” covariate is “significant” depending on which other covariates are available such as age, sex, education and zip code.

I think you need to get to the bottom of why you feel you need to make a judgement about some parameter’s marginal posterior interval containing zero or not. If it’s because a journal editor or advisor wants a p-value, then I doubt anything you do with Bayes is going to make them happy.

The thing to read is McElreath’s book, Statistical Rethinking—it’s aimed at exactly someone like you asking exactly these kinds of questions. He also has online videos and a lot of other teaching material.