P-value from Bayesian Logistic Regression

Algorithms
1

Hi all.
I have run the analysis using frequentist approach logistic regression. To know whether the independent variable is significant, I will be looking into the p-value. Then, I repeat the same process by using Bayesian.

I am currently at the stage of drawing the posterior distribution using rstan. After getting the median, MAD_SD, posterior summary statistics, etc, I would like to know how I can confirm that each independent variable listed is significant.

I wish to compare the results from Bayesian with the frequentist approach.

Thank you.

2

Hello Agnes, and welcome to the forum!

I suspect that you aren’t going to get a lot of traction on this post, because p-values are a Frequentist thing, and don’t have a place in Bayesian analyses/tests/models, and as such we don’t really talk about “statistical significance” at all.

Can I ask why you are running both a Frequentist and Bayesian analysis? If you’re just curious to compare the difference results, you may wish to just plot the confidence intervals from the Frequentist model with the credible intervals from the Bayesian model. The plot_models() function from package sjPlot might be useful there.

1 Like
3

Thank you very much for the reply.

I would like to compare the final model by using both approach to see which method perform better. Previously, I run the bayesian model using rjags package, however, I think this package is quite outdated. Therefore, I wish to try rstan. I am impressed with the rstan especially the visual and outputs given by the ShinyStan.

Anyway, thank you very much for your kind help.

4

The notion of ‘significant’ means very little and is dependent on highly arbitrary thresholds, plus it covers up high false negative probabilities. You would be better served to compute an uncertainty interval for the importance of each variable, either using Bayes or a bootstrap procedure. Importance should be based on things like Statistically Efficient Ways to Quantify Added Predictive Value of New Measurements – Statistical Thinking

5

Hi, @Agnes and welcome to the Stan forums.

May I ask why?

I think that depends on the kind of Bayesian you are. If you look at something like Gelman et al.'s book Bayesian Data Analysis, you’ll see posterior predictive p-values introduced to mirror the way a chi-squared goodness of fit test works in a frequentist setting to evaluate in-sample fit of a regression, for example.

Similarly, if you look at the earlier chapters, there’s a lot of analysis of where Bayesian posterior intervals and frequentists confidence intervals either agree or disagree that doesn’t reduce things to a binary significant/not-significant.

Some people will look at a regression coefficient’s 95% interval and declare it “significant” if it doesn’t overlap zero. I’m not recommending doing this, just reporting what some people do in practice.

So it really depends what your goal is in calculating p-values.

1 Like