Dear all, I would like to run a Bayesian logistic regression model using Jeffreys priors for the regression coefficients using the R/brms package. How do I do that?

Kr, Shiny

Dear all, I would like to run a Bayesian logistic regression model using Jeffreys priors for the regression coefficients using the R/brms package. How do I do that?

Kr, Shiny

Hi, @shinyrhino. Stanâ€™s not set up with built-in Jeffreys priors as far as I know, because we recommend using at least weakly informative priors in most cases. If you want to use a Jeffreyâ€™s prior, you will have to figure out what the parameters are and specify your own prior parameters for brms.

With multiple variables, things get a lot trickier. See: Jeffreys prior - Wikipedia.

1 Like

Dear @Bob_Carpenter, thanks for the advice! Assuming that there are multiple covariates of various scale levels (categorical, continuous), measurement units/coding, and distributions, I thought a scale-invariant, non-informative prior like Jeffreys would be an elegant solution. Following your recommendation of using weakly-informative priors (per regression coefficient), is there a recommended approach so the varying priors are equally weakly informative for the regression coefficient of each covariate?

Itâ€™s definitely elegant, but the resulting priors can be really difficult in cases where thereâ€™s not much data. Also, if you have multiple parameters, you need to work out the Fisher information matrix to define a proper multivariate Jeffreys prior.

What our more sophisticated regression packages like brms and rstanarm do, I believe, is apply a QR decomposition to the covariates, work in the reduced Q space, then reconstruct back to the original scale. Then a simple prior works well, but we tend to use ones that are weakly informative rather than purely scale-invariant.

Thereâ€™s a lot more discussion in the Bayesian Workflow paper

and associated book draft.

Generally, I wouldnâ€™t worry so much about the prior independently, but think about the combination of (a) data, (b) likelihood, and (c) priorâ€”they can only really be understood together as all three determine the shape of the posterior.

1 Like

Dear @Bob_Carpenter, thanks for your kind input and the reading tip! Following Gelman et al. (2008) who stated that most regression coefficients for logistic regression are between -5 and +5, a simple normal prior N(m=0,sd=100) would only allocate about 4% (pnorm(5,0,100)-pnorm(-5,0,100)) of the probability of this generic prior to the typical range of coefficients, and would therefore be reasonable weakly uninformative for a broad range of covariates. Too simplistic?

Kr, Shiny

Ref:

Gelman, A., Jakulin, A., Pittau, M. G., & Su, Y.-S. (2008). A weakly informative default prior distribution for logistic and other regression models. *The Annals of Applied Statistics*, *2*(4), 1360â€“1383. A weakly informative default prior distribution for logistic and other regression models

You have to be careful about that. Aleks and Andrew wound up recommending a Cauchy prior which Andrew has largely disavowed. And it really depended on standardizing the regression predictors. Otherwise, the regression coefficients completely depend on the problem. For instance, if I give you distances in angstroms or in light years, the regression coefficients change scaleâ€”so you have to standardize everything (convert covariates to be centered with unit scale, i.e., standardized). Even better, apply a QR decomposition and work with the orthogonal, unit-scaled Q matrix.

Also, youâ€™re not dealing with â€śmost regressionsâ€ť, you have a specific regression about which you probably know at least a little bit or you wonâ€™t be able to assess whether your modelâ€™s working for whatever purpose itâ€™s intended.

Iâ€™d recommend something like `normal(0, 5)`

if you expect the regression coefficients to be on the order of 5 or smaller.

1 Like

@Bob_Carpenter and for all interested: The SAS manuals state that Jeffreys priors are available for PROC GENMOD and can be constructed for PROC MCMC:

"While Jeffreysâ€™ prior provides a general recipe for obtaining noninformative priors, it has some

shortcomings: the prior is improper for many models, and it can lead to improper posterior in some cases; and the prior can be cumbersome to use in high dimensions. PROC GENMOD calculates Jeffreysâ€™ prior automatically for any generalized linear model. You can set it as your prior density for the coefficient parameters, and it does not lead to improper posteriors. You can construct Jeffreysâ€™ prior for a variety of statistical models in the MCMC procedure. See the section â€śExample 80.4: Logistic Regression Model with Jeffreysâ€™ Priorâ€ť on page 6256 in Chapter 80, â€śThe MCMC Procedure,â€ť for an example. PROC MCMC does not guarantee that the corresponding posterior distribution is proper, and you need to exercise extra caution

in this case. " [1]

See the section â€śExample 80.4: Logistic Regression Model with Jeffreysâ€™ Priorâ€ť on page 6256 in Chapter 80 of the PROC MCMC manual [2].

Refs:

[1] https://documentation.sas.com/api/collections/pgmsascdc/9.4_3.5/docsets/statug/content/introbayes.pdf

[2] https://documentation.sas.com/api/collections/pgmsascdc/9.4_3.5/docsets/statug/content/mcmc.pdf

@Bob_Carpenter One more thought: Would be possible to use a Jeffreys Prior in a multivariable regression model and get around the problem of specifying a multivariate prior by by constructing a hierarchical prior where the parameters for the various Jeffreys priors per coefficient are drawn from a shared, overall Jeffreys prior?

Kr, Shiny