Hi
I recently ran several Bayesian analyses using the informed prior vs the default prior. I understand why the same analyses using the informed prior gives a bigger Bayes factor compared to the default, but I didn’t expect the effect sizes to all be smaller (and credible intervals narrower) when using the informed prior. Does anybody know why this might be?
For example, if one uses an extremely informative prior, the posterior would be affected quite much - then the credible intervals would be also quite narrow. So, having narrower CrIs would make sense.
The effect sizes could also be affected by the informative prior: suppose, if we take informative but relatively wrong prior, then the effect size would be smaller compared to non-informative but more correct prior
Just to note that the effect size estimates can also be smaller when the informative prior is “correct”. That is, the smaller prior-regularized estimates might be consistent with both the true generative process and the realized data, and the larger effect size estimates under the uninformative prior might just be noise, or worse might be an indication that the “uninformative” prior unintentionally informative (for example if it is relatively flat on the logit scale). @maguirea1 this shrinkage of effect sizes is in general a good thing, as long as the informative prior is chosen carefully to encapsulate genuine domain expertise.
Smaller effect sizes under the informative prior are only a sign of a (potential) problem if there is clear prior-data conflict (i.e. if the posterior under the informative prior shows little overlap with the prior and/or little overlap with the posterior under the uninformative prior). If this happens, it might be a signal to re-evaluate how you translated your domain expertise into a prior, to make sure that you didn’t make a mistake or make any overconfident assumptions.
1 Like
Related to the discussion, see priorsense GitHub - n-kall/priorsense: priorsense: an R package for prior diagnostics and sensitivity for prior and likelihood sensitivity analysis and checking possible prior-likelihood-conflicts
1 Like