How to think about prior effects on posterior distributions

Modeling
21

There are no inferential approaches that are robust to bad assumptions; garbage in, garbage out and whatnot. All we can do is thoroughly document our model development process so that others can accurately judge the motivation for those inputs.

I think that you’re falling into the philosophical trap of there being “one true approach to inference” or “one true praxis”. Bayesian inference performs a precise task (incorporating information encoded in the prior model and a realized likelihood function in a self-consistent way) given certain assumptions (information can be modeled with probability distributions, the assumed prior model and observational model capture the relevant structure of the system being studied). Frequentist inference performs another, albeit similarly precise task (calibration ensemble behavior of estimators in the context of a loss function) given certain assumptions (observational model, true data generating process in the observational model). Neither is universally correct in any way, rather one is likely more appropriate than the other in any given analysis based on the goals of that analysis.

Regardless of which approach is taken, the assumptions required of that approach will have to be carefully considered and validated for the resulting inferences to mean anything.

2 Likes
22

Or, as Larry Wasserman puts it:

Frequentist Inference is Great For Doing Frequentist Inference.
Bayesian Inference is Great For Doing Bayesian Inference.

Inference procedures are optimal under the goals they set out to fulfill. Same with models. With this whole COVID-19 thing, models have taken to the forefront of public discourse and the aphorism:

All models are wrong, some are useful

has been thrown around a lot. It might be the cranky daddy in me, but it seems to me people have been focusing too much on the “wrong” part and less on the “useful” part. To say a model (or inference procedure) is “useful” is meaningless until you explicitly add your goals. So, as Michael says, once you lay out clearly your assumptions and goals, you have a few self-consistent/reasonable ways of going about your inference/modelling task.

I personally prefer the Bayesian perspective, having been trained in it from an early point in my career. That said, I’m teaching Statistical Inference next semester and will be presenting both approaches. Students are expected to use Linux and R for their assignments, though. Any Windows or Python usage shall be met with unmitigated fury! ;-)

2 Likes
23

I’m not sure much changes if you replace “research” with “inference”. Unless you’re arguing that scientific inference is not what scientists do in scientific research :-) (which is overwhelmingly not Bayesian).

CSSR yields a valid markov chain so I’d be surprised if it was somehow inconsistent. Would you mind saying more about why the bootstrap is inconsistent? I’m fairly naive to this topic, so would be grateful of an example of a incosistent but frequently used estimator.

24

It isn’t bad assumptions I’m worried about. It’s scientists pretending to more prior knowledge than they actually have in order to get better results. I mean the less your prior is surprised, the better your error bars will be, which establishes a direct motivation to manipulate your prior. I feel data dredging (p-hacking) is already par for the course.

I think I may have been misunderstood here because I didn’t make any claims with regards to a single principled praxis. My interest in particular is in what makes your workflow (which I’m yet to study) a “Bayesian workflow”. I.e. what about it is particular to Bayesian inference as opposed to anything else. I.e. how much would I need to change before I could rename it into a “Principled Likelihoodist Workflow” for example (Pawitan, In All Likelihood).

25

The last part about Linux and R made me laugh :-) Regarding “both” ways, I think that Kullback’s information theoretic approach is a third way and Pawitan’s (and others) likelihood only approach is a fourth way.

26

This is quite the thread…I’m going to try to write this faster for my own sake, so bear with me…and I’m going to jump off the thread after this. Thanks for the discussion!

@emiruz I din’t think you were actually falling into the “one true approach” trap. It might apply to me though, at least a little bit, since I think there are some logical rules that apply to induction, learning from evidence, and we should try to flesh them out so we can benefit from them when approaching complex problems. And then we can also recognize, hey—that method isn’t following the rules, I wonder what magnitude of inferential problem thats creating here.

Debate about probability theory is sorely needed, but I think we should be aiming to build the best we can, not produce a menu that we can choose from at will. That would lead to a heck of a subjectivism—with far more leeway than choosing a prior, you’re free to choose your system of logic! I do recognize though that multiple theories will and should co-exist, its how we undertake healthy debate, and get rid of bad theories, advance the best.

Here’s a real example: someone in my family says that its their opinion that Coronavirus is not more deadly than the flu once you consider the number of elderly deaths, and its just my opinion that they are wrong. I say that effectively they are wrong given the information that we all have. that is, they are using incorrect reasoning. I need to appeal to some objective standard for reasoning with evidence; otherwise, they are basically right! Just my opinion. I think advancing and defending science requires we do our best to create an objective logic, i.e. outside of individual control, validated as best as we can, as well as some broader standards for acceptable practices that we can hold each other to. That’s the motivation for my comment suggesting that Bayes theorem may be a necessary but insufficient foundation for inference—its just to argue “we do need logic” and “Bayes theorem is sound logic,” so sound inference is in part “that which doesn’t contradict Bayes theorem.”

Again, to avoid misunderstandings, there’s lots more to data analysis than Bayes theorem.

Would you mind saying more about why the bootstrap is inconsistent?

Maybe someone else will correct me or contribute a much better answer! I guess would say that its a neat trick to calculate the sampling variability of an estimator; but that’s it–its clearly a neat trick (a method constructed to fit a very particular problem) rather than a method derived using rules of probability (obviously by taking that as my standard I’m presupposing Bayes theorem…but that is the point). The previous comment on Jeffreys and using data that we have not observed was intended as a reference to the bootstrap.

1 Like