However, I have a question about the emerging PBW (that’s principled Bayesian workflow, not Pabst Blue Workflow) that was motivated by a reviewers comment on a manuscript about whether a prior was truly weakly informative with respect to my observed data. Those questions are: " Are the priors weakly informative as claimed? Have you tried other settings and examined sensitivity to the results? How sensitive is the posterior predictive checks to the prior settings?" (Emphasis mine)
The PBW address prior sensitivity through simulated data with question three. But my reviewer seems more concerned with assessing prior sensitivity with respected to my observed data. So my question is what can I do to address the reviewers concern.
One option would be to choose some set of different priors and refit the model to see if the posterior predictive checks and posterior distributions change very much. Here I’m thinking something along the lines of the sensitivity analysis described in BDA3 Section 17.4. The only issues with this is that I have a lot of parameters and a single run of the model can take a day or more. So it isn’t clear how to do such a sensitivity analysis: do I change the priors for parameters one at time holding all others fixed? That could results in hundreds of model runs.
The question of how exactly to do such a sensistivity analysis got me thinking about other options. One thing I thought of would be to assess posterior shrinkage using the same measure as specified by @betanalpha in the workflow: s = 1 - \frac{\sigma^2_{post}}{\sigma^2_{prior}}. This would allow me to look at all of my parameters with one simple and consistent measure. But that only address the change in the variance by fitting to the observed data. Is there another measure I could use to assess a change in the central tendency? Something like the z-score mentioned in the PBW but comparing the prior and posterior mean?
I’m looking for general advice or suggestions for references. And I’m asking here in the hope that the answers provided here can be of use to other readers of the forum. Thanks!
This is an unfortunately common response from people who, in my opinion, have internalize incorrect perspectives on how assumptions work in data analysis.
Firstly in the context of the prior model weakly informative does not mean non-informative. Rather weakly informative is meant to convey that the prior encompasses just enough domain expertise to demarcate extreme, but not impossible, model configurations from those that are reasonable. More informative priors would go beyond this and further constraint even those model configurations within the “reasonable” neighborhood.
A consequence of this is that there is no expectation that weakly informative priors will not influence posterior inferences. When the data are not strongly informative through the likelihood function then even weakly informative priors can strongly inform the shape of the posterior distribution, which is exactly why we recommend them so strongly! We want our domain expertise to improve our inferences and one of the motivations of the workflow I advocate is to identify where that domain expertise is most needed in the model.
All a prior sensitivity analysis* demonstrates is that your prior assumptions influence your inferences, but there’s nothing wrong with that influence! Moreover, the particular assumptions encoded in the observational model will also have an influence so why isn’t anyone asking you to check the sensitivity of your inferences to those assumptions? In most cases this is due to an unfortunate bias against prior modeling based on fear and incomplete information that is common in applied fields.
Anyways your inferences will in general be sensitive to all of your assumptions, with the magnitude of the sensitivity varying depending on the exact structure of your model. What is important isn’t quantifying that sensitivity but rather being able to motivate and argue for the validity of those assumptions in the first place. I don’t care if my weakly informative prior scales influence the posterior under a collinear likelihood function so long as we can all agree that those scales isolate extreme model configurations that would be unreasonable in the context of the given analysis.
In any case that is unlikely to sway the reviewer. Your best bet may be to isolate a single function of the parameters that best characterizes your inferences and just repeat your analysis for five or so priors and plot a histogram of posterior samples for that single function output. Because the underlying critique is ill-posed, the solution will be too and often times you just have to provide something even if it’s not necessarily a great answer.
One additional complication is that “sensitive analysis” can mean many different things. For example it can refer to how sensitive a decision making process is to the particular observed data (i.e. sampling observations and running the analysis over and over again to see the range out of outcomes), how sensitive inferences are to the experimental design (i.e. sampling observations from various data generating process, running the analysis over and over again, and seeing how the range out of outcomes varies with the underlying data generating process), and how sensitive inferences are to model assumptions (the approximations made in the observational model, the domain expertise encoded in the prior model, etc).
To the reviewer’s credit, out of a hundred or so parameters including many using the logit-link, they asked specifically about only a subset of the parameters that used the identity link. At a glance of the specification, I can understand how they may seem somewhat concentrated and thus somewhat more than weakly informative.
So after a day or two thinking on it, my strategy in responding to this reviewer’s concern is twofold: 1) show some prior predictive checks demonstrating the range of values in the summary statistics, and 2) rerun the model for some small subset of alternative priors.
I didn’t show the prior predictive checks in the supplementary materials for the initial submission as I did the posterior predictive checks. That’s in part because when I was first preparing the manuscript I wasn’t familiar with prior predictive checks. That’s why I called the PBW “emerging,” you and others that haunt this forum have done a wonderful job of laying out a suite of practices for developing complex Bayesian models that while not exactly “objective” (objectivity is dead!), are logical, self-consistent, and transparent. But these practices are still new and unfamiliar to many.
I’m much more of an applied statistician than anything else. Most of my work will be applied analyses where the results and their interpretation are the primary takeaway for the reader. My plan moving forward with future manuscripts is to layout the steps of the workflow in the supplementary materials. It generally won’t be and shouldn’t be important to the main body of the manuscript but I think they are very important to include in the supplementary materials for interested readers. This, not only because the reader should be able to follow the breadcrumbs of the model development, but also to expose more people to the Bayesian workflow.
Take care – diffuse priors can be incredibly informative when link functions are involved! If anything the priors on parameters hidden behind link functions (especially log and logit) should be subject to more scrutiny.
If you’re going to consider alternative priors then I would suggest showing how the prior predictive distributions change with the changing prior assumptions. This might even be enough to avoid (2) entirely.
Yes! We’re writing as much material as fast as we can but applications like yours will be super helpful in demonstrating the utility of these methods and increasing adoption. Even if you don’t include things like prior predictive checks in the paper consider having an expanding preprint or case study or blog post that discusses the details in the language of your field.
Definitely! Focus on communicating the scientific goals of the analysis while keeping all the paper work easily accessible for those who want to more deeply scrutinize the underlying model.