New paper and package "Detecting and diagnosing prior and likelihood sensitivity with power-scaling"

[avehtari]

New paper Detecting and diagnosing prior and likelihood sensitivity with power-scaling with @n-kall, @topipa, and @paul.buerkner ,
with Supplementary code
and priorsense R package.

Our goal was a prior sensitivity diagnostic that would be fully automatic for brms/rstanarm type packages or would require minimal user work otherwise. The prior and the likelihood are part of the same model, thus diagnosing the likelihood sensitivity makes sense, too.

We diagnose how much the posterior changes when prior or likelihood is power-scaled, i.e., exponentiated with some \alpha>0. This corresponds to scaling how informative the prior or the likelihood is.

Using importance sampling, we can easily compute arbitrary quantities measuring how much the posterior changes. As the default choice, we use the symmetrised metric version of cumulative Jensen-Shannon divergence.

For some quantities like mean and variance, we can also derive local analytic sensitivity at \alpha=1.

Ideally, the posterior is likelihood sensitive, i.e., the likelihood is informative. If the posterior is also prior sensitive, there might be a prior-likelihood conflict. If the posterior is only prior sensitive, there might non-informative likelihood.

We can provide the diagnostic for all parameters and interpretation of the diagnostic. All this is available in the priorsense R package.

The presence of prior sensitivity or the absence of likelihood sensitivity is not always a problem. Sometimes the prior can be strongly informative, and then a prior sensitivity can be desired.

Power-scaling prior and likelihood sensitivity diagnostic is not perfect and can miss some things, but as it’s easily automated it will help quickly catch many typical issues and thus has a natural role in the bigger Bayesian workflow.

[hhau]

Nice idea, paper, and package.

An off-topic comment re: Section 2.2. I can see the intuitive appeal for not \alpha-scaling p(\theta \mid \phi) (hyperpriors seem to be the thing we have information about in most analyses and can reasonably adjust), but prior-data conflict can occur due to an inappropriate choice of p(\theta \mid \phi). I read the IWMM paper and thought about repurposing IWMM to estimate prior-data conflict the elements of \theta using the node-split p-values discussed in Conflict Diagnostics in Directed Acyclic Graphs, with Applications in Bayesian Evidence Synthesis (one issue they raise is the computational burden of exact CV, which IWMM would alleviate?). I guess you’ve had similar thoughts and quite reasonably ruled it out-of-scope for this work?

[hhau]

Also something does not quite match up between Figure 7, the text just below it, and Figure 8. According to Figure 7, the posterior moves to the right as \alpha increases for prior-scaling, but according to the text and Figure 8 it moves to the left. Maybe the labels on Figure 7 are reversed?

[n-kall]

Thanks for spotting this! It turns out the labels for Figure 8 are reversed, black square should be prior, white square should be likelihood. A new version will be uploaded soon :)

[n-kall]

Thanks for the positive comment! You bring up a good point that the hyperpriors are not the only possible cause of prior-data conflict (although they are an intuitive starting point). There is definitely room for further extensions for prior-data conflict checking (and sensitivity analysis in general) in hierarchical models, and it will likely require approaches other than power-scaling (but could still use PSIS/IWMM as you suggest). And thank you for the paper on conflict diagnostics, clearly highly relevant!