Ratio of prior to posterior odds and frequentist p-value

General
1

Hi,

I have a question concerning the validity of a proposed interpretation of the ratio of prior to posterior odds of directional hypotheses. Specifically, I am interested to know whether it is valid to interpret a particular instance of this ratio, with symmetric priors, as something very similar to a frequentist p-value of the null-is-nil hypothesis.

It is widely noted that directional tests about a parameter [e.g., Pr(b > 0|D)] do not provide information about the null-is-nil hypothesis. Rather, such tests assume the effect exists and investigate the probability of its sign.

On the other hand, the typical, Bayesian way to investigate the null-is-nil hypothesis is to compare nested models, e.g., via Bayes factor. Here, the key parameter is set to zero in one model (i.e., removed from the model formula) and compared with a model in which the parameter may vary.

But, imagine that you have a probability model with, for simplicity, one predictor. That predictor has some weakly informative prior symmetric about zero. It seems to me that, given this prior parameterization, the null-is-nil hypothesis uniquely corresponds to [Pr(b > 0)/Pr(b < 0)]/[Pr(b > 0|D)/Pr(b > 0|D)] = 1. Is this correct?

If that is correct, then it seems that one can piecewise define the posterior odds, such that values less than 1 [i.e., that favor Pr(b < 0|D)] are inverted. This gives something like the “hypothesis free evidence” (HFE) for the effect taking some sign, either positive or negative. For instance, the HFE of both Pr(b > 0|D) = .95 and Pr(b > 0|D) = .05 is 19. Thus, the ratio of prior to posterior odds is (.50/.50)/(19) = 1/19 = .05.

It is correct to interpret smaller values of the above ratio as greater evidence against the null-is-nil hypothesis, but not for it? I.e., analogous to a frequentist p-value, but with a fundamentally Bayesian interpretation as the improbability of the null-is-nil hypothesis?

Thank you for any engagement with this question.

2

I don’t think this follows through logically because Pr(b > 0)/Pr(b < 0) = 1 is a necessary but not sufficient condition for a point-null.

1 Like
3

In a Bayesian model whose posterior probability density is everywhere finite (and thus the probability mass over any specific exact value is infinitesimal), one can always take the total probability mass in a sufficiently small window around some value (e.g. zero) and get an arbitrarily small probability. Thus, you will never be able to make a consistent statement about some non-infinitesimal probabilty of a point null except by constructing a model that assigns finite prior and posterior mass (thus infinite density) to the null case.

Note, however, that even in the case of a model with just one parameter, you can still if desired compare that model to a simpler model with the restriction that the parameter be zero.

1 Like
4

Thank you, that makes sense. I had a similar sort of intuition but I wasn’t thinking as clearly in terms of necessity/sufficiency. Perhaps I am thinking too hard about this, but I am interested if you have any thoughts on the following.

Just to be certain that I’m being clear, my thinking was that it is the entire ratio of prior to posterior odds (equal to 1) that might serve as a statement of a true null-is-nil hypothesis, not just the numerator. That is, if the null-is-nil hypothesis were true, and the relevant parameter were estimated without error, wouldn’t that entail no reallocation of the probability mass from the prior to the posterior (i.e., and so the above ratio equals 1)? And, moreover, that this ratio would not equal 1 for any non-zero latent value? I understand that the ratio could equal 1 due to, say, error accompanying estimation from finite samples.

In terms of necessity/sufficiency: In hindsight, I suppose my thinking was that the lack of sufficiency is the reason why the ratio of prior to posterior odds of directional hypotheses might have the low-information, frequentist interpretation that I suggested. Specifically, the definition of the null-is-nil hypothesis provided by the proposed ratio is weaker, in the sense of being a merely necessary but insufficient definition of the null-is-nil. Therefore, it provides less information about the null-is-nil than, say, a Bayes factor (in which the null-is-nil is given a necessary/sufficient definition). Specifically, it can only provide evidence against the null hypothesis, but never for the null hypothesis.

5

Thank you for this explanation, your points are well taken. I have a quick follow-up question on this, if you have time.

Perhaps my suggestion that the ratio of prior to posterior odds of directional hypotheses (equal to 1) as a measure of the improbability of the null-is-nil hypothesis was too strong.

However, this ratio must equal 1 only if the latent parameter value is zero (ignoring error; please correct me if I am wrong on this). Thus, it still seems as if this ratio provides a measure of some relevance with respect to gauging what is, in some sense, the “non-credibility” of the null-is-nil hypothesis (whatever exactly “non-credibility” might mean).

I would be interested to know if this reasoning is incorrect. Thanks again.

6

I think that evaluating this reasoning requires formalizing this notion. However, note again that in Bayesian analysis, unless we construct a prior that assigns a point mass to zero, we are a priori absolutely certain that the point-null not strictly correct. Perhaps it would be more fruitful for you to investigate whether this ratio has any necessary (I suspect not) or probabilitic (perhaps so) relationship to the probability mass in some narrow window around the point null, or in other words to the probability density at the null. Note that if the data are very uninformative, then the posterior will recapitulate the prior, but that shouldn’t be taken as any evidence whatsoever that the null is even approximately correct. Even if the data yield a posterior that is perfectly symmetric about the null, that posterior can be very wide, very narrow, or somewhere in between. And again, even if the posterior is very narrow, the posterior probability that the null is exactly correct is still zero.

7

I agree that if you have evidence that any effect is greater than (or less than 0) that is evidence against a point null (which, by definition has no density above or below 0). And that evidence for the point null does not follow from the inverse. (When I stated that Pr(b > 0)/Pr(b < 0) = 1, I really mean it approaches 1 as the null’s prior’s width is shrunk to zero to approach a point null – an infinitesimal ROPE). But I think this just follows directly from either Pr(b > 0|D) or Pr(b < 0|D). I think a larger question is whether the point null is an even remotely interesting thing to test for in many fields of enquiry.

8

For what it’s worth, I don’t agree with this. The point null is already epistemically ruled out, except in the very special case where we assign a prior point mass to the null (in which case I totally agree that any evidence that the true value is either greater than or less than or [either greater than or less than] the null is evidence against the null; this seems tautological).

If by “the null” we really mean some ROPE, then I still don’t agree with this. For example, if the posterior is gaussian entered on the null, we could imagine a different posterior that reallocates some probability mass from left of the ROPE to right of the ROPE, but without changing the probability mass of the ROPE at all. This would constitute “evidence that the effect is greater than the ROPE” but NOT “evidence that the effect is outside of the ROPE”.

9

I agree with both points but my understanding was that the OP was referring to a point null in which case evidence for the true value being greater than or less than that null is clearly evidence against that null.