Implicit uniform prior for computing the log marginal likelihood

Modeling
,
1

I am trying to compute the Bayes factor via bridge sampling from Stan models.

According to the vignettes by Gronau (2021),

… to compute the (log) marginal likelihood for a Stan model, we need to specify the model in a certain way. Instad of using “~” signs for specifying distributions, we need to directly use the (log) density functions. The reason for this is that when using the “~” sign, constant terms are dropped which are not needed for sampling from the posterior. However, for computing the marginal likelihood, these constants need to be retained. For instance, instead of writing y ~ normal(mu, sigma) we would need to write target += normal_lpdf(y | mu, sigma)

In a model block, the Jeffreys prior for the error standard deviation \sigma_\epsilon is coded as

model {
  target += -log(sigma);     // Jeffreys prior
}

Moreover, the Jeffreys prior for the grand mean \mu implicitly follows a uniform distribution. My question (to compute the log marginal likelihood) is: Do I need to write it out in some _lpdf syntax or leave it undefined?

model {
 // mu ~ implicit uniform prior     // Jeffreys prior
}

A concrete example can be view at RPubs - Bayes Factor for One-Way ANOVA

Thanks for any comments.

2

brms models are compatible with bridgesampling by default and they generate code which uses implicit uniform priors, so I’d think you were fine. But the bridgesampling devs are also on discourse, so I’ll ping them to clarify just in case: @Quentin @Henrik_Singmann

In general though, I’d argue that it’s best practice to make priors explicit in models, to guard against any changes to default behaviours in the future

3

Thank you, Andrew.

The grand mean \mu, in this case, has been assigned an improper uniform prior (unbounded) (Stan User’s Guide, version 2.31). I doubt we can write it explicit in Stan, can we?

4

If you’re referring to the implicit uniform, then note that:

parameters {
  real mu;
}
model {}

Is the same as:

parameters {
  real mu;
}
model {
  mu ~ uniform(negative_infinity(), positive_infinity());
}

@avehtari what’s your perspective on whether or not to explicitly specify the implicit uniform prior?

5

Terrific! Thanks again, Andrew.

1 Like
6

Marginal likelihood is not defined when prior is improper, and uniform(negative_infinity(), positive_infinity()) is improper, so it is irrelevant in this thread whether it is defined implicitly or explicitly.

They are not fine with improper priors. The brms documentation example explicitly defines proper priors (Log Marginal Likelihood via Bridge Sampling — bridge_sampler.brmsfit • brms)

@DexterW you need to define proper priors to have valid (log) marginal likelihood computation.

4 Likes
7

Ah of course, thanks Aki!

8

Thank you, Aki.

I have tested: (1) undefined, (2) large-range flat, (3) large-spread normal.
// mu ~ implicit uniform prior
target += uniform_lpdf(mu | -1000, 1000);
target += normal_lpdf(mu | 0, 100);

The resulting Bayes factor values are very close. I guess it is because the same Jeffreys prior is assumed on \mu for both competing models of the Bayes factor.

9
  1. The title of this thread says marginal likelihood and not Bayes factor, and bridgesampling package first estimates the marginal likelihoods
  2. When computing the Bayes factor directly, it is possible that some terms cancel out, but bridgesampling package estimates the marginal likelihoods first separately and the cancellation of improper prior is not happening.
  3. With improprer prior, the marginal likelihood is not defined, but the algorithm used by the bridgesampling package is still returning a finite estimate (which is infinitely wrong). The priors you tested are all flat in the interesting part of the posterior, and as bridgesampling package uses an algorithm that focuses around the posterior draws, and thus provides similar estimates, but the estimates also for the proper priors are likely to be far from the true marginal likelihood. Unfortunately, bridgesampling package doesn’t provide good diagnostic when the estimate is likely to be unreliable.
2 Likes