# Manual computation of two-tailed p-value for coefficient: how to find ESS?

(brms version is 2.19.0, cmdstan version 2.32.0, cmdstanr version 0.5.3.)

For a logit coefficient of interest, whose posterior has been sampled by brms, I need to test the null hypothesis that the parameter equals 0. I’ve been told that this can be accomplished by “evaluating the posterior probability of the region of values having smaller density than at \theta_j = 0.” (Agresti 2015: 339). To this end I’ve written a convenience function that does the following (key lines only):

post <- as.data.frame(as_draws_df(brmsmodel))
...
densities <- sapply(post[,CoefName], function(x) density_ratio(post[,CoefName], point = x))
cutoff <- density_ratio(post[,CoefName], point = 0)
p.val <- mean(densities < cutoff)


I presume that this is better than simply dividing the coefficient by its SE and calculating 2 \times the p-value of values as or more extreme as the posterior mean, assuming a normal distribution with \mu = 0 and \sigma = \text{SE}, particularly because not all the relevant parameter posteriors look symmetrically bell-shaped.

But here’s the question: I read in the helpfile of density_ratio() that “you may need an effective sample size of 10,000 or more to reliably estimate the densities.” How do I find this effective sample size? The brms summary output only lists bulk_ess and tail_ess, and I don’t know how to infer the crucial quantity from those numbers.

Reference
Agresti, A., 2015. Foundations of linear and generalized linear models, Wiley series in probability and statistics. John Wiley & Sons Inc, Hoboken, New Jersey.

UPDATE: Since I’m interested in posterior tail probabilities here, could I use use ess_tail instead of the unreported “effective sample size”? I.e. could I just increase sampling iterations until summary(brmsmodel) reports ess_tail at \geq10000, and trust that all manner of things shall be well?

Second question: should I be concerned that posterior::ess_bulk() and posterior::ess_tail() return values which Differ Slightly from those reported by summary(brmsmodel) ?

You may want to look at hypothesis() instead of using density_ratio().

2 Likes