(brms version is 2.19.0, cmdstan version 2.32.0, cmdstanr version 0.5.3.)
For a logit coefficient of interest, whose posterior has been sampled by brms, I need to test the null hypothesis that the parameter equals 0. I’ve been told that this can be accomplished by “evaluating the posterior probability of the region of values having smaller density than at \theta_j = 0.” (Agresti 2015: 339). To this end I’ve written a convenience function that does the following (key lines only):
post <- as.data.frame(as_draws_df(brmsmodel))
...
densities <- sapply(post[,CoefName], function(x) density_ratio(post[,CoefName], point = x))
cutoff <- density_ratio(post[,CoefName], point = 0)
p.val <- mean(densities < cutoff)
I presume that this is better than simply dividing the coefficient by its SE and calculating 2 \times the p-value of values as or more extreme as the posterior mean, assuming a normal distribution with \mu = 0 and \sigma = \text{SE}, particularly because not all the relevant parameter posteriors look symmetrically bell-shaped.
But here’s the question: I read in the helpfile of density_ratio() that “you may need an effective sample size of 10,000 or more to reliably estimate the densities.” How do I find this effective sample size? The brms summary output only lists bulk_ess and tail_ess, and I don’t know how to infer the crucial quantity from those numbers.
Reference
Agresti, A., 2015. Foundations of linear and generalized linear models, Wiley series in probability and statistics. John Wiley & Sons Inc, Hoboken, New Jersey.