Basic question regarding simple groups differences, credibility and evidence

Interfaces brms
1

Hi! I am new in Bayesian Statistics and I’m trying to perform some analysis using brms. However, I’m facing some difficulties. I have two really basic questions and I would be grateful if you could give me your opinion. They can be simplified as follows:

  1. Consider the following model:
    m <- brm(y ~ Group + (1|Subj), data=dat)
    where y is count data and Group is a categorical variable with two levels, A and B. Since y has underdispersion and currently brms has no specific family for underdispersion, may I trust the results from this model when convergence is achieved?

  2. Suppose I can trust these results. I want to assess the differences between the two groups. I thought about assessing the difference by analyzing the (Bayesian) 95%CI and the Evidence Ratio (ER).

-95%CI: The output of model m gives me an estimate, x=.61, and a 95% Bayesian CI = [-0.04, 1.29].

-ER: I specified the model
m_prime <- brm(y ~ 0+Group + (1|Subj), data=dat, save_all_pars = TRUE, sample_prior=TRUE);
and used the ‘hypothesis’ function [that is, hypothesis(m_prime, “GroupB - GroupA > 0”)], and obtained the same estimate (x) and ER = 26.97.

So, it seems to me that the concepts of ‘credibility’ and ‘evidence’ are very different. Can you please tell me an opinion about how to report this between-groups difference? Is there a better way to assess this difference using Bayesian tools?

2

  1. brms currently does not have a family for underdispered count data. You may check model fit graphically via the pp_check method. Convergence does not tell you anything about whether the distribution is reasonable for your data.

  2. Can you tell me why you think the results you obtain are inconsitent with each other? Perhaps the “Details” section of ?hypothesis

3

Professor Paul Buerkner,
Thank you very much for your reply.

  1. While the CMP family is not available in the brms package, it would be nice to have the “gamma count distribution” or the “generalized Poisson distribution” in the brms’ set of families.

  2. I don’t think the results are inconsistent; I just think that one (IC) provides a weak evidence of the between-groups difference (equivalent to a marginally significant difference), and the other (ER) provides a strong evidence about the difference. I will read more about these concepts.

Thanks a lot!

4

  1. Agreed, but these distributions are not trivial either for different reasons. Once I am happy with one of these distribution (and have a stable Stan implementation), I will surely implement them in brms.

  2. You shouldn’t use the “Bayes factor rules of thumb” for an evidence ratio computed on the basis of X > Y hypothesis as you did in your code. Both provide weak evidence. You see that once you note that one is a two-sided “test” and the other is a one sided “test” and that an ER of 19 corresponds to a posterior probablility of 95% for the test hypothesis.

5

I understand.
Once again, thank you very much for your valuable help!