Getting BFs for specific contrasts in categorical or ordinal models

Modeling
, ,
1

I am trying to use the BayestestR package to obtain BFs for pairwise contrasts; this typically works very well with linear models, but seems to not work (at least with my coding skills) for categorical or ordinal models in brms.

I use primarily for its describe_posterior which provides BF for all emmeans contrasts I specify. However, I have tried doing so with a categorical model computed in brms, and it does not seem to take the dpar argument when trying to run the sampling.

This is part of my code which may illustrate my point:

#model (with a categorical -1, 0, 1 response); Condition has two levels, and Original has 3.
m1.bias <- brm(Bias ~ Condition*Original + (1| Participant),
data = SEacc,
family = categorical(),
warmup = 2000,
iter = 40000,
save_pars = save_pars(all = TRUE),
prior = my_bias_priors,
control = list(adapt_delta = 0.9))

#the contrasts I care about - works for getting what i want w/o BFs
pw.bias.profile <- emmeans(m1.bias, specs = pairwise ~Original, point.est = median, level = .89, dpar = "mu0")

#the code I am failing to run for the BFs
describe_posterior(pw.bias.profile,
estimate = "median", dispersion = TRUE,
ci = .89, ci_method = "hdi",
test = c("bayesfactor"),
bf_prior = m1.bias,
dpar= "mu0")

#the error
Sampling priors, please wait...
Error : The select parameter is not predicted by a linear formula. Use the 'dpar' and 'nlpar' arguments to select the parameter for which marginal means should be computed.
Predicted distributional parameters are: 'mu0', 'mu1'
Predicted non-linear parameters are: ''
Error in emmeans::ref_grid(prior) :
Perhaps a 'data' or 'params' argument is needed

So I am stuck. Any help would be greatly appreciated.

1 Like
2

It is quite possible that either bayestestr or emmeans don’t have support for those models and/or there is a bug somewhere. brms lets you do a lot of stuff and as far as I know at least emmeans did not very recently support all features of brms models (not clear if this should or should not be supported).

I’ll tag @DominiqueMakowski and @rvlenth as authors of the packages who might know better whether this is a bug or what to do about this - hope they have time to answer.

My understanding of Bayes factors is not very deep, so take the following with a grain of salt: I am somewhat suspicious of using Bayes factors in a categorical model, as the results would very likely depend on your choice of reference category and it seems harder than usual to find justifiable priors for the coefficients. Also the way you build the Bayes factor seems to imply that you will compare model with the main effect for Question for both odds ratios against a model that has Question predictor only for a single odds ratio, which seems strange. Maybe you actually want to compare a model without the Question term for odds ratios against a model that doesn’t have the predictor at all? Also, are you sure the interaction Question:Original is handled correctly?

Best of luck with your model!

3

Tagging @mattansb the BF master

4

Thank you, again. And sorry for the cross-posting with the GitHub forum (Bayes factors per contrast from emmeans for categorical brms model error · Issue #316 · easystats/performance · GitHub). I haven’t had much luck with posting things on the stan forum.

5

@mz555

There are a few things things here which might be getting in the way:

  1. For Bayes factors, the model cannot contain any flat priors (which is the default in brms for fixed effects).
  2. The current CRAN version of brms does not have full emmeans support, but this should be resolved soon (and is currently available on the GitHub version).
  3. I suggest the following workflow for brms + emmeans + Bayes factors:
# 1. Do it all with the posteriors:
## Fit model with non-flat priors
m <- brm(...)

## Get the contrasts:
em_ <- emmeans::emmeans(m, ...)
c_ <- emmeans::contrast(em_, ...)

# 2. Do it all with the priors:
## Get the "prior" model:
m_prior <- bayestestR::unupdate(m)

## Get the ***same*** contrasts with the prior model:
em_prior <- emmeans::emmeans(m_prior, ...)
c_prior <- emmeans::contrast(em_prior, ...)

# 3. Get Bayes factors:
bayestestR::bayesfactor_parameters(posterior = c_, prior = c_prior)

Let me know if this helps!

2 Likes
6

Hi @mattansb , thanks for looking into this.

I am using the current github version of brms (2.15.0), and did specify weakly informative priors for all my coefficients (here, n(0,5)).

I tried now the code you mention, and I get the same error I described when i try:

em_ <- emmeans::emmeans(m1.bias)
Error : The select parameter is not predicted by a linear formula. Use the 'dpar' and 'nlpar' arguments to select the parameter for which marginal means should be computed.
Predicted distributional parameters are: 'mu0', 'mu1'
Predicted non-linear parameters are: ''
Error in (function (object, at, cov.reduce = mean, cov.keep = get_emm_option("cov.keep"),  : 
  Perhaps a 'data' or 'params' argument is needed

So it seems the issue is with emmeans and not bayestestR
However, when I simply want to get estimates (not BFs) from emmeans, I can if I specify the code as I mentioned in my first post:

pw.bias.profile <- emmeans(m1.bias, specs = pairwise ~Original, point.est = median, level = .89, dpar = "mu0")

and

pw.bias.profile2  <- emmeans(m1.bias, specs = pairwise ~Original, point.est = median, level = .89, dpar = "mu1")

But the argument seems to be ignored in bayestestR’s describe_posterior() function.
(a small aside, this code works when I have ordinal data, as I tried on a cumulative probit model with some likert-type responses, and I receive no issues)

This is as far as my R abilities take me…
I look forward to any insights.

1 Like
7

I […] did specify weakly informative priors for all my coefficients (here, n(0,5)).

Note that Bayes factors are quite meaningless with weakly informed priors.

Also keep in mind that for ordinal models there is only partial emmeans support in brms.

Without a reproducible example, I’m not quite sure where things went wrong… So it’s not clear what you mean by:

But the argument seems to be ignored in bayestestR’s describe_posterior() function.

Does this note work?

# Descriptives
pw.bias.profile <- emmeans(m1.bias, specs = pairwise ~Original, point.est = median, level = .89, dpar = "mu0")
pw.bias.profile2  <- emmeans(m1.bias, specs = pairwise ~Original, point.est = median, level = .89, dpar = "mu1")

bayestestR::describe_posterior(pw.bias.profile)
bayestestR::describe_posterior(pw.bias.profile2)

m1.bias_prior <- bayestestR::unupdate(m1.bias)
pw.bias.profile_prior <- emmeans(m1.bias_prior, specs = pairwise ~Original, point.est = median, level = .89, dpar = "mu0")
pw.bias.profile2_prior  <- emmeans(m1.bias_prior, specs = pairwise ~Original, point.est = median, level = .89, dpar = "mu1")

bayestestR::bayesfactor_parameters(pw.bias.profile, pw.bias.profile_prior)
bayestestR::bayesfactor_parameters(pw.bias.profile2, pw.bias.profile2_prior)
1 Like
8

It worked!
using the unupdate() and the bayesfactor_parameters() worked to provide BFs.

Thank you for the assistance!

9

Awesome!

Though I must emphasis that these BFs are not indicative of anything and probably “over support” the null due to the wide priors.

3 Likes