Modelling two mutually informative decisions

Interfaces brms
, ,
1

I’m trying to model experimental data in which participants make two dichotomous forced choice decisions. One of the hypotheses is that each decision (and how the other predictors affect it) depends on the other in a certain way. I’m new to brm and Baysian modelling, so apologies if this is either trivial or confused. My first question is whether the following kind of model specification is legitimate:

decision1_fm <- bf( decision1 ~
					  otherPredictors*decision2 +
                      (1|p|participant)) +
  bernoulli()

decision2_fm <- bf(decision2 ~
                    otherPredictors*decision1 + 
                    (1|p|participant)) +
  bernoulli()

joint.md <- brm(
  decision1_fm + 
  decision2_fm + 
  set_rescor(FALSE),
  data = data,
  chains = 4,
  cores = 4)

The model seems to work and the predictions of the model make sense, but I’m just not sure whether this is a legitimate way of modeling given the mutual co-dependence of the two response variables.

If this is in principle a viable model, then I have two questions about how to interpret and evaluate the model:

  • When I assess whether each decision affects the other based on the post_samples, it seems that there is strong evidence (with a weakly informative skeptical prior) that the relevant coefficients (the main effect of one response on the other, as well as some of the interaction terms with the otherPredictors) are different from zero, so the decisions do seem to mutually depend on each. This is compatible with the fact that when I do separate traditional logistic mixed effects regression on each decision: the main effect of the other response and some of the interaction terms with the predictors come out significant.

However, when try to check whether the decisions affect each other simply by a model comparison between joint.md and a simpler model that lacks the decision1/decision2 predictor terms in the respective formulas (using loo_compare based on WAIC criterion), the comparison suggests that the more complex model is not better at capturing the data.

So I’m not sure what to make of this–should I conclude that the decisions affect each other (based on post_samples), or that they do not (based on model comparison)?

  • I would also like to compute predictions for the likelihood of pairs of decisions given values for the predictors, but I’m not sure how. I can compute predictions with predict() for decision2 given decision1 and vice versa, but I don’t know how to compute predictions for pairs of decisions based just based on the otherPredictors. Can I somehow compute such predictions directly, or do I have to compute such predictions about pairs of decisions based on the conditional predictions for each of the responses? If the latter, how do I do this?

Any leads would be appreciated (or examples where someone has done something similar)

  • Operating System: macOs 2.15.4
  • brms Version: 2.11.1
1 Like
2

Hi and welcome! I am wondering why you have set_rescor(FALSE) for the model?

Usually (coming from ecology) when I have tow things that likely depend on each other I set my model to set_rescor(TRUE). Since this is outside of my discipline I could be way off about how your model is working.

3

I may be wrong but I think it has to be set to false because it’s a logistic model (and it returns an error if I set it to true)

2 Likes
4

I think using one decision to predict the other is slightly weird. It makes the model non-generative (i.e. that you can’t simulate data from it), which is then source of your additional problems with prediction. Nevertheless I don’t think there is “one best way” to model this. I’ve seen people handle this by moving to long format (i.e. so you know have columns decision and decision_id (a factor taking values “1” and “2”) and each row in the original dataset is know two rows). Then you would use formula decision ~ (otherPredictors | decision_id) + (1 | participant) where the correlation terms for the otherPredictors | decision_id can be interpreted as a dependence between the two decisions.

You can play with that making only some effects/intercepts depend on decision_id or even add a varying intercept decision_id | row_id_in_the_original_datset.

Alternatively, you might have some more theory to guide a more precise form of the depenedency.

I am guessing here, but I think it is possible that the other terms in your model are flexible enough to capture the variability (despite the other decision being a good predictor). Bernoulli response provides very limited information. You probably should run posterior predictive checks to see if any of the models has problems capturing some aspect of the data.

Hope that helps!

1 Like
5

Thanks, Martin, this is incredibly helpful! I understand now better what was weird about how I tried to model this, and why I couldn’t get the predictions I wanted. I am playing with your alternative way of modelling this, and I think it’ll take a me a while to understand how to interpret the correlation terms, but in principle I can see where this is going.

Just to clarify, apart from the joint model, I am also looking at (i) a model in which both decisions are independently of each other, and (ii) models in which one is made independently, and the other one is based on the first (decision1 before decision2 or vice versa). Just looking at these options, the empirical data looks more like one of the latter ordered hypotheses is correct. I guess what I tried to do with the above model is to avoid deciding in the model itself that one necessarily precedes the other—just because it’s not clear to me why perception should necessarily proceed one way or the other. It seems to me that this must be a recurring issue not just in speech perception (two in principle orthogonal decisions are made based no overlapping cues), but I haven’t found any discussion about how to model this best. This general issue is not a question about brms of course, I just thought there might be a way to do this with it. Anyway, I will explore your alternative of coding the model more.

Thanks also for the comment about the model comparison, that makes a lot of sense to me!

1 Like