Checking model and LOO with models of different observations

Modeling
, , ,
1

Hi,

I am trying to fit a model to my ordered categorical data of rating (1,2,3) and group (1,2,3). Other predictors are age (18-23, 24-30), continent, and use of medication (y/n). I have been using the helpful tutorial https://journals.sagepub.com/doi/full/10.1177/2515245918823199 to guide me through my first Bayesian regression but I am hoping someone can help explain how to ‘check’ the models?

I ran the following models:
modelall<- brm(formula = rating ~ 1 + cs(group) + medication + continent +agequartiles,
data = df,
family = acat(“cloglog”))

modelagroup<- brm(formula = rating  ~ 1 + cs(group),
data = df,
family = acat("cloglog"))

modelage<- brm(formula = rating  ~ 1 + cs(group)  + agequartiles,
data = df,
family = acat("cloglog"))

modelmeds<- brm(formula = rating  ~ 1 + cs(group)  + medication,
data = df,
family = acat("cloglog"))

As not everyone in my sample consented to answer each question, I cannot seem to do a LOO as they have different numbers of observations. I’m just not sure how to work out what should go in the model, other than looking at whether CI cross 0 or not. How can I check if my model is working?

Thank you

1 Like
2

Hi @GabriellaS-K,
I think there are two ways you could go here:

  1. Only use the subset of observations that answered all questions you are using. It is easy to implement but you throw away parts of your sample.
  2. You try to impute the missing values somehow. There are different ways to do this eg. just setting NAs to the mean values/reference category or modelling them somehow. This could be a good primer for how to do that.
2 Likes
3

Thank you so much that is really helpful! Do you know how I might tell if a model is a ‘good’ model? I am very new to Bayesian stats, and I’m not sure what to look for to report if there are differences between groups

1 Like
4

What a ‘good model’ is depends a lot on what question you want to answer with your model. But there are a few things that are always important:

  1. Check the high level sampling diagnostics in your model summary: Rhat, Bulk_ESS and Tail_ESS. If you run more than one chain, Rhat tells you if they converge. You essentially want this to be 1. The ESS ones tell you the ‘effective sample size’ and if this is very low compared to your iterations, there might be problems during sampling. There are a few handwavy rule-of-thumb ratios such as you want them to be >0.2 of your post-warmup samples.
  2. You can use pp_check(fit) to compare the predictions of your model with the real observations. This is a quick way to check if your model is working well or completely wrong.
  3. You can use conditional_effects(fit) to see the effect parameters have on the outcome scale. This varies one parameter while holding all other predictors on their mean or reference category.
  4. Use something like loo_compare() to compare the out of sample predictions of different models.

So in the end you want to check that the sampling worked reasonably well so that your model is reliable and then check the performance of the model on whatever question you trie to answer.

2 Likes
5

Not a problem, You can either do leave-one-answer-out cross-validation or leave-one-person-out cross-validation, or both.

2 Likes