Model Selection in BRMS

[Hunter24]

@avehtari

@avehtari

I’m curious, why do you change my text you quote to images?

I think I figured out how to do it without images, sorry.

@avehtari

Your p_loos were around 11 to 15, which are much smaller than 226, and thus there is no overfitting.

Ok so I read LOO package glossary — loo-glossary • loo, so since p_loo < N (number of observations) and p_loo < p (parameters of the model), my model is well behaving.

Model Interpretation: My model is well behaving, but since it is not significantly different than my other models because of the large se_diff than there is alot of uncertainty with my model either because of low sample size, uniformed default priors, etc. Is this correct?

How do you determine if p_loo << (much greater) p (parameters)

@avehtari

I also read that if p_loo is lower than than model parameters than it is problematic.

Where did you read that? It can be problematic if p_loo is larger than the number of model parameters. This what is written e.g. in LOO package glossary — loo-glossary • loo

Ok I must have misinterpreted what I saw before.

@avehtari

You can also look at the posteriors of the quantities of interest directly. They can sometimes tell more than loo comparisons.

Are you asking what is posterior or what are the quantities you are interested in?

Yeah I am asking how can the quantities of interest tell me more, for example my male_degree posterior. What would I be looking for?

@avehtari

Can you tell what is shown in the figures? E.g. show the code you used to generate them? Especially the first figure seems to have something wrong. Others look fine.

Figure code:

#First Figure
model<-bf(num|trials(denom)~status*male_degree+s(Date)+(1|mm(male_id,female_id)),phi~male_degree)
My_Model<-brm(model,
                  data = Data,
                  iter = 4000,
                  chains = 5,
                  thin=4,
                  warmup=1000,
                  init = "random",
                  cores = parallel::detectCores(),
                  family =  beta_binomial(link="logit"),
                  control = list(adapt_delta=0.99,max_treedepth=15),
                  set.seed(1945639))

y = Data$num/Data$denom
y_rep=posterior_predict(My_Model)

loo1 <- loo(My_Model, save_psis = TRUE, cores = 4)
psis1 <- loo1$psis_object
lw <- weights(psis1) #normalized log weights

color_scheme_set("orange")
ppc_loo_pit_overlay(y, yrep, lw = lw)

#Other figures:
pp_check(My_Model,type='stat')
pp_check(My_Model,type='hist')
pp_check(My_Model,ndraws=100)

@avehtari