Ppc_loo_pit_overlay

I am try to see if my model is valid. I ran a beta glmm in brms.

brm.formula.1 <- bf(
  prop ~ 0 + group + 
    # Group-specific independent random intercepts and slopes
    (1 | gr(ID, by = group)) + 
    # OU correlation structure
    gp(time,
       by = ID,
       k = 10, # optional number of basis functions for computing Hilbert-space approximate GPs
       cov = "exponential", # Ornstein–Uhlenbeck
       iso = TRUE, # (default)
       gr = TRUE, # (default)
       cmc = TRUE # (default) 
    ), 
  phi ~ 0 + group + group:x1
)

# fit model
fit.brms.ou.1 <- brm(formula = brm.formula.1,
                   data = ddd,
                   #prior = prior,
                   family = Beta(link = "logit"),
                   chains = 4,
                   iter = 8000,
                   thin = 2,
                   warmup = 4000,
                   cores = 4,
                   control = list(
                     adapt_delta = 0.95,
                     max_treedepth = 14
                   ),
                   backend = "cmdstanr"
                   
)

I did this model with and without the ou process. then ran a loo_compare
image

I then check the density overlay

ppc_loo_overlay

ppc_loo_pit_overlay(
    y = y_obs,
    yrep = y_rep,
    psis_object = loo.1$psis_object,
    samples = 100
  )


Which this shows that it fits into the other simulated unifrom, but there is a frowny face

I then included a spline

brm.formula.1 <- bf(
  prop ~ 0 + group+
    # Group-specific independent random intercepts and slopes
    (1 | gr(ID, by = group)) +
    gp(time,
       by = ID,
       k = 10, # optional number of basis functions for computing Hilbert-space approximate GPs
       cov = "exponential", # Ornstein–Uhlenbeck
       iso = TRUE, # (default)
       gr = TRUE, # (default)
       cmc = TRUE # (default) 
    ), 
  phi ~ 0 + group + s(x1, by = group, k = 5)
)

and my ppc_loo_overlay looks better

So overall this seems good to me, but Im not entirely sure.

Thanks

@jsocolar

If you want more certainty, I recommend to use new version of ppc_loo_pit_ecdf plot with argument method = "correlated" as demonstrated in Predictive model checking and comparison: Clinical trial – Bayesian Workflow book. The relevant paper is [2603.02928] LOO-PIT predictive model checking. The new plot is available in the github version of bayesplot which you can install with

remotes::install_github("stan-dev/bayesplot", dependencies = TRUE, build_vignettes = FALSE)

@avehtari ,
Thanks for these resources.

Ok so I did what you recommended and it looks like its ok, as its higher than the alpha.

Yes, looks good!

Amazing, thanks @avehtari

Also, I want to mention that the paper @avehtari provided was extremely helpful. I would highly recommend it to anyone using LOO-PIT for model validation.