Error when adding LOOIC to a zero one inflated beta regression model in brms

Interfaces brms
,
1

Hi all, I’m new to stan and brms, so I apologise if this is a very silly question and I’m missing something simple. Recently, I have needed to run a zero one inflated beta regression for a data set I am working with. The models have been running great, however, I cannot seem to be able to add LOOIC to the models, as I keep getting the following errors:

Error in validate_ll(log_ratios) : NAs not allowed in input.
Error: Moment matching failed. Perhaps you did not set ‘save_pars = save_pars(all = TRUE)’ when fitting your model? If you are running moment matching on another machine than the one used to fit the model, you may need to set recompile = TRUE.

I have made a dummy data set and put some reproducible code below that gives me the same error as with my actual dataset.

OS = macOS ventura13.1
brms version = 2.20.4

# here is a zero one inflated dummy dataset

set.seed(123) 

df <- data.frame(treatment = as.factor(seq(1, 5, by = 1)),
                 hole = as.factor(seq(1, 20, by = 1)),
                 response = rbeta(n = 100, shape1 = 2, shape2 = 5))

prop_zero = 0.08
prop_one = 0.21 

num_zeros = floor(prop_zero * length(df$response))
num_ones = floor(prop_one * length(df$response))

zero_indices = sample(1:length(df$response), num_zeros, replace = FALSE)
df$response[zero_indices] <- 0

one_indices = sample(setdiff(1:length(df$response), zero_indices), num_ones, replace = FALSE)
df$response[one_indices] <- 1

model_formula2 <- bf(
  response ~ treatment + (1|hole),
  phi ~ treatment + (1|hole),
  zoi ~ treatment + (1|hole),
  coi ~ treatment + (1|hole),
  
  family = zero_one_inflated_beta()
)

model2 <- brm(
  model_formula2,
  data = df,
  control = list(adapt_delta = 0.99,
                 max_treedepth = 12),
  chains = 4, iter = 2000, warmup = 500,
  cores = 4, threads = threading(2),
  backend = "cmdstanr",
  seed = 12345,
  save_pars = save_pars(all = TRUE)
)

model1 <- add_criterion(model2, "loo", moment_match = TRUE)
2

I’ve seen similar problem with another model.

This is likely due to the log-density evaluation failing. Moment matching is transforming the full data posterior draws to better match the leave-one-out posterior, and with some of the transformed draws, the log-density evaluation is getting numerical issues due to the limitations in floating point presentation of the values in computers and the code not using numerically more stable computation or checking that in case of failure something more useful is returned. When running Stan sampler, these same problems would cause a rejection of the proposal, and I guess we should modify the moment matching algorithm to also reject bad transformed draws.

Thanks for providing the example we can use for testing.

This is likely to be a problem in loo package, but pinging also @paul.buerkner as the example is made with brms

2 Likes
3

I created an issue describing the problem loo_moment_match fails if new log likelihood is -Inf · Issue #258 · stan-dev/loo · GitHub and a PR to fix it fix loo_moment_matching NaN issue by avehtari · Pull Request #259 · stan-dev/loo · GitHub

2 Likes
4

Since you’re new to Stan and brms, I think it’s worth adding a disclaimer here:

Even when this works properly, you need to interpret loo results from models that contain a mixture of discrete and continuous response components very cautiously. There is no universal formula for figuring out how to weight the log probability masses in the discrete part of the model versus the log probability densities in the continuous part. See Cross-validation FAQ • loo for more.

5

Thank you for this, I wasn’t aware! I really appreciate all the help, thank you both so much.

6

@JackL, the PR has been merged to main, and if you install loo from github

remotes::install_github("stan-dev/loo")

you can try if it works for you now

1 Like