I am attempting to run loo with moment_match = T in order to hopefully reduce the number of high pareto k values resulting from using default loo (moment_match = F). I would eventually like to compare models after moment matching via elpd_loo.
My model consists of 5 categories as my response variable that are proportions summing to 1 (family = “dirichlet”). I am using a single continuous predictor and 2 random variables. I set save_pars = save_pars(all = TRUE) (done using brms::brm, chains = 4, iter = 8000).
When I run brms::loo(my model) the results are:
Computed from 16000 by 709 log-likelihood matrix
elpd_loo 7799.6 118.8
p_loo 554.7 19.7
looic -15599.2 237.6
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 132 18.6% 759
(0.5, 0.7] (ok) 286 40.3% 218
(0.7, 1] (bad) 262 37.0% 19
(1, Inf) (very bad) 29 4.1% 8
See help('pareto-k-diagnostic') for details.
When I run brms::loo(my model, moment_match = T), I get the same exact output. The same goes for when I try reloo = T.
I’ve read other posts that pointed out an issue with this argument but they were from 2020 and they appeared to be resolved (as best I could tell).
Any help or insight would be greatly appreciated.
I am using R 4.0.3, Rstudio 1.4.1106, brms 2.15.0, rstan 2.21.2, and Windows 10
That might actually be part of the issue. My impression (didn’t test) is that brms::add_criterion makes the fit cache the loo results (which then could be returned by brms::loo). So I think that if you have already ran add_criterion once, you would need to call add_criterion(fit, "loo", overwrite = TRUE, moment_match = TRUE) to force update the stored values with the ones from moment matching.
I ran a model and then used brms::add_criterion(fit, “loo”, moment_match = T, cores = 4).
This produced the following errors:
Error in checkForRemoteErrors(val) :
4 nodes produced errors; first error: could not find function “bind”
Error: Moment matching failed. Perhaps you did not set ‘save_pars = save_pars(all = TRUE)’ when fitting your model?
I have hit exactly the same problem on a number of occasions too. I had thought it was fixed at one stage but then ran into the same issue again. I think I remember @paul.buerkner saying (somewhere on this forum?) this may be a specific problem related to macOS.
Wanted to debug this and it turns out moment matching crashes my R session (on Windows 10, brms 2.15.7 and loo 2.4.1; the same result with a version freshly installed from GitHub) - here’s some reproducible code:
options(brms.backend = "rstan", mc.cores = 4)
# Reliably results in a fit with large k-hats
dd <- data.frame(y = rnorm(10), id = 1:10)
fit <- brm(y ~ (1|id), data = dd)
# The following line crashes R session
fit <- add_criterion(fit, "loo", moment_match = TRUE)