I am attempting to run loo with moment_match = T in order to hopefully reduce the number of high pareto k values resulting from using default loo (moment_match = F). I would eventually like to compare models after moment matching via elpd_loo.

My model consists of 5 categories as my response variable that are proportions summing to 1 (family = “dirichlet”). I am using a single continuous predictor and 2 random variables. I set save_pars = save_pars(all = TRUE) (done using brms::brm, chains = 4, iter = 8000).

When I run brms::loo(my model) the results are:

Computed from 16000 by 709 log-likelihood matrix
Estimate SE
elpd_loo 7799.6 118.8
p_loo 554.7 19.7
looic -15599.2 237.6
------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 132 18.6% 759
(0.5, 0.7] (ok) 286 40.3% 218
(0.7, 1] (bad) 262 37.0% 19
(1, Inf) (very bad) 29 4.1% 8
See help('pareto-k-diagnostic') for details.

When I run brms::loo(my model, moment_match = T), I get the same exact output. The same goes for when I try reloo = T.

I’ve read other posts that pointed out an issue with this argument but they were from 2020 and they appeared to be resolved (as best I could tell).

Any help or insight would be greatly appreciated.

I am using R 4.0.3, Rstudio 1.4.1106, brms 2.15.0, rstan 2.21.2, and Windows 10

Hi,
just to rule out an additional cause of problems - what is your version of the loo package? Does the problem persist after upgrading to lastest loo (2.4.1) and brms (2.15.7)?

Thank you for the suggestion. I updated brms to 2.16.1 and made sure loo was up to date.

No effect unfortunately.

I am wondering if multi-categorical response variables (> 2) are not supported? I have some logistic regression models I was going to try but none of them have poor pareto K values.

That might actually be part of the issue. My impression (didn’t test) is that brms::add_criterion makes the fit cache the loo results (which then could be returned by brms::loo). So I think that if you have already ran add_criterion once, you would need to call add_criterion(fit, "loo", overwrite = TRUE, moment_match = TRUE) to force update the stored values with the ones from moment matching.

I ran a model and then used brms::add_criterion(fit, “loo”, moment_match = T, cores = 4).

This produced the following errors:

Error in checkForRemoteErrors(val) :
4 nodes produced errors; first error: could not find function “bind”
Error: Moment matching failed. Perhaps you did not set ‘save_pars = save_pars(all = TRUE)’ when fitting your model?

I have hit exactly the same problem on a number of occasions too. I had thought it was fixed at one stage but then ran into the same issue again. I think I remember @paul.buerkner saying (somewhere on this forum?) this may be a specific problem related to macOS.

Wanted to debug this and it turns out moment matching crashes my R session (on Windows 10, brms 2.15.7 and loo 2.4.1; the same result with a version freshly installed from GitHub) - here’s some reproducible code:

library(brms)
options(brms.backend = "rstan", mc.cores = 4)
# Reliably results in a fit with large k-hats
set.seed(1566845)
dd <- data.frame(y = rnorm(10), id = 1:10)
fit <- brm(y ~ (1|id), data = dd)
# The following line crashes R session
fit <- add_criterion(fit, "loo", moment_match = TRUE)

I would try to make as simple example of the problem as you can and file it as issue on the GitHub repo for brms…

You can obviously try to debug this yourself and see if there is workaround - the source for brms and loo is open, so it is possible, if you are comfortable debugging R code.

I saw your posts about crashing with moment_match = T and was wondering if you ever resolved this. RStudio is crashing on my Mac when I try this on a poisson glmm. I haven’t tried digging into it too much yet though.

Nathan, apologies for the late response. I was never able to resolve the issue. I ended up going with cross validation instead of using simple ELPD as my comparison metric.

I have been having this issue of R studio crashing with moment_match=T for a while and recently found a solution in the help for loo_moment_match which says:

“The moment matching algorithm requires draws of all variables defined in Stan’s parameters block to be saved. Otherwise loo_moment_match cannot be computed. Thus, please set save_pars = save_pars(all = TRUE) in the call to brm, if you are planning to apply loo_moment_match to your models.”

So I

re-ran my original brm model with the arguement save_pars = save_pars(all = TRUE)

ran loo on just the model, saving that output. i.e. loo_output<-loo(model)