Brms::loo(moment_match = T) does not appear to work

General
,
1

Hi,

I am attempting to run loo with moment_match = T in order to hopefully reduce the number of high pareto k values resulting from using default loo (moment_match = F). I would eventually like to compare models after moment matching via elpd_loo.

My model consists of 5 categories as my response variable that are proportions summing to 1 (family = “dirichlet”). I am using a single continuous predictor and 2 random variables. I set save_pars = save_pars(all = TRUE) (done using brms::brm, chains = 4, iter = 8000).

When I run brms::loo(my model) the results are:

Computed from 16000 by 709 log-likelihood matrix

         Estimate    SE
elpd_loo   7799.6 118.8
p_loo       554.7  19.7
looic    -15599.2 237.6
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     132   18.6%   759       
 (0.5, 0.7]   (ok)       286   40.3%   218       
   (0.7, 1]   (bad)      262   37.0%   19        
   (1, Inf)   (very bad)  29    4.1%   8         
See help('pareto-k-diagnostic') for details.

When I run brms::loo(my model, moment_match = T), I get the same exact output. The same goes for when I try reloo = T.

I’ve read other posts that pointed out an issue with this argument but they were from 2020 and they appeared to be resolved (as best I could tell).

Any help or insight would be greatly appreciated.

I am using R 4.0.3, Rstudio 1.4.1106, brms 2.15.0, rstan 2.21.2, and Windows 10

3 Likes
2

Hi,
just to rule out an additional cause of problems - what is your version of the loo package? Does the problem persist after upgrading to lastest loo (2.4.1) and brms (2.15.7)?

3

Thank you for the suggestion. I updated brms to 2.16.1 and made sure loo was up to date.

No effect unfortunately.

I am wondering if multi-categorical response variables (> 2) are not supported? I have some logistic regression models I was going to try but none of them have poor pareto K values.

If it helps, my process is as follows:

  1. Run the model
  2. Use brms::add_criterion to add both loo and waic
  3. Use rstan/brms/loo::loo(moment_match = T)

Thanks for your help

1 Like
4

That might actually be part of the issue. My impression (didn’t test) is that brms::add_criterion makes the fit cache the loo results (which then could be returned by brms::loo). So I think that if you have already ran add_criterion once, you would need to call add_criterion(fit, "loo", overwrite = TRUE, moment_match = TRUE) to force update the stored values with the ones from moment matching.

Would that help?

5

I tried but I got the following error:

Error in private:public : argument of length 0

I am running more models today. I will set moment_match = T in my second step, to avoid the potential need to overwrite, and see if that helps.

6

Update.

I ran a model and then used brms::add_criterion(fit, “loo”, moment_match = T, cores = 4).

This produced the following errors:

Error in checkForRemoteErrors(val) :
4 nodes produced errors; first error: could not find function “bind”
Error: Moment matching failed. Perhaps you did not set ‘save_pars = save_pars(all = TRUE)’ when fitting your model?

For the first error, I coded the bind function for my model as
bind ← function(…) cbind(…)
which I found here: Dirichlet regresion using brms - #3 by Jannik

I also found the suggestion to code bind as
bind ← cbind
from here: bind() fails with long variables names · Issue #1016 · paul-buerkner/brms · GitHub

Both codings produced the error. The function “bind” sits in my global environment so maybe add_criterion can’t find it there?

As for the second error, here is my model specification:

fit ← brms::brm(bind(cat1, cat2, cat3, cat4, cat5) ~ p1+ p2 + p3 + (1|g1) + (1|g2),
data = data, seed = 1234, control = list(max_treedepth = 15, adapt_delta = 0.99),
iter = 8000, chains = 4, cores = 4, family = “dirichlet”, save_pars = save_pars(all =
TRUE)))

It is odd that the error is asking if I set my save_pars to all = TRUE considering this issue was addressed and fixed here: Error with loo_moment_match - #18 by mattwilliamson13

Any other ideas or suggestions?

7

I have hit exactly the same problem on a number of occasions too. I had thought it was fixed at one stage but then ran into the same issue again. I think I remember @paul.buerkner saying (somewhere on this forum?) this may be a specific problem related to macOS.

8

Wanted to debug this and it turns out moment matching crashes my R session (on Windows 10, brms 2.15.7 and loo 2.4.1; the same result with a version freshly installed from GitHub) - here’s some reproducible code:

library(brms)
options(brms.backend = "rstan", mc.cores = 4)

# Reliably results in a fit with large k-hats
set.seed(1566845)
dd <- data.frame(y = rnorm(10), id = 1:10)
fit <- brm(y ~ (1|id), data = dd)

# The following line crashes R session
fit <- add_criterion(fit, "loo", moment_match = TRUE)
9

@martinmodrak what would you recommend as a next step to hopefully resolve this? I am rather new to forum so I am not aware of how the process works.

Thank you for your help

10

I would try to make as simple example of the problem as you can and file it as issue on the GitHub repo for brms…

You can obviously try to debug this yourself and see if there is workaround - the source for brms and loo is open, so it is possible, if you are comfortable debugging R code.

11

Unfortunately I have never debugged before.

I opened an issue on GitHub here:

1 Like
12

I saw your posts about crashing with moment_match = T and was wondering if you ever resolved this. RStudio is crashing on my Mac when I try this on a poisson glmm. I haven’t tried digging into it too much yet though.

13

Nathan, apologies for the late response. I was never able to resolve the issue. I ended up going with cross validation instead of using simple ELPD as my comparison metric.

14

I have been having this issue of R studio crashing with moment_match=T for a while and recently found a solution in the help for loo_moment_match which says:

“The moment matching algorithm requires draws of all variables defined in Stan’s parameters block to be saved. Otherwise loo_moment_match cannot be computed. Thus, please set save_pars = save_pars(all = TRUE) in the call to brm, if you are planning to apply loo_moment_match to your models.”

So I

  1. re-ran my original brm model with the arguement save_pars = save_pars(all = TRUE)
  2. ran loo on just the model, saving that output. i.e. loo_output<-loo(model)
  3. ran loo_moment_match(model, loo_output)

and it worked no crashing!! Hope this helps :)

2 Likes