Comparing complex multivariate models with loo_compare

Interfaces brms
1

I have run two complex multivariate models following this tutorial: Estimating Multivariate Models with brms. The models follow this format:

f1 = as.formula('y1~1+var1+var2+var3+(1+var1+var2+var3|GroupID)')
f2 = as.formula('y2~1+var1+var2+var3+(1+var1+var2+var3|GroupID)')
f3 = as.formula('y1~1+var1+var2+(1+var1+var2+|GroupID)')
f4 = as.formula('y2~1+var1+var2+(1+var1+var2+|GroupID)')
model_of_interest = brm(f1+f2,data=df,iter=6000,chains=4,cores=4,family="gaussian",control=list(adapt_delta=0.9),save_all_pars=TRUE)
null_model = brm(f3+f4,data=df,iter=6000,chains=4,cores=4,family="gaussian",control=list(adapt_delta=0.9),save_all_pars=TRUE)

and I now want to compute loo() for each and compare using loo_compare(). Using this tutorial as a guide Using the loo package (version >= 2.0.0) ā€¢ loo, I compute loo as follows:

loo1 <- loo(model_of_interest, moment_match=TRUE)
loo2 <- loo(null_model, moment_match=TRUE)

However, before I can enter this into loo_compare() I get this error for both loo1 and loo2:

Error in .update_pars(x, upars = upars, ā€¦) :
length(new_samples) == nrow(pars) is not TRUE
Error: Moment matching failed. Perhaps you did not set ā€˜save_all_parsā€™ to TRUE when fitting your model?

I am confused because I have definitely set save_all_pars=TRUE in both model fits, and I canā€™t seem to find helpful guidance online about how to solve this. I am using R version 4.0.2. I appreciate any advice, thanks very much.

1 Like
2

Hmm, this seems to be an error coming from brms and not loo itself (although it could be a loo error that brms is just catching). @paul.buerkner any ideas here? It sounds like save_all_pars is TRUE but they still get the error.

3

The error message just gives a suggestions that ā€œperhapsā€ this is related to ā€œsave_all_parsā€ (because this is the most common problem), but in this case it is not. This is an error that occurs with some 3.x versions of R but should be gone in R 4.x

1 Like
4

I have the same error with a distributional model (lognormal family with estimated sigma) but Iā€˜m not able to upgrade R to 4.x (older Mac which doesnā€˜t run Catalina)
Is there anything I could do?

5

Can you provide a minimal reproducible example?

6

The error happens when I truncate the outcome variable, which is also true for my original modelā€¦

model.test <-
  brm(
      bf(hp | trunc(lb = 52, ub = 335) ~ mpg + (1 | cyl), sigma ~ (1 | cyl)),
      family = lognormal(),
      data = mtcars,
      save_all_pars = TRUE
  )
model.test <-
  add_criterion(
    model.test,
    criterion = "loo",
    moment_match = TRUE
)

Error in validate_ll(log_ratios) : All input values must be finite.
Fehler: Moment matching failed. Perhaps you did not set ā€˜save_all_parsā€™ to TRUE when fitting your model?

7

In your case, there seems to be a numerical problem with the truncated lognormal distribution that is unlikely to be related to moment matching directly.

8

Could it be that the error was fixed by https://github.com/stan-dev/loo/pull/153?

@Pascal_J: if you do traceback() right after you get the error (running on 1 core), you may compare that with what I reported in https://github.com/stan-dev/rstan/issues/852.

9

This is the result of traceback():

31: stop(..., call. = FALSE)
30: stop2("Moment matching failed. Perhaps you did not set ", "'save_all_pars' to TRUE when fitting your model?")
29: loo_moment_match.brmsfit(x = .x1, loo = .x2, newdata = .x3, resp = .x4, 
        k_threshold = .x5, check = .x6)
28: loo_moment_match(x = .x1, loo = .x2, newdata = .x3, resp = .x4, 
        k_threshold = .x5, check = .x6)
27: eval(expr, envir, ...)
26: eval(expr, envir, ...)
25: eval2(call, envir = args, enclos = parent.frame())
24: do_call("loo_moment_match", moment_match_args)
23: .loo(x = .x1, newdata = .x2, resp = .x3, model_name = .x4, pointwise = .x5, 
        k_threshold = .x6, moment_match = .x7, reloo = .x8, moment_match_args = .x9, 
        reloo_args = .x10)
22: eval(expr, envir, ...)
21: eval(expr, envir, ...)
20: eval2(call, envir = args, enclos = parent.frame())
19: do_call(paste0(".", criterion), args)
18: .fun(criterion = .x1, pointwise = .x2, resp = .x3, k_threshold = .x4, 
        moment_match = .x5, reloo = .x6, moment_match_args = .x7, 
        reloo_args = .x8, x = .x9, model_name = .x10, use_stored = .x11)
17: eval(expr, envir, ...)
16: eval(expr, envir, ...)
15: eval2(call, envir = args, enclos = parent.frame())
14: do_call(compute_loo, args)
13: .fun(models = .x1, criterion = .x2, pointwise = .x3, compare = .x4, 
        resp = .x5, k_threshold = .x6, moment_match = .x7, reloo = .x8, 
        moment_match_args = .x9, reloo_args = .x10)
12: eval(expr, envir, ...)
11: eval(expr, envir, ...)
10: eval2(call, envir = args, enclos = parent.frame())
9: do_call(compute_loolist, args)
8: loo.brmsfit(.x1, moment_match = .x2, model_names = .x3)
7: loo(.x1, moment_match = .x2, model_names = .x3)
6: eval(expr, envir, ...)
5: eval(expr, envir, ...)
4: eval2(call, envir = args, enclos = parent.frame())
3: do_call(fun, args)
2: add_criterion.brmsfit(model.test, criterion = "loo", moment_match = TRUE)
1: add_criterion(model.test, criterion = "loo", moment_match = TRUE)
10

Updating to the master branch of stan-dev/loo had no effect.

11

Yes, the traceback looks quite different (it also doesnā€™t show the failure in validate_ll that made me think of that possibility). I had put some breakpoints using debug in inner functions such as loo_moment_match, then stepped line by line to see what the data looked like right before the failure.

12

Following up on the initial problem, I have managed to run brms::loo_subsample() instead of loo() without errors, but I receive lots of pareto k diagnostic warnings. I have tried to run brms::loo_moment_match() on the loo_subsample() result as follows:

loo1 <- brms::loo_subsample(fit1)
loo1_updated <- brms::loo_moment_match(fit1,loo1)

but I receive this very confusing error:

Error in prep_call_sampler(object) :
the compiled object from C++ code for this model is invalid, possible reasons:

  • compiled with save_dso=FALSE;
  • compiled on a different platform;
  • does not exist (created from reading csv files).

Does brms::loo_subsample not work with brms::loo_moment_match? Please let me know if I should start a new topic, Iā€™m not sure if this is related to the above issue. I have updated to the version of latest version of the loo package (loo_2.3.1.9000) via github.

13

Perhaps @topipa has some ideas for both this issue and the one that @Pascal_J described a bit further up.

14

Update: this issue has been solved here

3 Likes