A few months ago, I ran two brms models (called fit_sc2 and fit_sc4) and was able to compare them using the loo_compare(fit_sc2, fit_sc4, criterion = “loo”) function. Before being able to compare them, I had to use fit_sc ← add_criterion(fit_sc, “loo”, moment_match=TRUE) for the two models. Everything worked fine then.
Today I tried to recompare the two models (that I had previously saved as .RData files) but I now get an error messages when I run the function fit_sc ← add_criterion(fit_sc, “loo”, moment_match=TRUE):
Error in object@.MISC$stan_fit_instance$unconstrain_pars(pars) :
Exception: Variable Intercept missing (in 'modele046d6d6d70_ef57d33fa559ae9e20907c96b8a7d67d' at line 83)
Error: Moment matching failed. Perhaps you did not set 'save_all_pars' to TRUE when fitting your model?
Any idea where this problem might come from? I haven’t changed the code since the first time I ran it (and I did not have any save_all_pars parameter). The model is an ordinal regression.
There are often bumps saving models to disk and reading them back. If you need this analysis, then I think just run the fits again. This process can be a bit sketchy too – like if you upgrade your brms version I don’t think there’s any guarantee you can load old models. Instead of saving the models, can you extract the log likelihoods or whatever you’d need to do your calculations in the future?
If you need to be able to save models to disk, then I think put together a minimum R script that shows the problem and then we can work from there. But I don’t think serializing models is gonna be a long term solution regardless.
Thanks for your answer. Actually I have rerun the models but I still got the same error message. I followed the suggestion and added save_pars = save_pars(all = TRUE). But now I get a different error though. What surprises me is that I did not get these error messages the previous times I ran this model.
Here is the model (the Response variable is a 7 point Likert scale):
fit_sc2 <- brm(
formula = Response ~ 1 + Var1*Var2*Var3 + (Var1*Var2|Participant) + (Var3|Word),
data = datafile.for.model,
family = cumulative("probit"),
save_pars = save_pars(all = TRUE),
iter = 4000,
inits = 0,
set.seed(1802))
fit_sc2 <- add_criterion(fit_sc2, "loo",
moment_match=TRUE)
The model converges. The pb only arises with add_criterion().Here is the error message:
Error in validate_ll(log_ratios) : All input values must be finite. Erreur : Moment matching failed. Perhaps you did not set 'save_all_pars' to TRUE when fitting your model?
Any idea what is going wrong? Thanks!
R version 4.0.3 (2020-10-10)
I have a tried on Ubuntu 18.04 and MacOS Catalina 10.15.7
Ok, when I inspect the output of the loo() function, I get some further information on the problem:
Warning message: Found 1 observations with a pareto_k > 0.7 in model 'fit_sc2'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations.
I did use moment_match = TRUE in the function add_criterion().
So I also tried this below. According to the documentation, it is supposed to help when Pareto k estimates are large.
(loo1 <- loo(fit_sc2)) (mmloo1 <- loo_moment_match(fit_sc2, loo = loo1))
But I also got an error:
Error in validate_ll(log_ratios) : All input values must be finite. Erreur : Moment matching failed. Perhaps you did not set 'save_all_pars' to TRUE when fitting your model?
I am wondering what the next step is if these different moves did not help. Thanks!
I have changed save_pars = save_pars(all = TRUE) to save_all_pars = TRUE, as you suggested. But I get an additional error message and it did not solve the pb. Here is the message:
Warning message:Argument 'save_all_pars' is deprecated. Please use argument 'all' in function 'save_pars()' instead.
And then when I run fit_sc2 <- add_criterion(fit_sc2, "loo", moment_match=TRUE), I get the same error as before:
Error in validate_ll(log_ratios) : All input values must be finite.Erreur : Moment matching failed. Perhaps you did not set 'save_all_pars' to TRUE when fitting your model?
In case it can be of any help, I am showing you here the output of loo(fit_sc2). It appear that there is just one with a large (bad) pareto k value (>0.7).
Computed from 8000 by 4840 log-likelihood matrix
Estimate SE
elpd_loo -5904.4 72.2
p_loo 281.3 8.4
looic 11808.8 144.4
------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 4830 99.8% 733
(0.5, 0.7] (ok) 9 0.2% 381
(0.7, 1] (bad) 1 0.0% 42
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
I just tried with brms 2.14.8 and loo 2.4.1.9000 from Github (I used devtools to do that, it was my first time installing from Github, I hope I didn’t get something wrong). But I am getting the same error as above. I see that when loo loads there is a suggestion to use as many cores as possible (but I used the default setting of 1). Could it be due to that?
I think it is probaly a numerical problem during moment matching. I dont know if we have a proper fix for it at this point. You could open an issue on loo github and provide a minimal reproducible example. Perhaps we can eventually find a way around it.
I am not sure if this is helpful but apparently (as least in my case) the root cause is due to rstan::log_prob returning -Inf for some iterations. Calling tree was: brms:::loo_moment_match.brmsfit |_ loo:::loo_moment_match.defaut |_ loo:::loo_moment_match_i |_ brms:::log_prob_upars |_ brms:::loo_prob_upars_stanfit |_ rstan::log_prob
This should not be happening. My gut feeling is that this might be due to some niche case where the precision was messed up when casting R object to C.
// Discovered that a lot of gradient values were also Inf/NaN.