Loo_model_weights( ..) throws error with leave-one-group-out cv

Interfaces brms
1

I ran leave-one-group out cross validation on two models (md1 nested in md2):

md1LOGO = kfold(model1, group = “Subject”)
md2LOGO = kold(model2,group = “Subject”)

And then run

loo_model_weights(md1LOGO, md2LOGO)

which throws an error:

Error: Each log-likelihood matrix must have the same dimensions.

Could someone please point me to a solution?

Thanks

2

The dimensions of the log-likelihood matrix are number of posterior draws by number of data points, so this error message could mean that the two models either have a different number of data points or were run for a different number of MCMC iterations. I think you can check this by doing:

dim(log_lik(model1))
dim(log_lik(model2))

If you do that do you get the same or different dimensions?

3

Thankyou for answering!

I checked. Both models were run with “iter=10000, chains=2”. Both models are also using the same data.

However, when I run dim(log_lik(model1)) I get the error:

Error in UseMethod("log_lik") : 
  no applicable method for 'log_lik' applied to an object of class "c('kfold', 'loo')"

I’m sorry, but I’ve only ever used rstanarm and brms, never really looked under the hood in stan, so is really difficult for me to search for the errors.

I apologize in advance if (as it probably is…) some mistake I’ve made.

4

No worries at all!

According to that error message it thinks that model1 is the output from kfold() but I thought model1 was the output from brm().

But actually now that I check the documentation for brms::loo_model_weights it looks like the input should be the model objects themselves and not the output from kfold, so I’m not sure if using loo_model_weights after kfold is intended to work. Here’s the Examples section from ?brms::loo_model_weights:

# model with population-level effects only
fit1 <- brm(rating ~ treat + period + carry,
            data = inhaler, family = "gaussian")
# model with an additional varying intercept for subjects
fit2 <- brm(rating ~ treat + period + carry + (1|subject),
            data = inhaler, family = "gaussian")
loo_model_weights(fit1, fit2)
5

Thanks - the issue is that I am trying to run leave-one-group-out cross validation on two nested models and then calculate their stacking weights.

I got somewhat confused by the documentation, because it says:

`` We can use approximate or exact leave-one-out cross-validation (LOO-CV) or K-fold CV to estimate the expected log predictive density (ELPD).

But the function loo_model_weights doesn’t seem to have any parameters that allow you to specify “leave-one-group-out cv” .

I suppose a simpler (I hope) question is, if I have calculated “leave one group out cv” for two nested models:

md1LOGO = kfold(model1, group = “Subject”)
md2LOGO = kold(model2,group = “Subject”)

how do I calculate their stacking weights?

6

Ok I see what you mean. I think we need to make it easier to get stacking weights after kfold cv. For now, can you see if this works?

# assuming md1LOGO and md2LOGO are the output from kfold()
x <- cbind(md1LOGO$pointwise[, "elpd_kfold"], md2LOGO$pointwise[, "elpd_kfold"])
loo::stacking_weights(x)
7

Thanks so much, it works!

1 Like