Model weighing using kfold

Interfaces brms
1
  • Operating System: Windows 7
  • brms Version: 2.7.1

Dear Stan community,

We are using the model_weights function of the brms package to evaluate a set of multivariate, multilevel models fitted through brms.

We have noted that the result obtained through the model_weights function is relative, and thus varies when some models are withdrawn from the set, or new models are added to it, and then weighed again.

Given (i) the considerable amount of time some particular methods for weighting take (e.g. kfold), (ii) the considerable amount of models being tested (c. 25), and (iii) that most of the models tested in one set are then tested again with the addition of only one new model or the withdrawal of a pair of them:
Is there a way in which we could calculate a certain kfold value for each model, for only then calculating the weights a posteriori?

This way, we would only need to calculate kfold for the new models, if any, and then calculate weights separately, with the previously calculated kfold value, possibly saving an enormous amount of time.

Many thanks in advance,

Alan

1 Like
2

You should be able to do it as discussed in Bayesian Stacking and Pseudo-BMA weights using the loo package. Just change loo to kfold.

1 Like
3

Dear Aki,

Thanks for clarifying on this issue. I’ll calculate an elpd_kfold value for each model and compare them with the function pseudobma_weights(), and again with stacking_weights().

Cheers,

Alan