Plot stacked / averaged models

Please also provide the following information in addition to your question:

  • Operating System: Win 10
  • brms Version: 2.10

Hi,

I’m looking to plot / predict from stacked or averaged models in brms. I’ve computed model weights and two models have approximately similar model weights. My understanding is that in this situation, I should stack these models to take into account the uncertainty about the “preferred” model. The problem I’m having is, how do I go from getting the model weights (e.g., m1 = .54, m2, .46) to plotting / prediction of the stacked model. Any help on this would be super helpful; I’m new to stacking / model averaging so apologies if my questions don’t quite make sense.

Zach

1 Like

Hi! Question makes perfect sense. I looked into this some time ago, and it’s surprisingly straightforward.

Sample predictions from the models in proportion to the model weights (set the number of samples equal to some number multiplied by the model weight), and simply combine the draws to get the predictive distribution of the stacked models.

1 Like

I see. I was thinking I could do something like this but wasn’t totally sure.

Brms has a method for this task already. I think it is called pp_average.

2 Likes

I know what you mean - it kind of feels too simple.

I was playing with ppaverage today but wasn’t fully understanding the samples (?) it was returning.

Is it accurate to assume that if you use pp_average you doing Bayesian Model Averaging or does it just depend on if you specify weights yourself after, for example, using loo_model_weights (method = “stacking”)?

Or maybe you can use control in pp_average to do stacking, pseudobma etc?

This depends on the type of weights you use. For standard BMA you need to use weights = “marglik” but I don’t actually recommend using it. Rather go for the default loo2.