I want to extract the estimated parameters from distributional models via Bayesian model stacking. I am using pp_average to calculate the posterior predictive values averaged across models. But it seems that I can only get the predicted values for y rather than sigma from the stacked model. The simplified model is attached below.
Model 1:
y ~ x1
sigma ~ x1
Model 2:
y ~ x2
sigma ~ x2
pp_average(model1, model2, weights = 'stacking')
I am wondering whether it is also possible to get the predicted values for sigma, and extract the estimated parameters for x1 and x2 from the stacked model in brms. Thanks a lot!
did you try to add dpar = "sigma" to the pp_average call? Not sure if it works at the moment, but I could work in theory at least.
It seems that dpar = "sigma" does not change the dependent variable at all. The predicted values are still for y. I also tried resp = "sigma" from the prepare_predictions function. The results are the same. It seems that pp_average predicts y as default. It would be better to be able to specify the dependent variable.
I am also wondering whether it is possible to extract the estimated parameters for x1 and x2 from the stacked model. There is a function posterior_average, but it only supports for averaging the shared parameters across models, or setting missing values for certain parameters. Ideally, we are expected to get the posterior distributions of each parameter from the stacked model. Any better idea for this? Thanks a lot!
Ok, I now do see the point because pp_average always calls posterior_predict, which of course always creates predictions for y. If you like, please open an issue on github to suggest extending pp_average to work with posterior_epred as well, which will allow, for example, predictions of sigma to be averaged.
Thanks Paul! I just opened a new issue in the github of brms (https://github.com/paul-buerkner/brms/issues/1022).
That’s because the stacking is valid only for the predictions as the observation space is the same for all model, but it’s not valid in general for the parameters as the parameters are likely to have different meaning in different models.
Prediction is usually used only to refer predicting observables and for model parameters you have a posterior distribution.
Technically you can get the posterior draws from each model and then choose the number of draws from each model in proportion of the weights, but are you certain that it is what you need?
I am not quite sure whether the weighted sample from the posterior predictive distributions is what I need. Is it also applicable to combine the estimated parameters for x1 and x2?
We have a series of models including also random effects. My temporary solution is to build a complex model manually by dropping all the variables from models with very small stacking weights (close to 0) and including all other variables with observable weights. With this complex model, it gives us the prediction for both y and sigma, and also the posterior distributions for each parameter. Is this ok?
Of course, this did not make full use of model averaging and stacking weights to combine multiple models into a single complex model. But I had a hard time to figure out how we can extract the sigma and parameters of each predictor in the stacked model. Thanks!
That is one way to do it, but I recommend to check Projection Predictive Inference for Generalized Linear and Additive Multilevel Models and the accompanied vignette.
Projection predictive inference correctly takes into account the uncertainty related to dropped out terms. And no need for stacking.