How can I compute WBIC with for a brms fit

Interfaces brms
1

Short summary of the problem

I fitted a model used brms, I know WAIC can computed by loo package,

but I do not know how can I compute BIC,

is there any package to compute BIC for brms model?

thank you very much

2

I don’t think BIC is supported in brms. I looked around fairly quickly for another package that might be able to help. I didn’t find anything in a quick search.

3

We don’t recommend WBIC as its computation is likely to be unstable without good diagnostic. We don’t recommend either WAIC, but it’s still in loo package for historical reasons.

If you like you can tell why you would like to compute WBIC or WAIC, and I can try to convince why it’s better to do something else 🙂

2 Likes
4

Thank you vere much for your help, Mr. Ara_Winter.

5

Mr. avehtari
Thank you very much for your help.

I looked up some papers and online courses said that WAIC can check whether the models predict well, and WBIC can check the models how far from the truth value. So I tried to ues WAIC and WBIC for my model selection.
In my model, explanatory variable are type of canopy trees (Fagus crenataand Magnolia obovata and years (2014,2015,2017,2018,2019), response variable are culm number, biomass, mortality, leaf area etc. of floor plant. part of my models shows in the picture.

models
models700Ă—905 13.4 KB

I tride to choice one model for each response variable and explain the results in my journal.

thank you very much.

6

WAIC is close to leave-one-out cross-validation, but fails more easily without good diagnostic (see [1507.04544] Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC, published at Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC | Statistics and Computing). Thus instead of WAIC, we recommend PSIS-LOO with possible refits or moment matching importance sampling (see [1906.08850] Implicitly Adaptive Importance Sampling).

WBIC is approximation to marginal likelihood, but fails more easily than the best computational approaches like bridge sampling or path sampling (there is no direct comparison paper, but WAIC is worse than importance sampling LOO, and WBIC is the expected to be worse than importance sampling marginal likelihood, and bridge sampling and path sampling are fancy adaptive importance sampling algorithms). So instead of WBIC we would recommend bridge sampling or path sampling if you really need the marginal likelihood, but we usually don’t recommend because the marginal likelihood / posterior probability are in theory sensible only if the true model is included which we rarely can assume and they can be over-confident in case of model misspecification or small data (see [2003.04026] When are Bayesian model probabilities overconfident?). Only in special very simple cases we could say that marginal likelihood and its WBIC approximation are reliable to measure “how far from the truth value”.

I suggest to watch the videos

You can also find relevant case studies for model selection at Model selection tutorials and talks

There are also couple chapters in the book Regression and other stories explaining model selection, which could be good for you.

Looking at the WAIC values you show

  • M. Obovata doesn’t contain any predictive information for any of the target
  • year is predictive for “culm number…” and “culm mortality”,
3 Likes
8

Mr. avehtari

Thank you very much again for answering so many questions. I have no background in statistics, so even a simple problem, sometimes I need to spend more time to understand.

But Mr. avehtari, you are a professional, I think I shoud follow your advice carefully. I will learn about loocv and improtance sampling, and try to improve my paper with them.

I taught Bayes analysis by myself for half a year, no one helped me, so I am very grateful to your advise.

thank you very much.

4 Likes