General formula for computing WBIC?

Hello all,

I am interested in comparing Mixture IRT models using several indices such as LOO, WAIC, WBIC, AIC, BIC, and DIC.

WAIC can be obtained using the “loo” R package, but not WBIC.

Is there a general easy formula that I can code in R to compute WBIC, using the posterior samples?

I tried to understand how WAIC is computed in “loo” package in order to be able to compute WBIC, but unfortunately, I am not that good at programming.

I came across a similar question in this forum:

It seems that WBIC is not preferred in comparison to LOO, but it can be computed anyway.

Would you help me figure out how to compute WBIC.

Thank you,
Rehab

Unless you are doing research on how the behaviour of these approaches differ, there is no need to use all these for model comparison. From these, LOO is the best choice. WAIC is also justified for Bayesian modeling, but the computation of WAIC fails more easily than computation of LOO, and WAIC failure is more difficult to detect than LOO failure. AIC is for maximum likelihood and no constraints. DIC is for posterior mean point estimate. Thus AIC and DIC are not ok for full Bayesian inference. BIC is assuming flat prior and can be justified only for regular models asymptotically. WBIC has even worse computational issues than WAIC. You can read more about the theory in A survey of Bayesian predictive methods for model assessment, selection and comparison and experimental results comparing LOO and WAIC in Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC.

2 Likes

Dear Vehtari,

Thank you so much for your reply.
Actually, yes, I am doing research related to the perofrmance of these measures.

Then, I will use your answer as a justufcation for not including some measures on fully Bayesian analysis.