How to compute SE as in loo package

[Trung_Dung_Tran]

Hi all,

I am curious to see how to compute SE as in the output of the loo package when we use function waic(.).

I know that from the formula Gelman et al. (2013) we can estimate WAIC using

computed\mbox{ }lppd= \sum\limits_{i=1}^{N}\mbox{log}\left(\frac{1}{S}\sum\limits_{s=1}^{S}p(\boldsymbol{y}_i|\boldsymbol{\eta}^s)\right)
and
\sum_{i=1}^{N}V_s^{S}\left(\mbox{log }p\left(\boldsymbol{y}_i|\boldsymbol{\eta}^s\right)\right)

I think that after running the model, then S is taken as a fixed number (the sampling part). Therefore we can obtain one estimate.

Could anyone help me to understand why we have SE in the output?

Thank you for reading and Happy New Year!

[avehtari]

See Vehtari, Gelman and Gabry (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. In Statistics and Computing, 27(5):1413–1432. doi:10.1007/s11222-016-9696-4. Preprint

And this reply to a related question

SE of elpd_loo - loo package

It seems there is some confusion with the terms. Likelihood is a function of parameters. \widehat{\mathrm{elpd}}_{\mathrm{loo},i} are log predictive densities (or log predictive probabilities). Eqs 22 and 23 do not include estimation errors for individual \widehat{\mathrm{elpd}}_{\mathrm{loo},i}. We do mention in the paragraph starting at the bottom of page 19 and continuing at the top of page 20, that we could compute the Monte Carlo estimation errors for individual terms, too, and loo >2.0.…

[Trung_Dung_Tran]

Thank you so much, @avehtari.

I will look at them.

At this moment, may I have a question?

Using the output from LOO package, i.e. waic and its se, am I correct if I say that the 95% credible interval for waic is: [waic-1.96se, waic+1.96se].

Please correct me because I am using it for a paper under preparation and the paper is being submitted soon!

Thank you so much!

[avehtari]

@Trung_Dung_Tran

Using the output from LOO package, i.e. waic and its se, am I correct if I say that the 95% credible interval for waic is: [waic-1.96 se, waic+1.96 se].

Yes, but only if 1) you don’t get high k values from Pareto-k diagnostic from loo() function, 2) the model is well specified, 3) no outliers, and n is large (at least in hundreds). If you get high k values, then waic is biased, and if the model is badly misspecified, there are outliers or n is small, then the interval is not well calibrated (although it may still contain some useful information)

[Trung_Dung_Tran]

@avehtari:

Suppose that four conditions you mentioned above are satisfied and two CI are non-over lapping. The first WAIC and its CI is 73693 [72775, 74611] and the second is 114921 [113935, 115908].

In this case, is this safe to say that the first model is better than the second?

Please note that I had run two models and I only saved the output as an excel file containing WAIC and its SE, so I cannot use function compare or similar function in LOO package.

[avehtari]

@Trung_Dung_Tran

I know that from the formula Gelman et al. (2013) we can estimate WAIC using

computed\mbox{ }lppd= \sum\limits_{i=1}^{N}\mbox{log}\left(\frac{1}{S}\sum\limits_{s=1}^{S}p(\boldsymbol{y}_i|\boldsymbol{\eta}^s)\right)

and

\sum_{i=1}^{N}V_s^{S}\left(\mbox{log }p\left(\boldsymbol{y}_i|\boldsymbol{\eta}^s\right)\right)

Here you have log densities in equations.

@Trung_Dung_Tran

Suppose that four conditions you mentioned above are satisfied and two CI are non-over lapping. The first WAIC and its CI is 73693 [72775, 74611] and the second is 114921 [113935, 115908].

In this case, is this safe to say that the first model is better than the second?

For log densities, higher value is better. So you need to also clarify whether above values are based on log densities or negative log densities (and maybe even further multiplied by 2) , The difference in performance is clear.