How to compute SE as in loo package


#1

Hi all,

I am curious to see how to compute SE as in the output of the loo package when we use function waic(.).

I know that from the formula Gelman et al. (2013) we can estimate WAIC using

computed\mbox{ }lppd= \sum\limits_{i=1}^{N}\mbox{log}\left(\frac{1}{S}\sum\limits_{s=1}^{S}p(\boldsymbol{y}_i|\boldsymbol{\eta}^s)\right)
and
\sum_{i=1}^{N}V_s^{S}\left(\mbox{log }p\left(\boldsymbol{y}_i|\boldsymbol{\eta}^s\right)\right)

I think that after running the model, then S is taken as a fixed number (the sampling part). Therefore we can obtain one estimate.

Could anyone help me to understand why we have SE in the output?

Thank you for reading and Happy New Year!


#2

See Vehtari, Gelman and Gabry (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. In Statistics and Computing, 27(5):1413–1432. doi:10.1007/s11222-016-9696-4. Preprint

And this reply to a related question


#3

Thank you so much, @avehtari.

I will look at them.

At this moment, may I have a question?

Using the output from LOO package, i.e. waic and its se, am I correct if I say that the 95% credible interval for waic is: [waic-1.96se, waic+1.96se].

Please correct me because I am using it for a paper under preparation and the paper is being submitted soon!

Thank you so much!


#4

Yes, but only if 1) you don’t get high k values from Pareto-k diagnostic from loo() function, 2) the model is well specified, 3) no outliers, and n is large (at least in hundreds). If you get high k values, then waic is biased, and if the model is badly misspecified, there are outliers or n is small, then the interval is not well calibrated (although it may still contain some useful information)


#5

@avehtari:

Suppose that four conditions you mentioned above are satisfied and two CI are non-over lapping. The first WAIC and its CI is 73693 [72775, 74611] and the second is 114921 [113935, 115908].

In this case, is this safe to say that the first model is better than the second?

Please note that I had run two models and I only saved the output as an excel file containing WAIC and its SE, so I cannot use function compare or similar function in LOO package.


#6

Here you have log densities in equations.

For log densities, higher value is better. So you need to also clarify whether above values are based on log densities or negative log densities (and maybe even further multiplied by 2) , The difference in performance is clear.