Advantages of LOO over DIC?

@avehtari’s paper states in its abstract “… loo have various advantages over… DIC” Alas, after reading the paper I don’t think I can enumerate those advantages. Could someone illuminate me or point me to the right reading?

Thanks!

From the paper

For example, DIC can produce negative estimates of the effective number of parameters in a model and it is not defined for singular models.

Unlike DIC, WAIC [and LOOIC] is invariant to parametrization

For DIC, there is a similar variance-based computation of the number of parameters that is
notoriously unreliable, but the WAIC [and LOOIC] version is more stable because it computes the variance separately for each data point and then takes the sum; the summing yields stability.

Moreover, DIC is not that Bayesian since you plug-in posterior means instead of integrating over a posterior distribution. And it is easy to tell from the LOOIC output when its assumptions do not hold, which is not the case for any other information criterion.

5 Likes

Ben was faster, but here’s my enumeration (and sorry for not making it more clear in the paper)

  1. Bayesian LOO estimates the predictive performance given the predictions are made using the posterior predictive distributions (ie integrating over the posterior of parameters) while DIC estimates the predictive performance if the predictions are made using plug-in predictive distributions (ie using a point estimate like the posterior mean of the parameters)
  2. Bayesian LOO is applicable also for singular models where the posterior converges to an analytic set instead of a point.
  3. DIC works well only if the posterior is close to Gaussian (not easy to say when it’s close enough) and thus it’s also not invariant to parameterization
  4. Bayesian LOO computed with PSIS-LOO is almost as fast to compute as DIC, but has good diagnostic to tell when the computation fails, while DIC doesn’t have good diagnostic
7 Likes

Thanks @avehtari and @bgoodri ! One more question. Any idea for when a vignette or similar for doing k-fold with loo 2.0 will be available?

Not soon, but maybe these two

and

  • kfold_split_random()
  • kfold_split_balanced()
  • kfold_split_stratified()

help?

3 Likes