Questions about LOO-IC

General
1

I have some questions about LOO-IC:

  1. Can LOO-IC be used to compare models with the SAME likelihood but DIFFERENT priors? (assuming all of the pareto-k diagnostics are fine etc?)

  2. Can the LOO-IC be used to compare models with a DIFFERENT likelihood / parameterisation, and also DIFFERENT priors? (i’m pretty sure this is a “yes” as otherwise surely LOO-IC would not be very useful?!)

Also, I have the same questions but for the DIC.

2
  1. Yes, but do not use LOO-IC to select your priors. In particular, LOO-IC will prefer priors that are tightly concentrated around the point estimates that yield the best leave-one-out performance. There is no guarantee that the point estimates that yield maximal LOO performance are actually (very close to) the true parameter values.
  2. Yes.

For DIC we’ll have to let somebody else chime in, but I’m pretty confident the general attitude of this forum will be that you should prefer LOO-IC to DIC anyway.

1 Like
3

See Cross-validation FAQ questions 13, 22, and 23.

But also when using LOO, there is no sense in multiplying by -2 (there can be sense in multiplying by -1). CV-FAQ 23.

1 Like