Just want to ask a quick clarification about elpd_diff: if there is a sizable difference between models (e.g., reference model has elpd_diff of 0 and the next best model has an elpd_diff of -300), am I able to conclude that the better model is massively better? Or can I only draw a conclusion that it is “better” without any qualifications on just how much more successful it is?
Assuming LOO diagnostics are ok and also that each fit has yielded a good unbiased sample from the correct posterior, a difference of 300 is a massive difference. ELPD is the expected log pointwise predictive density on a held-out observation, which is a natural metric for assessing a model’s predictive performance. See LOO package glossary — loo-glossary • loo
Excellent, thank you! I thought as much, but I didn’t want to make a silly blunder
It also depends on the standard error of the difference. Rule of thumb is that the diff should be twice the size of the error to be significant. Should be part of the loo_compare output.