- If you can use posterior predictive checking to show that some of the models are not sensible (or all), you don’t need to care about loo comparison
- if one of the models is the encompassing model that includes the special cases, and the parameters are not correlated a posteriori then you can look directly at the posterior instead of predictive performance (see, e.g. Beta blocker cross-validation demo and counter-example Bayesian version of Does model averaging make sense?)
- If you’re using RStan, you could use Moment matching for efficient approximate leave-one-out cross-validation (LOO) — loo_moment_match • loo which can sometimes get rid of high k values
- If not using RStan, and k is just above 0.7, you may try running MCMC for more iterations to get more accurate Pareto k estimate.
- If two above ones don’t help, and only some Pareto k are high, consider running MCMC just for these leave-one-out-folds (just make the observations as missing data to keep the model structure, this is discussed in Section 3.1 of Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models)
- If none of the above helps, then the issues is that you don’t know how big the error is, but if the difference between models is very big then there is only a small probability that the order of the models would be different with more accurate computation. With k larger than 0.7, it’s difficult to give a generic advice what does very big difference mean.
2 Likes