Compare with exact LOO for observations with too high Pareto k diagnostics

[tomli]

@avehtari

@tomli

Hmm… My impression was that the inclusion of factors without predictive information would add additional uncertainty that would reduce elpd of left-out observations (either exact or PSIS-LOO), thus punishing the model for being over-parameterized.

That is the old folklore for maximum likelihood. This is much less likely to happen if we can integrate over the posterior accurately. If the integration works, I’ve seen only extreme examples where this would happen and some of them have silly priors. I have seen this also if the integration approximation fails. See also Comparison of Bayesian predictive methods for model selection | Statistics and Computing and Model selection tutorials and talks

@tomli

If the removal of redundant factors doesn’t increase elpd, then wouldn’t the best predictive model always be the one that includes everything?

Usually it is the best, and if not you should think harder about your priors and prior predictive simulation. See, e.g. Sparsity information and regularization in the horseshoe and other shrinkage priors, [1810.02406] Projective Inference in High-dimensional Problems: Prediction and Feature Selection, and Visualization in Bayesian Workflow | Journal of the Royal Statistical Society Series A: Statistics in Society | Oxford Academic.

I’ve read these papers. Will do it again. It bothers me that “include everything” would always be the best option; then I might as well include the zodiac sign of the landowners where each data point was collected. But I guess the strictly Bayesian view is that I (should) have some reasonable prior belief about how much such a factor would explain that I could incorporate in the analysis.

@avehtari

@tomli

than an a priori specified transformation.

I recommend to use splines or GPs for unknown non-linearities. It’s easy with rstanarm and brms. Exp would make sense only if there would be physical model with such parameter.

If you don’t mind me asking, why do you think exponents make no sense. My only worry is that the units become problematic and it’s difficult to elicit reasonable priors for the associated coefficients. But for predictive purposes, it seems pretty reasonable to me–it ends up with a monotonically increasing/decreasing transformation with linear (no transformation) as a special case.

@avehtari

Did you forgot to upload them?

Have you looked at plots that bayesplot provides?

oops. These are made with Bayesplot.