A few projpred questions:
Is there a way to recover k values from cv_varsel when using loo? I often receive the “slightly high” warning message.
Any general guidance on when to use regularized horseshoe versus different priors (e.g., weakly informative)? I noticed a couple examples at https://github.com/avehtari/modelselection_tutorial use the rstanarm defaults rather than horseshoe. Maybe when n >> p?
Any updates on when projpred and brms get to play together?
loo is used only for the full model, so you can use loo function for the full model in the usual way to get k values and effective sample size estimates, and easy way to plot k values.
It’s not just n \gg p, it’s also whether you assume that there are some irrelevant covariates. In all those examples I tested both Gaussian and horseshoe prior, and just because of speed for notebooks I used Gaussian prior if there was not much difference in the posteriors and performance. stan_glm has also a weakness that the scale of Gaussian prior is fixed and thus in the examples the prior is very weak. If you are uncertain which prior to use, test both and check the posteriors and elpd from loo :)
@jpiironen and @paul.buerkner are working on this and maybe they can give some estimate?
@jpiironen has also an example how to use projpred for any model (but projecting only to linear models with certain likelihoods), and we’ll add this to the vignettes soon, but if you are in a hurry email me or @jpiironen.