Projpred - model with horseshoe prior does not converge

[fweber144]

I have not much to add to what @avehtari said, only the following:

For the prior: As @avehtari said, you are free to use any prior you like in your reference model. Personally, I have made some good experience with the R2-D2 prior (Zhang et al., 2022) in cases where I was unable to make the regularized horseshoe work as desired (but my understanding of the regularized horseshoe prior might not be good enough and I also did not always spend too much time on tweaking it).

For the multiple imputation thing, a similar issue has come up at Using projpred with brm_multiple and mice · Issue #275 · stan-dev/projpred · GitHub (related to Projpred interpretation). Conclusion is that projpred currently does not support multiple imputation, so you would have to come up with your own solution for this (as @avehtari said: in principle, this is possible). If you find a general way to handle multiple imputation in projpred, a pull request on GitHub is very welcome.

For the validation of the model on the independent hold-out dataset, I just want to add that if you have some other plausible model, I also think it’s better not to compare the predictive performance of your final model to some artificial “null hypothesis” consisting of the empty model but rather to that other plausible model. This goes in the same direction as @avehtari’s recommendation not to use p-values and basically means the same as you suggested above:

@Jacquie

fit an alternative model that only has the intercept (also on the 1st data set of course) and compare the logwise predictive density (on the holdout data set) between the two models (using loo_compare, using some cut-off, like elpd_diff>4 to say ‘yes, it was a good model’)

but with the important distinction that you replace the model consisting of only the intercept by that other plausible model. For that, you won’t be able to use loo::loo_compare() (as far as I know), but you will have to write some own code to perform the ELPD comparison.

Concerning the proj_linpred() issue

@Jacquie

I have another small issue. I want to get the predicted y for my new data (independent test data), where I’ve only selected the xs that were included in the final model of the train/test data. When I use proj_linpred it says that other x values (not included in the final model) are missing (it says 'following variables can neither be found in ‘data’ nor in ‘data2’). I am confused because I thought that the model was going to be only based on the small subset of terms selected (soltrms_final)?

this is a specialty of brms and not caused by projpred, if I remember correctly. But you can set the unselected predictors to some dummy value (e.g., zero for numeric predictors). This should not affect the results because—as you said—only the selected terms (those from argument solution_terms) will be used. As a long-term objective, a possibility could be to create a pull request for brms, adding an optional argument by the help of which not all predictors are required. projpred could then make use of that argument.