I would appreciate being able to get frequentist crossvalidated estimates of some (potentially arbitrary) loss. The use case is essentially reporting models to an audience that is uncomfortable with Bayes (or just ELPD) but might still get excited about things like horseshoe regression.
I brought this up on Twitter and @avehtari suggested following up here (cc @anon75146577 as well). It looks like people have some tricks that work at the moment, but an official Stan solution might be nice.
1 Like
Can you elaborate what you mean by â€śfrequentist crossvalidated estimatesâ€ť?
Sorry, wrote that question late at night. I think what Iâ€™m really asking for is the ability to use other loss functions than ELPD (currently Iâ€™m thinking of ELPD as a loss, please let me know if this is a bad idea).
My understanding of what loo
currently does:

Given a model fit on full data, gets efficient estimates of leaveoneout posterior predictive distributions. For kfold crossvalidation, there isnâ€™t an efficient approximation to the posterior predictive, but loo
provides infrastructure for refitting models from scratch on the appropriate folds.

For each heldout data point/fold, evaluates the expected log posterior density of that point/fold using the posterior predictive distribution.
It seems like it would be easy to report RMSE/MAE and some other standard losses based on point estimates from the posterior predictive distribution and the heldout labeled data. This would be nice since ELPD isnâ€™t as intuitive to me as RMSE and friends.
1 Like
Ok. Loss and utility functions are independent from the inference method (e.g. frequentist or Bayesian). ELPD is utility as higher is better. If you prefer to minimize loss, you can use expected negative log predictive density (ENLPD), which is just ELPD.
Correct, although you can also get other useful information.
Yes, itâ€™s easy. See related examples
As itâ€™s this easy, there hasnâ€™t been pressure to make it even easier, but I agree it would be useful and there is now an issue for this.
We havenâ€™t been in a hurry to add RMSE as it is much weaker to detect model differences and ignores whether the uncertainty in the predictions is modelled well, but I agree that it can be useful to give some scale for the goodness of a single model.
If you like, you can list here or in that issue what you think are â€śstandard lossesâ€ť in addition of RMSE/MAE/R^2
1 Like