In the loo vignette and the cross-validation FAQ I came across this very intriguing capability:
We also recommend to use application specific utility and loss functions which can provide information whether the predictive accuracy is good enough in practice as compared to application expertise. It is possible that one model is better than others, but still not useful for practice.
Does anyone have any examples using an application specific utility function? Bonus points if the example goes over how the application specific utility function was elicited. Thanks in advance!
If I remember correctly @avehtari has an example of this in one of his case studies, but I forget which one! Hopefully he can point us in the right direction.
Companies do use application specific utilities a lot (related to money, clicks, retention, waiting time, etc), but rarely talk about them openly. I have cited the following interesting papers
Miyamoto, J. M. (1999). Quality-Adjusted Life Years (QALY) Utility Models
under Expected Utility and Rank Dependent Utility Assumptions. Journal of Mathematical Psychology, 43, 201-237.
Fouskakis, D. and Draper, D. (2008). Comparing stochastic optimization
methods for variable selection in binary outcome prediction with application
to health policy. Journal of the American Statistical Association, 103, 1367-1381.
Fouskakis, D., Ntzoufras, I. and Draper, D. (2009). Population-based
reversible-jump Markov chain Monte Carlo for Bayesian variable selection
and evaluation under cost limit restrictions. Journal of the Royal Statistical
Society, Series C: Applied Statistics, 58, 383-403.