Case Study: Efficient Leave-One-Out Brier Score for time-dependent evaluation of Bayesian Survival Models

Dear Stanimals,

I just published (a first draft of) a small case study about implementing a Leave-One-Out version of an estimator for the Brier Score, that allows for a time-dependent evaluation (and comparison) of Bayesian Survival models:

https://ermeel86.github.io/case_studies/loo_pec_psis_ncog.html

The efficiency statement mainly refers to computational efficiency: I propose to use Pareto Smoothed Importance Sampling to approximate the LOO calculations for this time-dependent score instead of running exact LOO.

As a by-product, I show how to implement time-dependent hazards in proportional hazard models using M-splines.

Please share your honest feedback with me. It is highly appreciated, as this is a working document.

Thank you,

Eren.

5 Likes