K-fold predictive validation of survival (Matlab Interface)

[avehtari]

Great that you have found our code useful!

@Ashwani_Jha

I am trying to predict survival from brain cancer (n=70; of which n=40 censored)

That seems like quite a challenging data.

@Ashwani_Jha

I am trying to predict survival from brain cancer (n=70; of which n=40 censored) using Peltola’s/Vehtari’s freely available weibull survival model (single-group model with horseshoe priors):
http://becs.aalto.fi/en/research/bayes/diabcvd/ - thanks so much for this!

Note that this code dosn’t have our new recommendation for setting the horseshoe hyperparameters [1707.01694] Sparsity information and regularization in the horseshoe and other shrinkage priors

@Ashwani_Jha

I have a number of models with different numbers of imaging biomarker covariates

Can you tell how many? I’m asking to get a better feeling whether horseshoe is useful and whether you might need regularized horseshoe ([1707.01694] Sparsity information and regularization in the horseshoe and other shrinkage priors) in this case. As you have quite many censored observations, and censored observations are like binary classification observations, you may need regularized horseshoe if the number of covariates is larger than the number of observations.

@Ashwani_Jha

I have repeated each model k-times with each model trained on 8/9th of the n=30 uncensored survival data (and all of the n = 40 censored data) and used to predict the remaining 1/9th of the deaths.

I think you should include censored observations in your cross-validation.

@Ashwani_Jha

For each out-of-sample subject:

1. Evaluate weibullpdf(alpha, exp(-(mu + Xobs_bg * beta_bg + Xobs_biom * beta_biom)/alpha)) with the covariate values (Xobs_bg and Xobs_biom) for that subject, at time t, where t = actual time of death.
2. Repeat this for all posterior samples of mu and alpha. Take the mean and then log this.

Correct. You could also compute for the censored cases
weibulldccdf=1-weibullcdf(…) with same parameters and with the censoring time.

@Ashwani_Jha

‘out-of-sample marginal log-likelihood’ aka ‘log evidence’ of the model

It’s not ¨log evidence’. It could be called ‘log pseudo-evidence’ following Geisser and Eddy (1979).

@Ashwani_Jha

assuming all models have equal prior probabilities, difference of 3 is good evidence

It’s not evidence and you should not combine it with model prior probabilities. Although you could call it pseudo-evidence and use the same scale as for the evidence, it seems that it’s not so good approach and you should instead take into account also the uncertainty in the difference (Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC | Statistics and Computing, [1507.04544] Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC). Further evidence on the importance of taking into account the uncertainty can be found in our paper on combining models ([1704.02030] Using stacking to average Bayesian predictive distributions), where we compare also Pseudo-BMA (based on pseudo-evidence differences) and Pseudo-BMA+ (based on using both the difference and the uncertainty), and the latter gives clearly better results.

Unfortunately there is also some challenges in interpreting the difference and the related uncertainty as discussed in my other answer today Interpreting elpd_diff - loo package - #4 by avehtari

Note also that if you are using cross-validation to compare many models, you may overfit in the selection process as shown, e.g. in Comparison of Bayesian predictive methods for model selection | Statistics and Computing

For a survival models you might also want to compute something like Harrell’s C (http://onlinelibrary.wiley.com/doi/10.1002/sim.4026/abstract), which might have more easily interpretable scale.