Relationship between LOO/WAIC and out-of-sample MSE?

General
1

Hello!

Grateful for any assistance here… I might just be being daft and not seeing what’s going on.

The WAIC and LOO give an estimate of elppd, the expected log pointwise predictive density for a new dataset.

Suppose that I have a normal observation model, ie
y[i] ~ normal(pred[i], sigma)
for the ith data point and pred[i] is the model prediction for the ith data point.

Is there a simple relationship between either WAIC or LOO, or elppd_WAIC or elppd_LOO, and the expected mean squared error (MSE) on a new data set?

I’m trying to communicate my findings to a non-technical audience and expected MSE on new data is much better understood by them. My model is quite expensive to compute so I’m trying to avoid doing explicit cross-validation.

Thanks in advance!

2

No, but you can use the E_loo function in the loo R package to obtain an estimate of the expectation of any function using the same weighting scheme that is used for LOOIC. In your case, the first argument would be a matrix with rows equal to the number of draws, columns equal to the number of observations, and each cell is the squared error between the posterior predictive distribution and the observed value of the outcome.

1 Like
3

Perfect! This is exactly what I needed. Thanks so much for your help and the fast reply!

:)