Time Series Model Quality, Out of Sample Likelihood

Modeling
1

How to access quality of time series model? To compare two models A and B - which one is better?

An example - Stochastic Volatility with T distribution, data - historical daily log returns \{r_t\} for stock.

\begin{align} r_t &= \mu + \exp\!\left(0.5h_t\right)\varepsilon_t \ \qquad \varepsilon_t {\sim} T_{\nu}(0,1)\\ h_t &= \omega + \phi (h_{t-1} - \omega) + \sigma \eta_t \qquad \eta_t {\sim} \mathcal{N}(0,1), \\[6pt] \end{align}

If I understand correctly Stan MCMC can’t estimate out of sample likelihood efficiently, right? It can’t do efficient incremental state update with next data point and require full refit, so time series of 5k points would require 5k mcmc refits, which is unpractical.

\sum p(r_t|\Theta,r_{1:t-1}) \quad t \in [2:T]

I assume the proper way to estimate it would be a) fit model with MCMC and get params posterior \Theta b) use these params \Theta with Particle Filter to estimate latent state without look ahead and calculate proper out of sample likelihood.

Strictly speaking it looks ahead a little bit, for params \Theta estimation, but not for state, but that’s ok.

I wonder if Stan has some extension that can run Stan Model with Particle Filter? Or nothing like that available, and some separate Particle Filter library required?

There’s PSIS-LOO approximate, but it produces absolutely terrible results. Models A and B have very different moments, and one model is clearly better, yet PSIS-LOO almost identical.

Maybe PSIS-LOO can’t be used in this specific case, because the data and the model is heavy tailed, and has jumps. The maximum of Pareto shape parameter k_\text{max}=2.4, there’re 1.9% points with k > 0.7.

Another approach to use Moments estimated on real and simulated data, like a) static moments, parameters of estimated T_\nu(\mu, \sigma) b) dynamic moments GARCH parameters.

In my specific case moments show a big difference between models A and B, while PSIS-LOO show identical Expected Log Predictive Dencity.

Wondering how people access the time series model quality? Do they use additional Particle Filter libraries? I’m new to MCMC, and maybe missing something obvious.

2

See

This question is also answered in Cross-Validation FAQ