I’ve searched the forums for the Lorenz system and found this by @idontgetoutmuch, this by @bbbales2 and this by @yizhang, also this on github.
It looks like the Lorenz system gets used as a test problem within stan, but only to test the accuracy of the integration?
What are your experiences in actually fitting the problem?
I haven’t tried fitting any chaotic system, based on the following speculation:
For a general chaotic system, during HMC the interaction between numerical perturbation and system dynamics makes its behavior indiscernible to stochasticity. One manifestation of that is the the joint posterior should actually include components of solver controls such as tolerance. So if we want to brute force it in Stan as other ODEs we should include tolerance (and any tunables that affect solver behavior) as a parameter, and perform postprocessing to marginalize those tunables to get the posterior we want. Currently this cannot be done. Even it can be the outcome may not be usable in real applications. IMO it is generally hopeless to fit a chaotic system based only on observation. One must enlist system-dependent information such as previous trajectory to make prediction beyond Lyapunov time.
Thanks for the response @yizhang!
I would assume that if we suspect/know that our solver will have trouble with the current configuration, there is no way around cranking up the accuracy.
I do agree that if the system is so badly conditioned that we can have no faith at all in our numerical solution, then we should also have no faith in our model fitting.
However, we can trust our computation if we can be (relatively) sure that the accuracy of our numerical solution is within a small to moderate fraction of the measurement error. This we can relatively robustly verify in the same way as @jtimonen did here (sorry for always mentioning you, I just think the link is great :) ), by successively lowering our tolerances and computing the change in trajectory for samples from the prior.
So let us assume that we have a problem that is just chaotic enough to be interesting while allowing us to accurately solve it. I do think the Lorenz system qualifies.
Then I would assume that each trajectory serves as some kind of rather unique fingerprint for its associated initial conditions and ODE parameters. In fact, large Lyapunov exponents should actually help us identify our initial conditions (edit: initial conditions, not necessarily parameters). Just imagine the Lyapunov exponent being zero. Then we have less information to differentiate between different initial conditions than if the trajectories actually separate rather quickly.
What do you think?
Edit: Better example yet: Imagine a Lyapunov exponent of negative infinity, i.e. all initial points “immediately” contract to a single point. Then we have literally zero information to differentiate initial conditions.
It is more difficult to be sure about it with chaotic systems. As said in Springer et al., 2019: “A new
integration of the same system with the very same parameter values and initial
values, but using a different solver or slightly different solver tolerances, typically
leads to a totally different trajectory after an initial time interval over which the
trajectory is predictable.”
If there is data from an initial time course that behaves non-chaotically, then parameter fitting could be possible but generally I am suspecting it s very hard, because trajectories can change qualitatively as a function of parameters, which makes things difficult for MCMC. I am also speculating that with chaotic systems it is easier to accidentally miss high-probability regions in the parameter space because the ODE solver always fails there.
Huh. The settings discussed in the link may be slightly excessively chaotic, I will concede that :D Thank you for the link :)
I guess for the traditional approach you do rely on good enough data. For their setting of the Lorenz system, there are several periods in between measurement points, this might be slightly too much…