Well that was anti-climactic. Two problems!
Adjoint ODE branch is here: https://github.com/bbbales2/math/tree/feature/adjoint-ode
For the Lorenz test problem (3 states, 3 parameters), forward sensitivity is about 20-30% faster than adjoint sensitivity. rk45 blows them both away cause it’s a non-stiff problem, but that and the harmonic oscillator were the easiest things to play with since they were already coded up.
So the forward sensitivity problem is going to have 12 equations, and adjoint will have 3 going forward and 6 backward. Not a huge difference, but I’d still have liked to see adjoint eek out the win because of the simplicity of evaluating the right hand side of the adjoint sensitivity equations with autodiff. For the harmonic oscillator, adjoint was about 50% slower or something (2 states, 1 parameter).
The problem seems to be though that the backwards integration is just harder than the forward integration. It takes around 2x the right hand side evaluations for the backwards integration as the forward sensitivity does. So even if the autodiff is more efficient, it’s just gotta do more work. (edit: each autodiff right hand side for the reverse mode should take 3x less work than for the fwd in this case)
There’s a lot going on in Sundials though. I tried to wiggle the common knobs, and these results seemed stable, but I really didn’t get a good sense of what is going on.
Now we could make this problem bigger or choose a different system and tip the scales in the favor of adjoint, but I’m not super keen on that. My original tests were with like 20 states and 20 parameters. In that case the forward sensitivity problem would have 420 equations and the adjoint one would have 20 forward and 40 in reverse. But I’d really like it to work better for tiny problems cause I think that’s more representative of what people work on (and One Size Fits All solutions are really nice).
Second of all, there’s a memory problem in how adjoint sensitivity uses the autodiff. During the chain of its vari it needs to call nested autodiff. This can cause segfaults and such if the stack tries to reallocate itself (cause the nested autodiff is growing) while it is busy reading itself (for the outer autodiff).