Sure, but there is in addition always a function defined for the double only case. At least this is what is the case for the softmax… and the function you refer to always returns a var type of thing. So this function cannot just return an Eigen::VectorXd thing.
Oh no… I just realized that, if you have a Jacobian already, then using adj_jac_apply to as a way to inject the Jacobian into the ODE solver is gonna be really inefficient.
The issue is you have the Jacobian, supply it to adj_jac_apply, and then the jacobian function in the ODE solver will reconstruct the jacobian one matrix vector product at a time (where only one element of these vectors is non-zero).
precomputed_gradients will be a better option for what you’re doing.
+1. As soon as we get it tested, it’s totally the way to calculate directional derivatives. I don’t know how well that’ll plug into the ODE solvers, but it’ll be faster even for systems with as many outputs as inputs (or even slightly fewer given the lower constant factors).
One issue is that we haven’t optimized any of our forward-mode derivatives. So the gain is in theory and should hold for simple things, but I wouldn’t trust it for complicated matrix stuff (though even then, plugging in the custom types should be reasonable).
You certainly won’t need it for programming, but the union card helps for jobs at universities and research labs.
A 10% speedup is “nice”. 4x is at least “great”, if not “fantastic”. It helps to log scale the y axis for performance reports so we can read off the speedup and see how it depends on N.
Yes. The design is set up to be mutable after the functor call, then immutable until the adjoint call, so you can save intermediate values for use in the Jacobian. You just need to use variables on our autodiff stack.
Yes, indeed. Having zeros in double-based matrix arithmetic’s no problem as they don’t incur overhead. At the same time, the adjoint-Jacobian-apply thing isn’t really set up for sparsity of the whole adjoint vector, as it assumes each adjoint needs to be incremented.
Hey @stevebronder, can I borrow your forwarding experience when you’ve got a minute? I’m working on adding forwarding (and generalising eigen inputs) to adj_jac_apply and want to check that I haven’t missed anything (still a new area for me).
I’ve got the changes up in this branch, would you mind having a quick review? I’ve definitely over-forwarded in some areas (scalars), but may have missed other things (or done them less than efficiently). Thanks!