Dear Stan community,
@betanalpha and I released a preprint which attempts to build a unifying framework for the application of automatic differentiation to implicit functions. The latter includes solutions to algebraic equations and differential equations.
@betanalpha has a very nice twitter thread which summarises our key results, as well as our motivation.
Here are some highlights:
- We review specialized methods used to construct autodiff algorithms for implicit functions. The two main approaches we discuss are the implicit function theorem and the adjoint method.
- We propose generalizations of these methods, with the goal to apply each approach to problems they were not originally designed for. Some of these generalizations are fairly abstract. We discuss the conditions under which these generalizations yield practical approaches and, as importantly, cases where they fail to work.
- Attempts at generalization shed light on why certain methods work well. In my experience, derivations for autodiff methods often involve prescient choices and steps which seem pulled out of thin air. Attempting to generalize these methods gives us insight into why certain steps are necessary and even natural.
- The Difference Equation problem provides a very nice example for how various methods can be used. A difference equation is finite dimensional but it kind of behaves like a differential equation. Essentially, you can apply every method we review to a single problem and compare them!
- Writing the paper was also an opportunity to offer a thorough introduction to autodiff. In @betanalpha 's words: “autodiff is not just an algorithmic implementation of the chain rule”.
We hope this paper we’ll further fuel ongoing discussions about autodiff in our community.