I’m writing here to ask for help in completing an introduction to autodiff with an encyclopedic coverage of forward- and reverse-mode tangent and adjoint rules. I put myself down as “editor”, but I’m going to manage the whole thing open source and give everyone who contributed credit so that it’ll be like one of those bio or physics papers if lots of people get involved.
People have been urging me to revise our autodiff paper for Stan and extend it to forward-mode. Stan’s forward mode is simple and generic, so instead, I decided to write a mini-textbook/encyclopedia on autodiff based on Mike Giles’s extended matrix autodiff paper. It’s a public repo on Git with code licensed under BSD:
- Bob Carpenter’s GitHub: AD Handbook
But I need help completing the tangent and adjoint rules for further matrix results, diff eq solvers, algebraic equation solvers, 1D integrators, etc.—all the fun stuff we support in Stan.
The draft is written in bookdown and there are makefiles and build scripts or you can just go into RStudio, load
bookdown, and type
In addition to expanded derivations following Giles, there’s
- intro to forward-mode
- intro to reverse mode
- intro to mixed mode
- HMM derivatives (Qin, Auerbach, and Sachs 2000)
- worked examples
- working C++ reference code
The reverse-mode code is based on the continuation-based reverse-mode I defined as an exercise on Discourse a while ago. The forward-mode code and functionals are based on Stan’s math library.
Feedback is also welcome, either here, through GitHub issues, or via email at