The branch issue-7/implicit-functions has the chapter on algebraic equations with two derivations: one that uses the implicit function theorem, and one that uses the more general adjoint method. You can check out this branch and then compile the AD-handbook to read the chapter. The cool thing about this approach (pedagogically) is that algebraic solver provides a very simple example of the adjoint method. The traditional case of ODEs is more complicated.
This post covers how we should differentiate the algebraic solver in Stan. @betanalpha, @vianeylb, and I review the ODE case in this manuscript and provide further references; I’d said the benefit of this review is that we do the derivations in great details. Other papers I like:
- the DAE adjoint method by Gao et al, 2003
- implicit functions and optimal values by Bell & Burk, 2008
I have work on constrained optimization, but I haven’t shared this yet. It still needs refinement.