I talked to @Bob_Carpenter about this and he mentioned that the adjoint functions might also need to have access to the original arguments as well.
Given that I kinda think we might need functor-like functions in Stan that have local storage to make this work. Maybe something that isn’t too far even from how the adjoint jacobian stuff is implemented in C++ now. This one makes use of temporary storage: https://github.com/stan-dev/math/blob/develop/stan/math/rev/mat/fun/simplex_constrain.hpp .
For the linear algebra stuff these temporary values are non-trivial.
So something more monolithic like you are suggesting here: Stanc3 type system overhaul - #10 by seantalts