Closures discussion

Why is this necessary? If we copy vars, don’t we lose the connection to autodiff?

Is there a description of why this won’t work somewhere? Obviously I thought it would when writing the spec, but I didn’t prototype it.

Why is the functions qualifier necessary here? I would’ve thought the ambiguity with variable declarations would be resolved because <type> <identifier> '(' doesn’t match the variable declaration syntax.

Will there also be anonymous lambdas? And asisgnments like

vector(real, vector) dz_dt
  = (real t, vector z) { ... return [du_dt, dv_dt]'; }

Everything doesn’t need to be implemented all at once, but I think this should be the eventual goal (modulo whatever concrete syntax winds up making sense).

The higher order functions use nested autodiff that is disconnected from the main stack and then manually copy the gradients. I think reduce_sum would segfault if accessed the same stack concurrently from different threads. I don’t know why ODE solvers need it, I just copied the code @bbbales2 wrote.

Yes, it does look unambiguous. But menhir complained and I didn’t know how to fix it. Hopefully we’ll get rid of the qualifier eventually.

I don’t think anonymous functions are useful. Stan isn’t a functional programming language and doesn’t have a lot of higher-order function. Giving every function a name isn’t so bad.

The reason Stan doesn’t have a lot of higher order functions is that they’ve been too hard to write and we don’t have lambdas.

I’m OK adding anonymous functions later. The problem with anonymous functions or actually having function variables is that we’d need to code the whole type language for them.

If we’re going to compute Jacobians in nested autodiff, we can’t be touching the autodiff variables of the parent stack (and to do the Jacobian we’ll need to zero adjoints and such).

Maybe deep_copy isn’t quite the right name, cause what it’s doing is making a copy of the autodiff variables that don’t point back to their sources (which is useful for computing Jacobians on nested stacks).

It might be possible to pull this logic out of the functor, but would have to stare at the code.

A few comments:

  1. This all looks great, I love the prototype Stan code!
  2. I’ll update the algebraic solver (there are several things that need to happen here…). But before doing this, I want the ODE prototype to be close to done; that way, if you find an error or revise the spec, we only need to do this one. My understanding is we’re almost there. I’m not quite sure whether variadic and closures are redundant. We can discuss and decide which one to prioritise.
  3. Regarding higher-order autodiff, we’ll probably want higher-order adjoint methods. Maybe we can bite the bullet if all we’re doing is a simple Laplace approximation but I’m not 100% sure about this.