Fvars

With forward mode only all you can get out are directional derivatives, so you can associate each component with a certain inner product of a derivative with vectors. And at higher-order those inner products mix many different orders of derivatives making things very confusing.

For example, consider a many-to-one function, f : R^{N} -> R. Evaluating the function with a fvar<double> will require 2N inputs: the N function inputs, call them x, and the N sensitivities, call them u.

The fvar<double> returned by the evaluation will contain two values. The first value will be
the function value, f(x), which can be thought of as 0th-order derivative, if you will. The second value will be the inner product <u, df/dx(x)>.

For fvar<fvar<double>> you’ll need 4N inputs, the input x and three vectors which I’ll call u, w, and z. The evaluation of the function then yields

ffd.v.v: f(x)
ffd.v.d: <u, df/dx(x)>
ffd.d.v: <w, df/dx(x)>
ffd.d.d: <z, df/dx(x)> + <u, d^{2}f / dx^2 * w>.

In particular, ffd.d.d braids together first and second-order derivatives. This is related to the fact that the compositional algebra of higher-order Taylor expansions is wacky.

For more details check out Chapter 1 of the Nomad Manual, https://github.com/stan-dev/nomad/tree/master/manual, which you’ll have to clone and compile.