I think you’re thinking about the reverse-mode complexity. Perhaps a little more concretely, suppose we have a three-argument function: f(x, y, z).

In reverse mode, we get this out for O(1) all in one pass:

- the function value, f(x, y, z)
- df(x, y, z) / dx
- df(x, y, z) / dy
- df(x, y, z) / dz

In forward-mode, we get the function value and a single directional derivative. So this is O(1):

- the function value, f(x, y, z)
- df(x, y, z) / dx

In order to also compute df(x, y, z) / dy, you will need to run forward-mode again, which will incur another O(1) cost. And the same for df /dz.