Talked with @syclik on the phone and he mentioned that people were confused about what “supported external API” means. This post will clarify what that means and what the API is.
The API is a description of the functionality that we advertise as working and supported to other people, and that we promise to semantically version against (meaning we increment the major version number when we break backwards compatibility, etc). This idea comes from the closed source world and refers only to what external users can expect from the product. The external API makes no comments about what ELSE might be in the file system, what is still being worked on, and that kind of thing. Supported in this case basically means that we test it and care for it and version the library according to that specific set of functions. Since this is a header-only library, it’s slightly weirder and it does make sense to expose one particular file as the necessary include, but everything else in the API is at the level of doxygen - doc and function signatures. We do not guarantee implementations will or won’t change.
Specifically, this does not mean we need to get rid of prim, fwd, or mix modes just because they aren’t currently supported - it just means we aren’t versioning against them (so we reserve the right to change them), we don’t promise that they work in all cases (so we reserve the right not to test them completely), and we don’t promise to provide fixes for them (though we may anyway).
For anyone working on the Math library, please continue to work on what you’re working on. Because it’s not in the supported API now doesn’t mean that it will be removed. We’ll continue to work on expanding functionality, parallelization, and high order autodiff as planned.
I didn’t intend to introduce an orthogonal issue. I thought we were talking about the same thing, namely how to test the various versions of functions (prim vs. rev vs. mix, etc.). If that’s not what we’re talking about, I definitely missed the point.
To me, it comes from wanting to provide backward compatibility, which is independent of open/closed source.
Not for me. That Wiki says that
Here are the parts of the Math library that we regularly use and test and consider to be “supported.” …
All math-related classes and functions (e.g. stan::math::var and lgamma )
What isn’t a math-related class or function in the math lib? I’m not just trying to be pedantic here, I really don’t know. Last time I asked about things like our various vari extensions. What about the memory handling or the way the autodiff stacks work? Is the API at the level of var and vari data structures or just at the level of our functionals?
From the earlier discussion in this thread I think just at the level of the functionals (ie autodiff internals, memory layout, etc is not part of the API) - I’ll update the wiki to mention that.
I think for “math related” the most generous definition would be everything in the stan::math namespace supported to the API its doc string indicates. I would be okay reducing this to the list of functions exposed to the Stan language - @syclik what do you think about that?
I think that’d be OK for most uses of the Stan C++ lib to compute derivatives using our existing functions.
But what about all our internal helpers like operands_and_partials? Let’s say a user writes a new density function and we change operands_and_partials?
I don’t have a strong opinion one way or the other as to where we should draw the line, I just think we should make a choice.
We were talking about a wide variety of topics but I was specifically trying to get an item onto the list of things that would be affected by using Stan language usage of the math lib as a justification to limit math lib testing.