Good points. Intellectually, I agree, but pragmatically speaking, I've seen where it's bitten us time and time again. The future maintenance cost when there are no tests are really, really, really bad.
Part of the problem is defining API boundaries, especially when we're doing things generically. Just looking back at Stan, we had originally templated things only expecting
var. Then we had to take care of
int and we expanded things to
Eigen::Matrix<var, R, C>. Then again expanding the API boundaries to replace
fvar<fvar<var>>. So, if we're really careful in defining the API boundaries and testing it all the way through, then I'm ok. But then again, I'm more comfortable breaking down something that is so generic and testing individual pieces and making sure they behave properly.
Only if you're willing to go through the combinatorics and call it with every possible template instantiation crossed with boundaries conditions.
Here's where I'm torn. I completely agree that there's a cost to maintaining tests. But I don't think the burden is as large as either: introducing an error due to negligence and having to track through the code or having a future person change a non-tested piece of code to something they feel is safe and that having unintended side-effects that weren't considered.
The problem with code that's only intended to be used internally may only be safe for a certain input space. Not documenting that causes problems in the future. Not documenting it also causes issues of repeated code. See the meta functions. We didn't document it and there are definitely duplicate functions in there.
Like I said, I understand the arguments and I'm happy with a case-by-case evaluation depending on the complexity of the code and test coverage of the API boundary. For the pull request in question, I'll push more on the API boundary and make sure we have it tested. For something that may affect every Stan program, I tend to unit test every component so that I've verified the behavior I expect. I also find it easier to change implementation and design when tests are in place, but I can definitely see how this might discourage more novice C++ programmers (there really aren't good refactoring tools that I've found).
Anyway, back to the original question: should we come up with some policy and put that up somewhere?