Test coverage


#1

Hey,

I’m looking into some coverage tools so we can report test coverage (like in pull requests!) and Bob was saying @bgoodri had done this at one point? I have some nice coverage printing out but right now it’s on a per test binary basis, which isn’t so great when the tests for a single idea are spread out across up to 9 files (across prim/scal up to fwd/mat), etc. clang has some built in that I’m using instructions from here for: https://alastairs-place.net/blog/2016/05/20/code-coverage-from-the-command-line-with-clang/

An interesting move might be to try to compile all the unit tests into a single gtest binary (which would solve my current issue). I might expect this to speed up compile times, even…

Thanks,
Sean


#2

It might with clang. But you have to make sure to #include each test file within {} to keep them from interfering with each other.


#3

I looked into this once upon a time. I think there are at least a few challenges:

  • for Stan thus is fine, but for Math we may miss compiler errors that the user may see. We can add different tests to fix that, so it isn’t the biggest concern.
  • we were pushing the boundaries of the compiler last I tried for being able to handle the instantiations, etc. in one executable
  • requires the whole test to be compiled on any change. Makes it slower for incremental development when the bottle neck is already the compilation.
  • this can be fixed, but will require work: we were sloppy with test code. There are functions with the same name / signature in the tests, not all consistent, that would prevent including all due to having the same definition of a function in the translation unit. That’s just a challenge and can be fixed.

#4

To add to Daniel’s list, there’s also the issue of making sure we have all the instantiations of templates we care about. So what we really need is coverage from the collective template instantiations. For example, in the density functions, some execution paths are explicitly turned off for some instantiations, but you could get complete coverage with just reverse-mode varaibles, but that wouldn’t test all the forward-mode instantiations.