Test coverage


#1

Hey,

I’m looking into some coverage tools so we can report test coverage (like in pull requests!) and Bob was saying @bgoodri had done this at one point? I have some nice coverage printing out but right now it’s on a per test binary basis, which isn’t so great when the tests for a single idea are spread out across up to 9 files (across prim/scal up to fwd/mat), etc. clang has some built in that I’m using instructions from here for: https://alastairs-place.net/blog/2016/05/20/code-coverage-from-the-command-line-with-clang/

An interesting move might be to try to compile all the unit tests into a single gtest binary (which would solve my current issue). I might expect this to speed up compile times, even…

Thanks,
Sean


#2

It might with clang. But you have to make sure to #include each test file within {} to keep them from interfering with each other.


#3

I looked into this once upon a time. I think there are at least a few challenges:

  • for Stan thus is fine, but for Math we may miss compiler errors that the user may see. We can add different tests to fix that, so it isn’t the biggest concern.
  • we were pushing the boundaries of the compiler last I tried for being able to handle the instantiations, etc. in one executable
  • requires the whole test to be compiled on any change. Makes it slower for incremental development when the bottle neck is already the compilation.
  • this can be fixed, but will require work: we were sloppy with test code. There are functions with the same name / signature in the tests, not all consistent, that would prevent including all due to having the same definition of a function in the translation unit. That’s just a challenge and can be fixed.

#4

To add to Daniel’s list, there’s also the issue of making sure we have all the instantiations of templates we care about. So what we really need is coverage from the collective template instantiations. For example, in the density functions, some execution paths are explicitly turned off for some instantiations, but you could get complete coverage with just reverse-mode varaibles, but that wouldn’t test all the forward-mode instantiations.


#5

Although the cmake branch of math probably isn’t ready for review/merging yet, basically works and it can give coverage stats for tests with ctest -T coverage. I haven’t done this yet for two reasons.

  1. There’s one set of tests we’re not building and we should put that into our cmake build before we look for holes in coverage.
  2. I’m already using both of my computers to run some tools that are easy run if you have a working cmake build and those tools are using all of the CPU on both of my machines.

Once we get the missing tests into the cmake build, it should be easy for anyone to do.

For template coverage, it’s possible to manually instantiate classes, which should cause the coverage tool to see that the code exists. An example of this trick is given in this stackoverflow answer.


#6

I don’t think we can use that trick for many of our template instantiations because we have a combinatoric explosion of possible template parameter combinations, but the distribution tests do basically this for all of the ones we care about for distributions (which should also stress / cover most of our other templates fairly well).