Performance tests for the math library

@seantalts wrote in this math PR comment:

@syclik what do you think is a good pattern for performance tests for math? Maybe my default would be a new test/performance directory that we somehow configure such that Jenkins runs all of the tests in there and somehow charts the time taken?

I think that @seantalts’s suggestion makes sense:

  1. create a new test/performance directory that’s meant for performance tests
  2. have Jenkins run these tests
  3. have Jenkins show the timing difference between develop and a proposed branch, or even what we do with the Stan tests which is to just run and record the state of develop

To make this easier, we could use a library like Google benchmark. Or use Google test’s ability to rerun multiple times. (I don’t want to add yet another dependency to the math library, but this seems less onerous because it’s a dev-only dependency.)

3 Likes

Yup—dev only is much better. I’d much rather use someone else’s framework than invent one. I’ve liked google test a lot so far.