Yes. But maybe that’s because I’m not clever enough to reason about this. Here’s what I’m thinking about:
- we need to specify two git hashes. Let’s call that A and B.
- we want to check out the repo at A, run the tests, stash the output of the test, then do the same for B
- while we do that, we want to time the tests
- we then want to compare the output of A and B, ignoring differences in things we know should differ (this will be in output; reported timing numbers)
For a particular OS / compiler / compiler options / git hash, we should be able to save the output of the test, but for any other combination, it doesn’t apply. That’s where I thought it made sense to be outside of the repo.
Thoughts? (I have most of this written in a draft wiki page; it’s hard writing about this. Can anyone point me to any resources about this sort of testing?)