I found one way to speed up Jenkins a bit - we can save (an estimated) 2 hours from the Math pull request job by letting the upstream stan tests run in parallel; they were just doing “./runTests.py src/test” and now it’s “./runTests.py -j{PARALLEL} src/test”.
Anyone see any issues with that? @syclik especially might know
More broadly, it’d be nice if we could do the distribution tests using the new testing framework ideas in @Bob_Carpenter 's work. Is there still work being done on that? Is it ready for someone else to take a look at upgrading the distribution tests with those techniques? Might be a good new-dev-friendly task for someone of the right background…
Awesome. Something else we could do is to just kick off the job before waiting on the other jobs to finish.
Re: distribution testing. Perhaps it’s wise to split it up into rev / fwd / mix tests. And if we were really clever, we could generate the dependencies and search for the ones that the pull request actually affects.
Hmm. I’m a little scared of that kind of thing - those dependency systems can be complicated and hard to test correctness for… Not that we shouldn’t do it, but it seems like it could be pretty time consuming to get right.
For now, I just added Ben’s Linux box to the menagerie, so that should double our bandwidth if not addressing our latency (currently at 9+ hours for a math pull request (down from 11!), 7.5 or so of which is distribution test compilation).