There are no free executors typically, so I am skeptical this would help. If we had many executors it would be a good idea, that I agree.
When we added the headers check with opencl I could not decide whether to put it separate or just do the check on the Full unit with GPU. For a jenkins test run for a PR that passes there is not much difference. However, if you are first and foremost interested in seeing if the Linux Unit tests pass, this might actually be a bottleneck. I havent really thought of that.
My proposal would be to change the pipeline to the following (after the “Linting & Doc checks” stage):
- Headers check
- Linux Unit with MPI
- Parallel stage with: Full unit with GPU (with OpenCL header check), Distribution tests, Threading tests, Windows Headers & Unit, Windows Threading
- …
Reasons:
- Most of the time Linux Unit with MPI finishes in 20 minutes (unless if it gets assigned to one specific worker, then its an hour) and if I am not mistaken the largest number of the executors are Linux so this should not be such a bottleneck
- a lot of jobs wait a lot of time for the GPU device. This way we reduce the executions of GPU jobs (jobs that will fail on linux will not run on GPU)
- the wait for GPU executors is done in parallel with the distribution tests
- if Linux unit tests pass most of the time the tests will go green (provided there arent any false positives due to CI issues)
Additionally we really need to remove the gelman linux machine from the distribution tests workers, as that job runs for 5-6 hours there.
Does that sound good?