Does that mean Stan builds have to wait for Math builds to finish?
Should we get the pipelines PR fixed first then? It seems like that would make sense? But I’m not sure.
Absolutely. This thread has wandered a bit from where it started. There are a lot of important things being mentioned, but they’re getting buried in here.
As part of moving to pipelines, I’ve parallelized a lot more of the stuff and made the entire job run using a single executor (with no executors used up in a sort of idle fashion, as was the case with the old jobs). So as a result I’ve made it so that each machine just has a single executor (well, I will finish doing this once they’re all pipelines) because adding additional ones would decrease throughput. So if there’s distribution tests going on gelman-group-mac, nothing else will run on that box until it’s done, and right now the Stan job’s performance test can only be run on gelman-group-mac (because it gets the wrong results on other compilers).
I think that makes sense! I’m hopeful that it finally goes through (last couple days it’s failed due to develop breaking, hah).
Makes sense. If I understand this right, that means things are locked down by box and not by job, so if there was another box available, a Stan build could work while the gelman-group-mac box is tied up?
Btw, thanks for all of this!
Yep, each box has its own track. Though most jobs are also set up to use multiple boxes if they’re available, so it won’t really interleave jobs or anything, it will go in the order the task is queued. So the first stage for Stan and Math is on a single executor, and after that passes it will queue up a bunch of tasks that can go to different boxes if available. All of those queued be executed in order, so in some sense a Stan build might go all the way through before a math build queued right afterwards starts.
I think that’s right.