Two-speed CI?

[achamayou]

Our current CI job takes a relatively long time to complete (about 30 minutes). Historically, we've seen that time creep up to that level, and we've done cleanup/speedup passes to reduce it down. While this approach may work once again, we could also contemplate having a lightweight CI, which is always enabled, even on Drafts, and a more expensive CI, which is perhaps only run on PRs that are out for actual review.

This would allow a quicker turnaround early on, less usage of executors, but retain complete checks before the merge to master.

A possible implementation for this would be to query the status of the PR in the CI job: https://docs.github.com/en/rest/reference/pulls#get-a-pull-request exposes a draft attribute and could be used to decide how many configurations to build and test. A problem with this is that a status change on the PR would presumably not invalidate the check and trigger a new CI.

Ideas are welcome.

[letmaik]

Maybe an easy hack could be to split the test suite in two parts to allow for more parallelism. If all agents are always busy during the day then an additional one could be added, but I have a feeling that there is quite a bit of room.

[achamayou]

That wouldn't help with executor usage unfortunately. We've done this in the past, and we've found that the time it took to transfer artifacts from one executor to a new job running on another executor was very high (almost a minute). It's possible to lower it by transferring just what's needed for some tests, but makes the pipeline very brittle to test changes.

[letmaik]

I'm not sure I understand. Instead of "NoSGX Debug" there would be "NoSGX Debug 1" and "NoSGX Debug 2". No artifacts need to be transferred between the two. The jobs run exactly like now, except that only half the tests run in each job. Sure, there is a 1-2min extra overhead due to duplicated compilation, but in total it would still bring down time roughly by half, given that the runtime of the tests is what takes the longest.

[achamayou]

Compilation is variable depending on how warm ccache is, a cold build in the morning is more like 5 minutes. There can also be up to a couple of minutes of repo and container set up:

Doing it twice is fine for individual job latency, but reduces total throughput, since the other executor(s) doing the same build can't be used to run someone else's build/tests. This in turn will hit individual latency in busy times because it raises the probability that some of the jobs will be queued.

So it may help at time when there's a single CI running, but at the expense of creating more congestion at busy times.

[achamayou]

Having looked at this, there doesn't seem to a way to trigger on PR status change (draft true to false), so this isn't effectively possible.

[eddyashton]

I was looking at Mergify, because it supports the merge train behaviour we were discussing to minimise CI reruns. It also has options to trigger a pipeline based on some other checks, and I think we could get to multi-stage CI by fiddling with labels. Of course all of this comes at the cost of giving the Mergify bot write access, and letting it create and merge PRs.

[achamayou]

@jumaffre had a look at bot-based solutions, but one unpleasant aspect was the need to give them write access to forks so as to be able to update them with latest commits. I'm not sure how mergify deals with that.

[jumaffre]

My attempt to have auto merging PRs is here: #1035 It worked fine for the main repo's own branches but as @achamayou mentioned, it wasn't ideal on forks (details here: pascalgn/automerge-action#46).

[achamayou]

The consensus seems to have landed on making efforts to improve CI speed rather further split jobs. In particular, de-duplicating the suite and standalone tests, and enabling concurrency.