Verification tests fail in CI with 403 status codes

[iaindillingham]

It's not uncommon for the verification tests to fail in CI with 403 (forbidden) status codes.¹ The verification tests aren't run locally (e.g. with just test), so there's no reason to think the failures are specific to CI, however.

The failures result in long stack traces that conclude with coverage errors. However, coverage isn't to blame. When looking at the following examples, focus on the "short test summary info" section:

• Run 9761605098
• Run 9761601271
• Run 9745846966

Notice that the associated PRs may not touch the tests or the functions under test. Indeed, they may relate to JavaScript rather than to Python.

Considerations

• Are we hitting GitHub's API too many times in one run?
• Or in several concurrent runs?
• Can we run the verification tests nightly?
• Are we hitting primary or secondary rate limits?
• Are we following best practices?
• Should we set GITHUB_TOKEN? (CI test run can be flaky due to exceeding GitHub API limits reports#613)

Footnotes

1. https://bennettoxford.slack.com/archives/C069SADHP1Q/p1719926997444489?thread_ts=1719821734.604979&cid=C069SADHP1Q ↩

[evansd]

Sorry, I've had the Slack conversation regarding this bookmarked to come back to for ages but only just got round to it.

My high-level comment is that I think the right approach here is to not run these tests as part of CI and instead to run them on a schedule, as you suggest:

Can we run the verification tests nightly?

I think it's really important for developer sanity that CI be, as far as possible, deterministic and hermetic: CI should fail because of some change in the PR, and for no other reason.

Tests that external services still behave as expected are important, but don't relate to specific code changes and so should be run on a schedule rather than in CI. And that's better because operationally as well because you find out about potential breakages soon after they occur, rather than the next time you happen to open a PR.

Of course that still leaves the problem that the scheduled tests might end up being flaky too! I don't have any particular expertise to offer here, but the things you've already listed cover everything I'd look at in the first instance.