discussion: Shall we let the CI run more rounds of the tests for highly critical/complicated PRs

[lmatz]

Shall we let the CI run more rounds of tests for highly critical/complicated PRs?

i.e., We associate a PR with a GitHub label critical. Buildkite figures out the label, and then test multiple rounds/spend more time in CI.

Originally posted by @lmatz in #4857 (comment)

For deterministic ones, can we simulate more cases, e.g. use multiple random seeds.....?

[wangrunji0408]

After #4917, deterministic parallel e2e tests will be run 10 times with different seeds for each PR.
We will continue optimizing the compile time and execution time to allow running more rounds in a reasonable time.
Meanwhile, we will explore the ways to improve the efficiency, to hunt bugs within less rounds.

[lmatz]

I see, more efficient testing and more effective bug detection are definitely the way to go.

I guess I am just not sure how good the ways to improve the efficiency, to hunt bugs within less rounds. would eventually be, considering it depends on the indeterministic nature of the bug. Not in the sense of how much is improved relatively, which I am very much optimistic, but how good it will be at the absolute scale.

Is 10 times good enough?
Since we largely cannot predict the chance of detecting the bug, I feel we need to be on the pessimistic side. And we don't want to run the same amount of tests for a simple/deterministic PR.

[wangrunji0408]

Good point! I think we should first evaluate the simulator itself about how effective it can hunt bugs. We can manually construct several common bugs or use existing bugs to see how many rounds the simulator needs to run before catching it. After quantifying the effectiveness, we can determine the number of rounds in CI with confidence.

[lmatz]

Seems we have demand for longevity testing #4966 (review), it is now done manually.

After #5330 is done, maybe we try it first.

As the longevity test takes a lot of time, we may schedule the test for these critical PRs at night only.

A trivial way to achieve this is to assign high-priority to all the other non-critical PRs, and use the standard label for critical ones only.
But we may find a better way to achieve this.

[github-actions]

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.