CI: add a GHA that fails on PRs that contain a :SKIP test

[BBBmau]

Description

We sometimes run into PRs introducing tests that must be skipped due to k8s environment mismatch. This leads to CI returning all passing with the test never actually being tested. This can be misleading to reviewers on first glance when looking at a 100% passing PR.

A solution would be to include a GHA that checks the PR for new tests being added and to check whether or not it's being skipped. GHA would then fail providing that the following tests in the PR are skipped. The only way to bypass this would be to include a label (ideally the label would be added after the reviewer does their manual test of the PR). Label could be named manually-tested.

Here's an example of some confusion as to why CI was passing before realizing that it's being skipped.

#2557 (comment)

References

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[JaylonmcShan03]

Thanks for tracking this! I can put this on my backlog for Q1 :)