Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve playwright comments #1709

Merged
merged 4 commits into from
Dec 28, 2024
Merged

Improve playwright comments #1709

merged 4 commits into from
Dec 28, 2024

Conversation

djahandarie
Copy link
Collaborator

In #1676 we merged a change that completely broke all popups.

Our E2E testing (playwright) did detect the issue, but its output was ignored.

My analysis is that it was ignored because the output of playwright-report-summary:

  1. does not make a clear visual differentiation between flaky tests and failures, requiring the user to carefully analyze the output
  2. does not link to the report artifact, making it high friction for the user to actually check the report to see what changed
  3. does not have a clear call to action instructing the user what they should do

I believe it is better to have our own custom comments (basically revert #1543 and improve the original comments a bit to properly differentiate between flaky and failure). That is what this PR does.

This achieves:

  • Very small comment size, with just a single icon, making it easy to read and interpret
  • Clear call to action to user instructing them what to do based on results
  • Low friction via a direct link to the playwright-report (made possible by the new v4 artifact system on GitHub)
  • Proper differentiate between flaky and failure, choosing to interpret flaky as ✔️ (but still comment to instruct user to improve tests) and failure as ⚠️ so it's hard to miss.

@djahandarie djahandarie requested a review from a team as a code owner December 28, 2024 04:38
@djahandarie
Copy link
Collaborator Author

Example result: djahandarie#2 (comment)

@djahandarie djahandarie added this pull request to the merge queue Dec 28, 2024
Merged via the queue into master with commit d2bce97 Dec 28, 2024
13 checks passed
@djahandarie djahandarie deleted the improve-playwright-comment branch December 28, 2024 05:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants