Fix flaky cypress tests

[Sabr1n4W]

Fixes #

Type (Highlight the corresponding type)

• Bugfix
• Feature
• Documentation
• Refactoring (e.g. Style updates, Test implementation, etc.)
• Other (please describe):

Checklist

• Code updated to current develop branch head
• Passes CI checks
• Is a part of an issue
• Tests added for the bugfix or newly implemented feature, describe below why if not
• Changelog is updated
• Documentation of code and features exists

Changes

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 54.27%. Comparing base (896dc40) to head (878868c).

❗ There is a different number of reports uploaded between BASE (896dc40) and HEAD (878868c). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (896dc40)	HEAD (878868c)
	6	2

Additional details and impacted files

@@              Coverage Diff               @@
##             develop    #1645       +/-   ##
==============================================
- Coverage      81.05%   54.27%   -26.79%     
  Complexity      1340     1340               
==============================================
  Files            361      361               
  Lines           9223     9223               
  Branches         851      851               
==============================================
- Hits            7476     5006     -2470     
- Misses          1747     4217     +2470     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cypress]

PILOS    Run #1457

Run Properties:   Passed #1457  •  878868c0e8: Fix flaky cypress tests

Project	PILOS
Branch Review	fix-flaky-cypress-tests
Run status	Passed #1457
Run duration	11m 00s
Commit	878868c0e8: Fix flaky cypress tests
Committer	Sabrina Wüst
View all properties for this run ↗︎

---

Test results
Failures	0
Flaky	0
Pending	215
Skipped	0
Passing	112
View all changes introduced in this branch ↗︎

[Sabr1n4W]

Overview

1. Run (no modifications)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 92 passed, 8 failed)
• RoomsViewDescription
  • edit description (100 runs, 97 passed, 3 failed)
• RoomsViewSettings
  • transfer ownership (100 runs, 97 passed, 3 failed)

2. Run (no modifications)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 96 passed, 4 failed)
• RoomsViewDescription
  • edit description (100 runs, 100 passed, 0 failed)
• RoomsViewSettings
  • transfer ownership (100 runs, 91 passed, 9 failed)
• RoomsViewMemberActions
  • add new member (100 runs, 100 passed, 0 failed)

3. Run (modifications RoomsViewSettings, RoomsViewMemberActions)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 94 passed, 6 failed)
• RoomsViewDescription
  • edit description (150 runs, 150 passed, 0 failed) (Increased number of runs)
• RoomsViewSettings
  • transfer ownership (100 runs, 100 passed, 0 failed)
• RoomsViewMemberActions
  • add new member (100 runs, 100 passed, 0 failed)

4. Run (change check order, add selection check, add click before select)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 98 passed, 2 failed)
• RoomsViewDescription
  • edit description (150 runs, 150 passed, 0 failed)
• RoomsViewSettings
  • transfer ownership (100 runs, 100 passed, 0 failed)
• RoomsViewMemberActions
  • add new member (100 runs, 100 passed, 0 failed)

5. Run (type before selectall)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 100 passed, 0 failed)
• RoomsViewDescription
  • edit description (150 runs, 149 passed, 1 failed)
• RoomsViewSettings
  • transfer ownership (100 runs, 100 passed, 0 failed)
• RoomsViewMemberActions
  • add new member (100 runs, 100 passed, 0 failed)

6. Run (no new changes)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 100 passed, 0 failed)
• RoomsViewDescription
  • edit description (150 runs, 150 passed, 0 failed)
• RoomsViewSettings (Skipped)
• RoomsViewMemberActions (Skipped)

7. Run (check class before selectall)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 100 passed, 0 failed)
• RoomsViewDescription
  • edit description (200 runs, 200 passed, 0 failed) (Increased number of runs)
• RoomsViewSettings (Skipped)
• RoomsViewMemberActions (Skipped)

8. Run (remove typing before selectalll)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 100 passed, 0 failed)
• RoomsViewDescription
  • edit description (200 runs, 200 passed, 0 failed)
• RoomsViewSettings (Skipped)
• RoomsViewMemberActions (Skipped)

9. Run (change back check order)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 100 passed, 0 failed)
• RoomsViewDescription
  • edit description (200 runs, 200 passed, 0 failed)

10. Run (remove unneccessary clicks)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 89 passed, 11 failed)
• RoomsViewDescription
  • edit description (200 runs, 183 passed, 17 failed)

11. Run (Revert: remove unneccessary clicks)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 100 passed, 0 failed)
• RoomsViewDescription
  • edit description (200 runs, 200 passed, 0 failed)

12. Run (Add missing checks)

• RoomsViewDescriptionTipTapEditor
  • all tests (50 runs, 50 passed, 0 failed)
• RoomsViewDescription
  • edit description (200 runs, 200 passed, 0 failed)

13. Run (Switch to checking focus)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 100 passed, 0 failed)
• RoomsViewDescription
  • edit description (200 runs, 200 passed, 0 failed)

14. Run (Use fewer checks to better represent user actions)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 89 passed, 11 failed)
• RoomsViewDescription
  • edit description (200 runs, 0 passed, 200 failed)

15. Run (Add waiting)

• RoomsViewDescriptionTipTapEditor
  • edit link (50runs, 50 passed, 0 failed) reduced number of runs
• RoomsViewDescription
  • edit description (50 runs, 0 passed, 50 failed) reduced number of runs

16. Run (Add additional checks for text selection)

• RoomsViewDescriptionTipTapEditor
  • edit link (50runs, 50 passed, 0 failed)
• RoomsViewDescription
  • edit description (50 runs, 0 passed, 50 failed)

17. Run (Add additional waiting)

• RoomsViewDescriptionTipTapEditor
  • edit link (50runs, 49 passed, 1 failed)
• RoomsViewDescription
  • edit description (50 runs, 50 passed, 0 failed)

18. Run (Remove waiting and add mor focus checks)

• RoomsViewDescriptionTipTapEditor
  • edit link (50 runs, 45 passed, 5 failed)
• RoomsViewDescription
  • edit description (1 runs, 0 passed, 1 failed)

19. Run (Readd waiting)

• RoomsViewDescriptionTipTapEditor
  • edit link (50 runs, 50 passed, 0 failed)
• RoomsViewDescription
  • edit description (50 runs, 50 passed, 0 failed)

19. Run (Remove waiting (edit description)

• RoomsViewDescriptionTipTapEditor
  • edit link (50 runs, 43 passed, 7 failed)
• RoomsViewDescription
  • edit description (100 runs, 100 passed, 0 failed)

20. Run (Remove waiting (edit link)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 85 passed, 15 failed)
• RoomsViewDescription
  • edit description (100 runs, 100 passed, 0 failed)

21. Run (Add additional typing (edit link)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 100 passed, 0 failed)
• RoomsViewDescription
  • edit description (100 runs, 100 passed, 0 failed)

21. Run (Check with p)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 88 passed, 12 failed)
• RoomsViewDescription
  • edit description (100 runs, 100 passed, 0 failed)

22. Run (Check link with contains)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 100 passed, 0 failed)
• RoomsViewDescription
  • edit description (100 runs, 100 passed, 0 failed)

23. Run (Change other link tests)

• RoomsViewDescriptionTipTapEditor
  • edit link (50 runs, 50 passed, 0 failed)
  • add link (50 runs, 49 passed, 1 failed)
  • delete link (50 runs, 47 passed, 3 failed)
• RoomsViewDescription
  • edit description (100 runs, 100 passed, 0 failed)

24. Run (Change other link tests)

• RoomsViewDescriptionTipTapEditor
  • edit link (50 runs, 50 passed, 0 failed)
  • add link (50 runs, 47 passed, 3 failed)
  • delete link (50 runs, 50 passed, 0 failed)
• RoomsViewDescription
  • edit description (100 runs, 100 passed, 0 failed)

25. Run (Change other link tests)

• RoomsViewDescriptionTipTapEditor
  • edit link (50 runs, 50 passed, 0 failed)
  • add link (50 runs, 50 passed, 0 failed)
  • delete link (50 runs, 50 passed, 0 failed)
• RoomsViewDescription
  • edit description (100 runs, 100 passed, 0 failed)

26. Run (Skip edit description test and rerun link tests)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 100 passed, 0 failed)
  • add link (100 runs, 100 passed, 0 failed)
  • delete link (100 runs, 100 passed, 0 failed)

27. Run (Take back some changes for edit link test)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 83 passed, 13 failed)

28. Run (Rerun tests)

• RoomsViewDescriptionTipTapEditor
  • edit link (110 runs, 94 passed, 16 failed)

29. Run (Retry selection)

• RoomsViewDescriptionTipTapEditor
  • edit link (110 runs, 110 passed, 0 failed)

30. Run (Improve text selection)

• RoomsViewDescriptionTipTapEditor
  • edit link (110 runs, 110 passed, 0 failed)

31. Run (Move select function to seperate file)

• RoomsViewDescriptionTipTapEditor
  • edit link (100 runs, 100 passed, 0 failed)

32. Run (Use select function in other tests)

• RoomsViewDescriptionTipTapEditor
  • edit link (50 runs, 50 passed, 0 failed)
  • add link (50 runs, 50 passed, 0 failed)
  • delete link (50 runs, 50 passed, 0 failed)

33. Run (Use select function in other tests)

• RoomsViewDescription
  • edit description (100 runs, 87 passed, 13 failed)

34. Run (Add function to clear tiptap content)

• RoomsViewDescription
  • edit description (100 runs, 100 passed, 0 failed)

[Sabr1n4W]

Tests failed sometimes because of timing issues. Main problem when fixing these tests is that tests never failed locally.

RoomsViewSettings (transfer ownership)

Failed because autofocus was not considered inside the tests.
Was modified to check autofocus.

RoomsViewMemberActions (add new member)

Test did not fail but autofocus was also not considered here.
Was modified to check autofocus.

RoomsViewDescription, RoomsViewDescriptionTipTapEditor (edit description, edit link, add link, delete link)

Mainly 2 reasons for failing tests. Second one mostly became an issue when unneccessary clicks and selections inside the tiptap where removed because they were not really needed and did not really represent the user actions.

1. Apperently the tiptap sometimes was not fully loaded / did not have the focus / could not be typed into when trying to type into it. Because of that the selection (selectall) did not select the text inside the tiptap and therefore the tests failed

• Changes to fix this? (Seems to work but could be because of longer time needed for checks)
  • Check edit button not visible after click
  • Check that tiptap has focus when checking that it is visible
  • Check Content with contains() (test fails because of timing when should("have.text", "...") is used)
    • not completely sure why this fixes the failing tests but could be that it just takes longer and gives the tiptap enough time to load properly
    • Currently no other solution found (checking focus or class that is set when clicking inside tiptap did not really prevent the tests from failing)
  • Check that selection is correct (Use cy.window().should((win) => { ... to prevent second reason for failing tests)
• Better solution
  • Retry selecting the tiptap content till the content was successfully selected

2. Focus and selection need some time to be applied again after performing an action (e.g. change Text Type)

• Changes to fix this
  • Add checks for selection
  • Use cy.window().should((win) => { ... so that checks are retried until the selection was applied