FE: Audit Platform flaky Cypress test report and ticket/fix issues

[wesrowe]

Description

User story

Describe the audience/user, enhancement or fix, and value / outcome desired.

AS A PM/PO/Eng
I WANT our flaky tests de-flaked
SO THAT Veteran experience is protected and we are allowed to keep merging code.

Engineering notes / background

Top products in the flaky-test report:

• Income Limits
• PACT Act Wizard
• Homepage

Acceptance criteria

• Located Platform flaky-test report in vfs-all-teams
• Addressed all problematic tests in report
  • Evaluated why tests are flaky
  • Ticketed fixes for all flaky tests
  • Fix home page tests
  • (We are open to removing a test, if it can be justified that it's not needed)

[randimays]

@jilladams @wesrowe

I asked Platform about this because I ran some of the PAW and IL tests 100 times locally and they passed 100 times. Curt told me the BigQuery data they're using to create the report is working as expected, but the user-facing portion is not displaying correctly. Curt said he's going to start troubleshooting this right away and hopes to have a fix in, in the next couple of days. I'll put this in blocked for now and check up on it on Wednesday.

See this Platform thread for more info.

[jilladams]

Nice! Love the fact that because we hopped on this quickly, we were able to a) learn our tests may actually be mostly fine, and b) stress test the reporting so Plat can get it cleaned up before the bigger testing deadlines. Thanks, Randi.

[randimays]

Platform thread continues. Curt made some changes and it seems like some of the cruft got cleaned up, and others remain. I'll follow up with him specifically about the ones still hanging around that shouldn't be. But I also still need to poke around at the homepage tests and see what's up there.

[randimays]

@wesrowe @jilladams

In this disallow list, the new-homepage.cypress.spec.js test is failing because we did a redirect from /new-home-page to /. That redirect went in ~Aug 28 but according to the chart, the test has been flaky for nearly a year.

I don't think it's valuable to keep this test spec since it now would test outdated functionality (if we didn't have the redirect), and we plan to update home page tests in the future. I'm going to remove this one from the repo if you agree.

[jilladams]

@randimays I think that makes sense but to be clear: what is the test testing? It's just testing for the existence of the /new-home-page path?

[randimays]

@jilladams Thanks for asking this clarifying question, I wasn't looking at it through the right lens. The test targets /new-home-page, and then tests all of the new functionality that currently exists on the va.gov homepage. We could turn it into a valuable test easily and make sure it's comprehensive. I think that would be a good outcome for this ticket.

[jilladams]

Ok! Yes, would be good to maintain for the real homepage, now that newhp is live!

[randimays]

PR open for updating the /new-home-page tests to test the current home page instead. Waiting for CI to see if it makes it through.

There are still 2 Income Limits Cypress tests on the disallow list. Platform is helping to diagnose but Curt is swamped this sprint and hasn't had a chance to figure out why the tests are still on the report even though the latest run should have fixed them.

[randimays]

Moved the PR for new home page tests to content-build. Turns out the vets-website does not have access to content-build data in its CI, so the full home page tests will not pass there.

Instead, I'm adding thorough tests for the global header and footer to vets-website. This will carry over to next sprint.

[randimays]

Updated the ACs a bit per this Slack conversation.

[randimays]

The content-build PR for the full surface-level homepage tests has been merged and is now running in the CI. No issues so far!

vets-website has a PR as well for thorough header and footer tests. Trying to get through seemingly unrelated CI failures and then we'll be able to review and get that merged.

[randimays]

Updated the vets-website PR to have more robust header and footer tests. Pipeline is passing. Ready for review.

[randimays]

The PR got merged in, but I'm noticing the tests are not running in CI; they're getting skipped because the test that comes before them (unrelated to PW) is causing a runner error, and the runner starts over from the top of the list. See Slack thread. Before closing this ticket I'd like to see a successful run of these tests in the CI.

[randimays]

The Cypress tests for the desktop header and footer are merged in. I ticketed the Cypress tests for the mobile header and footer, and to review the existing unit tests for both (#15699).

[randimays]

All desktop header and footer tests have been merged in and passed CI several times. Will keep an eye on vets-website CI to be sure nothing is flaky. Closing!