Add monitoring of prod to alert if injected header / footer fail during injection

[swirtSJW]

User Story or Problem Statement

As a maintainer of injected headers and footer, I want to be alerted if a defect occurs on prod teamsite injections so that I don't have to rely on others to report the issue.

Given the number of sites and subsites that rely on the injected header and footer, it is vital that we know when they fail in production so monitoring of prod is needed.

Assumptions

• DataDog has visual monitoring capability, per @jtmst.
• Have to get access to Datadog. (Thread) & other thread
• Using a synthetic test, to verify header on 2 example subdomains:
  • https://benefits.va.gov/benefits/
  • https://www.va.gov/health/

Acceptance Criteria

• Get Datadog access
• If there is a visual failure of the injected header it is logged and notified.
• If there is a visual failure of the injected footer it is logged and notified.
• Documentation is updated.

Implementation steps

Visual regression testing could be very useful in this.

Design principles

Veteran-centered

• Single source of truth: Increase reliability and consistency of content on VA.gov by providing a single source of truth.
• Accessible, plain language: Provide guardrails and guidelines to ensure content quality.
• Purposely structured content: Ensure Content API can deliver content whose meaning matches its structure.
• Content lifecycle governance: Produce tools, processes and policies to maintain content quality throughout its lifecycle.

Editor-centered

• Purpose-driven: Create an opportunity to involve the editor community in VA’s mission and content strategy goals.
• Efficient: Remove distractions and create clear, straightforward paths to get the job done.
• Approachable: Offer friendly guidance over authoritative instruction.
• Consistent: Reduce user’s mental load by allowing them to fall back on pattern recognition to complete tasks.
• Empowering: Provide clear information to help editors make decisions about their work.

CMS Team

Please check the team(s) that will do this work.

• Program
• Platform CMS Team
• Sitewide Crew
• ⭐️ Sitewide CMS
• ⭐️ Public Websites
• ⭐️ Facilities
• ⭐️ User support

[wesrowe]

Removed estimate since the ACs need refinement.

Per DaveC, injected is higher prio than non-injected.

[jilladams]

@chriskim2311 For your reference re: injected header: https://github.com/department-of-veterans-affairs/vets-website/blob/main/src/applications/proxy-rewrite/

[jilladams]

Note: once #15195 and #12978 are refined tomorrow, this ticket might get swapped out in favor of one of those. Pursuing datadog access is smart, but other sprint tickets should maybe come first for coding in case this one gets deprioritized.

[chriskim2311]

Some Insights from doing this test in Datadog

Datadog Browser Synthetic Tests

Datadog has browser tests that can be run directly in the browser against production websites. The use-case for these tests can be to verify actual user-flow and elements that are currently in production. Tests can target and verify elements, buttons, input and text, navigation on the FE of a webpage.

Setting up the tests

Test Details
Private Zenhub Image

1. Add starting URL
2. Name the test
3. Include details about Team, and Environment
4. Indicate browsers and devices to run the test against

Private Zenhub Image

1. Select locations to run the test, ie: prod, dev, staging
2. Can define retry conditions for a failed test
3. Define recurring schedule for the test

Private Zenhub Image

1. Configure the monitor alert for the test. Can be sent via slack or email. If setting up a new slack alert that has not been set up yet. Will need to set up the channel in the datadog slack integration before adding it here.
2. Set Priority
3. Set permissions if needed.
4. Save the test details

Setting up recording
Private Zenhub Image

1. When setting up the recording you can click Start Recording and open the pop up window.
2. You can select elements while you're recording and can record any interactions with the page.
3. You can also set up assertions to verify elements are on the page, verify contents, verify URL, etc.
4. Other things to consider that can be important. Wait can be used for things that may not load quick initially or if you want to test a link you can wait for it to load. Hover, Scroll, Press Key are other special assertions that could be useful when trying to capture certain actions.
5. After setting up your test it will make an initial run, then will run based on your initial test settings.

[chriskim2311]

@jilladams @wesrowe

@randimays We can go over the notes if you have any questions!

I have left some quick notes for setting up a new synthetic browser test.

Here are the current tests that have been created.

https://benefits.va.gov/benefits/
https://vagov.ddog-gov.com/synthetics/details/yms-fpg-s8t

https://www.va.gov/health/
https://vagov.ddog-gov.com/synthetics/details/a5r-z2x-qzv

[jilladams]

Nice!

[jilladams]

@chriskim2311 took a deeper look at these today, and it's making sense to me. I'm content that ACs are met, and we've established we'll have you do a Topic Dive on 10/16 to share notes about the Data dog learnings / put these findings int he Topic Dive documentation for future. Closing!