Create separate Datadog dashboard for content-build and Content Release reporting

[ndouglas]

Description

We should create a new, separate dashboard for new metrics and reporting for content-build and Content Release.

For instance:

• content release
  • successes
  • failures
  • time between successes
  • time to fail (e.g. durations of failed releases)
  • time to recover from failures
  • releases to recover from failures (different from above)
  • failures / successes as a percentage
  • possibly something as a proxy for CFR since we don't have a general change denominator
• Dependabot security alerts
• PRs opened
• PRs closed
• any other useful information we uncover.

The reason I say a "separate" dashboard is because since content-build is, ah, "between parents," some of these metrics are very unappealing. While we achieve clarity on the team's responsibilities and SLOs/SLAs in this area, we should use this to build an understanding of the current state of the system and not necessarily something we should report as part of team metrics.

Acceptance Criteria

• Datadog dashboard is created or expanded to include the metrics and reporting mentioned above and anything else that comes to mind, within reason.
• Optionally, backport some widgets to CMS reporting dashboard.

[teeshe]

content-build dashboard already taking structure, I would probably update some more build related metric. dsva-vagov-content-build

[BerniXiongA6]

Hi @teeshe -- do we know how many points were completed in Sprint 96 and how many points are left to carry over to Sprint 97? Thanks! cc: @ndouglas

[teeshe]

I have created two dashboards to help visualize the content build metrics and content release status. I would review the created dashboard with team.
content release dashboard

[teeshe]

Content build dashboard was populated with build metrics from content release jobs, metrics includes;
https://vagov.ddog-gov.com/dashboard/k57-r4p-u52/dsva-vagov-content-build?refresh_mode=sliding&from_ts=1697382208787&to_ts=1699974208787&live=true

1. Content build duration
2. build environment setup time
3. GraphQL pages count
4. GraphQL query duration
5. Build archive duration
6. deployment duration
   Private Zenhub Image