2/23/2024 Build and Deploy Breakdown Postmortem

[gracekretschmer-metrostar]

Background

On 2/23/2024, Christian Burk (Public Facilities Drupal Engineer) notified the CMS team that staging.cms.va.gov did not operate as expected and displayed an error message (see below). Soon after, the #cms-notifications channel began displaying error messages about the deploy process for BRD.

User Story or Problem Statement

The root of the staging.cms.va.gov and deploy errors need to be fully understood to be able to create a solution that will full address the problems and give the CMS team an understanding of more sustainable long term solutions.

Affected users and stakeholders

• Developers of the public websites product team
• Developers of the facilities product team
• Developers of the cms team

Hypothesis

The errors were caused during a failure in the build process due to a missing token from Gitlab. Creating a login and token for Gitlab will resolve the errors from both staging.cms.va.gov and BRD.

Acceptance Criteria

• The root cause of why staging.cms.va.gov was displaying the error message and BRD was producing errors is fully understood.
• A solution is implemented that resolves the current errors for both staging.cms.va.gov and BRD.
• Opportunities to proactively prevent the staging.cms.va.gov and BRD errors from happening in the future are identified.

Team

Please check the team(s) that will do this work.

• CMS Team
• Public Websites
• Facilities
• User support
• Accelerated Publishing

[gracekretschmer-metrostar]

Demo link here: https://docs.google.com/presentation/d/1cXv93lyyeh36RbPvr8NHmcynOG4TMW2z/edit?usp=sharing&ouid=105824821536597668866&rtpof=true&sd=true