Investigate Why Prod Taken Offline

[gracekretschmer-metrostar]

User Story or Problem Statement

As CMS DevOps staff, I need to know why prod.cms.va.gov was unexpectedly taken offline on 2/13/2024, so that I prevent that from happening again in the future.

Description or Additional Context

On 2/13/2024, Tim Cosgrove (Accelerated Publishing PM) notified the CMS team that prod.cms.va.gov was down since 11pm EST. No one was directly notified of the outage, Tim Cosgrove noticed outage notifications in the DSVA slack channel: #cms-notifications. The outgoing CMS Tech Lead (Nathan Douglas) was able to identify that prod.cms.va.gov was down because it was removed from service and resolve the issue by performing an emergency prod deploy.

Acceptance Criteria

• The reason why prod was removed from service is understood.
• Opportunities to prevent prod from being removed from service are identified.

Team

Please check the team(s) that will do this work.

• CMS Team
• Public Websites
• Facilities
• Accelerated Publishing

[7hunderbird]

With the help of @edmund-dunn and @Hassantariq-MetroStar we investigated what happened and identified the reason. We will report in #17244 the results.

In brief too many database connections were overloading the prod vm and caused it to crash.