2/13/2024 Prod Offline Incident Post Mortem

[gracekretschmer-metrostar]

Background

On 2/13/2024, Tim Cosgrove (Accelerated Publishing PM) notified the CMS team that prod.cms.va.gov was down since 11pm EST. No one was directly notified of the outage, Tim Cosgrove noticed outage notifications in the DSVA slack channel: #cms-notifications. The outgoing CMS Tech Lead (Nathan Douglas) was able to identify that prod.cms.va.gov was down because it was removed from service and resolve the issue by performing an emergency prod deploy.

User Story or Problem Statement

How might we understand why prod.cms.va.gov was unexpectedly removed from service and prevent that from unknowingly happening in the future?

Assumptions

Prod.cms.va.gov and it's replacement instance were removed from service because it failed health checks.

Tasks

• Investigate Why Prod Taken Offline #17255

Acceptance Criteria

• The reason why prod was removed from service is understood.
• Opportunities to notify CMS staff before prod is removed from service are identified.
• Opportunities to prevent prod from being removed from service are identified.
• A post mortem report is submitted in Github.

Team

Please check the team(s) that will do this work.

• CMS Team
• Public Websites
• Facilities
• User support
• Accelerated Publishing

[gracekretschmer-metrostar]

[gracekretschmer-metrostar]

Action items from 2/21/2024 meeting:

• @EWashb will begin the process of creating a SLA for the new Oddball CMS team and pull in @gracekretschmer-metrostar for next steps.
• @gracekretschmer-metrostar will work with @michelle-dooley and @EWashb on defining the structures and definitions for escalations for CMS incidents in pager duty.
• @Hassantariq-MetroStar and @7hunderbird will continue to investigate to understand if their hypothesis that an increase of database connections is the root cause of the outage.
• @Hassantariq-MetroStar and @garnertb will continue to prioritize gaining access to VA systems and will document blockers in their onboarding tickets.

Slide deck from the meeting is attached to this comment.

Prod Offline Postmortem 02212024 (Oddball Template).pptx

[7hunderbird]

@gracekretschmer-metrostar myself and Hassan would be blocked on continuing to investigate if apache failed because of database connections or if database connections were high because apache failed, until we get more access to things and we add the recommendations from the post-mortem as tickets and get the work done in those tickets.

As mentioned in the Post-mortem PR:

• Discovery ticket to answer, "What failed in post-deploy scripts that prevented automatic healing of the instance?"
  • Hassan would like us to consider using a 2nd standby instance in the auto-scaling group
• Discovery ticket to answer, "What logs can we turn on at the operating system level?"
• Discovery ticket to answer, "What logs or monitoring do we want to turn on at the Amazon Web Services (AWS) level?"
• Discovery ticket to answer, "What is the current version of Apache? What is available now? What benefits would upgrading Apache provide?"

@gracekretschmer-metrostar @michelle-dooley I'm thinking I'd put these into issues in the backlog of our Project?

• https://github.com/orgs/department-of-veterans-affairs/projects/1194/views/1

[7hunderbird]

Submitted Post-mortem final draft to: https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/pull/1513

[gracekretschmer-metrostar]

Sent postmortem to Erika to review. https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/pull/1513/commits/9708c431774d4091b7090d5e9304b948bf6e02eb

[EWashb]

Approved