Monitoring: We'd like to know when PPMS is not functioning

[eselkin]

User Story or Problem Statement

Create datadog alert for error logs when PPMS is down

Description or Additional Context

Steps for Implementation

Acceptance Criteria

• Testable_Outcome_X
• Testable_Outcome_Y
• Testable_Outcome_Z
• Requires design review
• Requires accessibility review

Team

Please check the team(s) that will do this work.

• CMS Team
• Public Websites
• Facilities
• Accelerated Publishing

[jilladams]

We don't have access to know if PPMS is giving errors.
The request to vets-api will always return 200 even if PPMS signals an error. e.g. PPMS may send a 400 (warning) or 404 (also a warning) when the request has no data. But the request through vets-api returns 200 (success) and says empty data.

If we ever wanted to create this monitor, we would first need a vets-api ticket to better handle PPMS errors at that level to send back 400/404, rather than just 200s. (a la this work on Search to add 503/504 handling: https://github.com/department-of-veterans-affairs/vets-api/pull/15337/files) This is likely a product question: do we want to better process errors to show users, for example "no providers are available in this bounding box". That's the sort of thing that would return a 400.

If PPMS is down, though, we would be getting errors (500s). That never happens. PPMS is very slow, and will timeout or need 1min to get a response.