Validate New Relic in ingesting data from Tock

[edwintorres]

Description:

During the Tock Weekly Log Audit, we observed that no alerts have been generated since March of this year. This issue aims to:

Confirm the absence of alerts since March.
Ensure that New Relic is correctly ingesting data from Tock.

Tasks:

• Validate the absence of alerts since March.
• Verify that New Relic is properly ingesting data from Tock.

[nateborr]

In New Relic, via a manual query I've confirmed the last date that we recorded metric data from any of the Tock apps:

• staging: March 16, 2024
• production: April 2, 2024

We are still recording Deployment events for both applications: https://onenr.io/0VjYrayr2Q0 If the deployments are being reported on the application's behalf by its build pipeline, that would explain why we have that data without the APM data we'd expect to get from the application instances.

I don't see any alert data in the system at all at this point, although the data from this spring may be past its data retention period. I can't directly confirm the data retention policies as a basic user.

I'll keep digging in later this week. The existence of the deployment data likely rules out issues in the account configuration or subscription on the New Relic side. Since there are a couple of weeks between the last-reported dates for the Tock staging and production applications, it looks like a change that was rolled out to the lower environment first, although license key issues could be another culprit.

[nateborr]

This looks like it's related to the egress proxy configuration for #1695. That was released to production on April 2, 2024, which is the day we stopped receiving APM data in New Relic.

Per New Relic's documentation, their agents send data to endpoints at collector.newrelic.com. If we add that domain to our egress proxy allow list I think we'll be in good shape.

We are continuing to receive Deployment events because Tock's run.sh script sends them via the newrelic_admin script. That admin script creates deployments via the New Relic REST API and in #1734 we allowed our egress proxy to send that traffic to api.newrelic.com.

[nateborr]

I've deployed a new release to production and upgraded the egress proxy settings in both staging and production to enable New Relic reporting in both environments. (See the linked PRs for technical details.)

This release also deployed @peterrowland's update for #1770 and all accumulated package updates since the last release on June 11, 2024.