Broken Sidekiq <> Datadog Integration

[LindseySaari]

We encountered an issue with the official Datadog Sidekiq integration on our Va.gov API after upgrading to version 5.6.1. Metrics (only for Sidekiq) stopped coming into Datadog. After downgrading back to 5.6.0, the metrics resumed as expected. For context, our Sidekiq pods operate within EKS. Additionally, we followed the direct Datadog instructions for setting up the Datadog <> Sidekiq Integration.

Ticket opened here in the gem repo and the maintainers have asked us to open a ticket because it may be related to our Sidekiq-Pro integration

[LindseySaari]

I opened an official issues with Datadog

[LindseySaari]

Private Zenhub Image

Link to the ticket opened up with Datadog: https://help.datadoghq.com/hc/requests/1414562

[LindseySaari]

In communication with a Datadog engineer. I am working through some back and fourth and discussing the necessary config for engineers on the Datadog side to analyze the issues between the working (5.6.0) and broken (5.6.1) versions for the sidekiq metrics

[laineymajor]

Looking at Monday or Tuesday BEFORE daily sync to act on this... TBD.

To do

• research how to send a flare

[laineymajor]

Lindsey is actively working this ticket today.
Changes are in review with appropriate team(s).

[laineymajor]

Waiting on datadog to analyze flare that was sent.

[laineymajor]

Waiting on DD. This is carrying over to new sprint.

[LindseySaari]

Datadog is still analyzing the logs this afternoon. The Solutions engineer assigned to the ticket will pass along any updates

[LindseySaari]

Datadog is still analyzing the logs this afternoon. The Solutions engineer assigned to the ticket will pass along any updates

[LindseySaari]

Update from Datadog

"Just reaching out to give you an update on this one. The team is still reviewing this ticket on our side and will let me know when they have next steps. In the meantime, feel free to reach out if you have any other questions on this ticket."

[LindseySaari]

I heard back from Datadog and they need us to execute a few more steps to help with the debugging process. I will aim for getting these changes into staging Friday morning and executing the necessary commands. It's important to start early to maximize the amount of time before the production deploy. Once the necessary information is gathered, this change will need to be reverted and the agent restarted before it goes on the conveyor belt to production.

Steps

1. Merge in dogstatsd gem update - make sure it deploys
2. Merge in the datadog-agent change. (This should autosync)
3. Make sure it syncs
4. Restart the agent: kubectl rollout restart daemonset datadog-agent
5. Run command kubectl exec -it ds/datadog-agent -- agent dogstatsd-stats
6. Run tcp dump: kubectl exec -it ds/datadog-agent -- tcpdump -i any "udp port 8125" -w output.pcap
7. Provide these reports to datadog
8. Revert Vets API and Datadog agent change
9. Merge & Verify Deploy/Sync
10. Restart the agent: kubectl rollout restart daemonset datadog-agent (edited)

[LindseySaari]

I adjusted the config this morning and ran the tcpdump command. After that, I copied the output/files to my local machine and forwarded them to Datadog.

[laineymajor]

Adding this to our next sprint as we need some additional devops help to move this work forward to close.

[LindseySaari]

Still following up with Datadog. They were mistaken on where the "proxy" resided in our setup. They thought it sat between the agent and the datadog endpoint, but it's actually more on the frontend of things where we use socat to proxy metrics from our rails app to the agent

[laineymajor]

@flooose to review ticket and sync with Lindsey if needed.

[LindseySaari]

Chris is looking at a possible workaround to test the issue here

[laineymajor]

Need time to test in the cluster

[RachalCassity]

Datadogstatsd 5.6.1 was deployed to prod.

Host 127.0.0.1 enforces IPv4 connection

[LindseySaari]

This has been fixed via the change to force IPv4 via 127.0.0.1