Monitor celery tasks in cloudwatch (PP-1150)

[jonathangreen]

Description

Add new custom metrics for Celery tasks and queues in Cloudwatch:

Tasks:

• TaskFailed
• TaskSucceeded
• TaskRuntime

Queues:

• QueueWaiting

Motivation and Context

Allow us to monitor our Celery queues via Cloudwatch.

How Has This Been Tested?

• Tested locally
• Tested in CI

This one is tough to test fully locally, so it may need some ineration after actually being deployed to AWS and fully pushing metrics into cloudwatch.

Checklist

• I have updated the documentation accordingly.
• All new and existing tests passed.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.04%. Comparing base (d6c4019) to head (75e0a6b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1813      +/-   ##
==========================================
+ Coverage   90.01%   90.04%   +0.02%     
==========================================
  Files         299      300       +1     
  Lines       39643    39742      +99     
  Branches     8596     8615      +19     
==========================================
+ Hits        35686    35784      +98     
- Misses       2626     2627       +1     
  Partials     1331     1331              

Flag	Coverage Δ
manager	89.84% <100.00%> (+0.02%)	⬆️
migration	24.56% <15.74%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dbernstein]

Looks good to me.

[dbernstein]

Assuming you fix the mypy issues first.

[dbernstein]

@jonathangreen : would it be helpful also to aggregate stats based on the type of task? It wasn't completely clear to me if the task type is being captured. If it is not being captured, it would be helpful to get a sense of the task run time based on the type of task. That way we would be able to monitor performance in a more granular way if necessary.

For example, if a change was introduced to a query within a job that started to cause a bottleneck we would be in a better position to troubleshoot it.

[jonathangreen]

@jonathangreen : would it be helpful also to aggregate stats based on the type of task? It wasn't completely clear to me if the task type is being captured. If it is not being captured, it would be helpful to get a sense of the task run time based on the type of task. That way we would be able to monitor performance in a more granular way if necessary.

@dbernstein The stats we push have a dimension for the task name. So per task name we get the number of tasks that ran, failed and the task runtime. I think this is what you are asking for. If not, what is it that you would use to group tasks into "type"?

[jonathangreen]

I'm going to merge this one since the tests are passing, so I can test it out on Minotaur tomorrow. @dbernstein if there are additional stats you want captured or dimensions added to the stats lets discuss on PP-1150. Can either add them as a follow up on that ticket.