Update some of the configuration defaults for celery (PP-1279)

[jonathangreen]

Description

Update a couple of the defaults for our celery worker config. I determined these values by overriding the config with env variables on the CA CM to determine their impact on performance. Since these seem to drastically help our performance, I'm updating the defaults.

Motivation and Context

The CA CM has been struggling to keep up with the number of indexing tasks coming in. Now that we have proper metrics, its a bit easier to see whats happening there. We were getting about 1000 indexing tasks coming in every minute, and we were able to process about 500 of them every minute. So the indexing process was continually under water.

Looking at: https://docs.celeryq.dev/en/stable/userguide/optimizing.html#prefetch-limits

And updating some of our config values, the CA CM is able to process about 2700 tasks a minute, which allows the indexing task to keep up with demand.

Since we have some long running tasks, we may want to follow the suggestion in that guide, and add another queue with different workers with different prefetch limits. I'm going to make a ticket to look into that, but this configuration has been helping the CA CMs, so I think its worth making right away.

How Has This Been Tested?

• Tested with ENV variable overrides on the CA CM

Checklist

• I have updated the documentation accordingly.
• All new and existing tests passed.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.14%. Comparing base (7777f85) to head (93977df).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1874      +/-   ##
==========================================
- Coverage   90.14%   90.14%   -0.01%     
==========================================
  Files         323      323              
  Lines       39634    39634              
  Branches     8619     8619              
==========================================
- Hits        35730    35728       -2     
- Misses       2594     2595       +1     
- Partials     1310     1311       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tdilauro]

👍🏽

[jonathangreen]

Closing this one, since it doesn't seems to have been unnecessary. It was restarting the container to make these changes apply that increased the indexing speed, not the env vars I set 🤦🏻