Scale up the callback processing

[jimleroyer]

Description

As a user of GCNotify,
I want to know when my email or SMS were sent or delivered in a timely manner when using the API callback feature,
So that I get rapid feedback on the notification's sending and delivery
And react accordingly on success or failure.

WHY are we building?

The callback processing is not scaling up as much as what we can currently send, triggering potential traffic in our callback queue as the celery workers cannot process as fast and introducing delays in communicating callbacks to API users.

WHAT are we building?

Without dictating how we can do this, the what is about scaling up the callback queue and Celery workers with a similar factor to the sending process.

VALUE created by our solution

Better user experience on proper notification status and callbacks.

Acceptance Criteria

• An increased volume of notifications sending should trigger an appropriate scale of the API callback processing.
• An increased volume of callback processing should not impede other normal operations handled by Celery workers such as the database saving operations (performed by Celery workers) or the actual sending of the notifications (performed by Celery workers).
• callbacks queue should not increase after implementing this feature

QA Steps

• Perform Rollercoaster tests and monitor the processing of callbacks.
• Performance test ( send 50K emails) doesn't show any additional slowdowns