Investigate the slowdown of SMS while testing in production

[jimleroyer]

Describe the bug

We could not uphold the higher SMS sending limit in a recent performance test in production. We want to make sure that we can uphold it and the system is behaving as expected.

Bug Severity

(SEV-1 Critical, SEV-2 Major, SEV-3 Minor, SEV-4 Low)

SEV-2

To Reproduce

1. Launch performance test in production (4000 fake 613-500-01** numbers.
2. See the send rate be around 530, well below the target 2000.

Expected behavior

The sending rate limit should reach above 2,000 sending rate. But we currently see 500-600 sending rate during the tests which is much lower than our previous results and expectations.

Impact

Describe overall user/system impact to consider when prioritizing this issue.

If applicable

Impact on Notify users: Potential lower SMS sending rate limit.

Impact on Recipients: Slower reception of SMS

Impact on Notify team: Outside normal parameter of the system, leading to less reliable system and less trust and capability for our team.

QA

QA items to verify the bug is resolved:

• Run performance test in production.

Additional context

• In August we determined that a task rate limit of 10/s and 6 sms delivery pods (3 primary and 3 scalable) would reach a send rate of approximately 2000 sms / minute (test: upload a csv of 40000 sms).
• However in December we saw that the same test only sent about 600 sms / minute.