Node out of sync post webhook triggers degrade response times over all endpoints

[jpalvarezl]

Problem Statement

• The current infrastructure usage is not optimized to our requirements safe-global/safe-pm#89

Describe the bug
Whenever we have nodes going out of sync the tx-service builds up a backlog of webhook triggers that are clearly visible in our kibana dashboard for the CGW. As soon as this starts, the worst response times for the CGW degrade drastically.

To Reproduce

Expected behavior
The CGW, having async handlers for all endpoints and a connection pool to redis is expected to handle the increased amount of calls to the webhook endpoint.

Environment (please complete the following information):

• Production, happens on bursts of cache invalidations

Additional context

1. We have currently have the connection pool hardcoded to 15 we should evaluate and see whether this value is appropriate for our setup.
   Rocket worker reference

2. We could also set a request connection timeout in the pool to avoid blocking for too long

3. We should consider a way to mitigate the excessive time it takes to look up keys in our redis setup, in the past we attempted to introduce a multiple database setup that was considered an anti-pattern. We need to find the correct way to reduce the redis response time. Link to issue (see also associated PRs) Redis database id selection for CacheResponse and RequestCached #344

[fmrsabino]

Part of this issue was mitigated via a new Redis instance for mainnet.

However this issue was not fixed and is currently part of a bigger refactor around caching for the Safe Client Gateway.