You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
DB connection error in the logs:
django.db.utils.OperationalError: connection to server at "cg-aws-broker-prodjbzr1h2a399wmmm.ci7nkegdizyy.us-gov-west-1.rds.amazonaws.com" (10.10.21.214), port 5432 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections
Error occurrences:
Jan 3, 2025 @ 06:28:53.166
Jan 3, 2025 @ 15:26:52.192
Jan 3, 2025 @ 15:26:55.530
Jan 3, 2025 @ 15:27:00.832
Jan 3, 2025 @ 15:27:01.182
Stack trace for reference:
Internal Server Error: /api/v1/available/
Traceback (most recent call last):
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "cg-aws-broker-prodjbzr1h2a399wmmm.ci7nkegdizyy.us-gov-west-1.rds.amazonaws.com" (10.10.21.214), port 5432 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/vcap/deps/0/python/lib/python3.10/contextlib.py", line 78, in inner
with self._recreate_cm():
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/transaction.py", line 198, in __enter__
if not connection.get_autocommit():
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/base/base.py", line 464, in get_autocommit
self.ensure_connection()
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
with self.wrap_database_errors:
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/home/vcap/deps/0/python/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "cg-aws-broker-prodjbzr1h2a399wmmm.ci7nkegdizyy.us-gov-west-1.rds.amazonaws.com" (10.10.21.214), port 5432 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections
Expected Behavior
Expect error not to occur in stable
Steps to Reproduce
Unknown - investigate occurrences in logs and draw conclusions about how to reproduce
Environment
stable
Additional Context
The error shown indicates we are hitting a throttle limit for our db connection.
The default behavior of Django is to drop and recreate the database connection for every request, which would cause a new DNS query and eventually hit our cloud.gov/aws throttle limits.
Temporarily slotting into 62 as this suggested fix may require merging into production and watching to see the effects. This may take a few iterations so makes since to fix this now.
Current Behavior
DB connection error in the logs:
django.db.utils.OperationalError: connection to server at "cg-aws-broker-prodjbzr1h2a399wmmm.ci7nkegdizyy.us-gov-west-1.rds.amazonaws.com" (10.10.21.214), port 5432 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections
Error occurrences:
Jan 3, 2025 @ 06:28:53.166
Jan 3, 2025 @ 15:26:52.192
Jan 3, 2025 @ 15:26:55.530
Jan 3, 2025 @ 15:27:00.832
Jan 3, 2025 @ 15:27:01.182
Stack trace for reference:
Expected Behavior
Expect error not to occur in stable
Steps to Reproduce
Environment
stable
Additional Context
The error shown indicates we are hitting a throttle limit for our db connection.
The default behavior of Django is to drop and recreate the database connection for every request, which would cause a new DNS query and eventually hit our cloud.gov/aws throttle limits.
See this django documentation
per the above I think this may be resolved by just setting CONN_MAX_AGE to 30.
Issue Links
No response
The text was updated successfully, but these errors were encountered: