Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DB connection issue - remaining connection slots are reserved for non-replication superuser connections #3308

Open
dave-kennedy-ecs opened this issue Jan 7, 2025 · 1 comment
Labels
bug Something that isn't working as intended dev issue is for the dev team

Comments

@dave-kennedy-ecs
Copy link
Contributor

dave-kennedy-ecs commented Jan 7, 2025

Current Behavior

DB connection error in the logs:
django.db.utils.OperationalError: connection to server at "cg-aws-broker-prodjbzr1h2a399wmmm.ci7nkegdizyy.us-gov-west-1.rds.amazonaws.com" (10.10.21.214), port 5432 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections

Error occurrences:
Jan 3, 2025 @ 06:28:53.166
Jan 3, 2025 @ 15:26:52.192
Jan 3, 2025 @ 15:26:55.530
Jan 3, 2025 @ 15:27:00.832
Jan 3, 2025 @ 15:27:01.182

Stack trace for reference:

Internal Server Error: /api/v1/available/
Traceback (most recent call last):
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "cg-aws-broker-prodjbzr1h2a399wmmm.ci7nkegdizyy.us-gov-west-1.rds.amazonaws.com" (10.10.21.214), port 5432 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/vcap/deps/0/python/lib/python3.10/contextlib.py", line 78, in inner
    with self._recreate_cm():
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/transaction.py", line 198, in __enter__
    if not connection.get_autocommit():
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/base/base.py", line 464, in get_autocommit
    self.ensure_connection()
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
    with self.wrap_database_errors:
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/home/vcap/deps/0/python/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "cg-aws-broker-prodjbzr1h2a399wmmm.ci7nkegdizyy.us-gov-west-1.rds.amazonaws.com" (10.10.21.214), port 5432 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections

Expected Behavior

Expect error not to occur in stable

Steps to Reproduce

  1. Unknown - investigate occurrences in logs and draw conclusions about how to reproduce

Environment

stable

Additional Context

The error shown indicates we are hitting a throttle limit for our db connection.

The default behavior of Django is to drop and recreate the database connection for every request, which would cause a new DNS query and eventually hit our cloud.gov/aws throttle limits.

See this django documentation

per the above I think this may be resolved by just setting CONN_MAX_AGE to 30.

Issue Links

No response

@dave-kennedy-ecs dave-kennedy-ecs added the bug Something that isn't working as intended label Jan 7, 2025
@abroddrick abroddrick moved this from 👶 New to 🍦 Backlog in .gov Product Board Jan 9, 2025
@abroddrick abroddrick moved this from 🍦 Backlog to 🎯 Ready in .gov Product Board Jan 9, 2025
@abroddrick abroddrick added the dev issue is for the dev team label Jan 9, 2025
@abroddrick
Copy link
Contributor

Temporarily slotting into 62 as this suggested fix may require merging into production and watching to see the effects. This may take a few iterations so makes since to fix this now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that isn't working as intended dev issue is for the dev team
Projects
Status: 🎯 Ready
Development

No branches or pull requests

2 participants