How to make Sentry with OTel integration work in gunicorn + gevent ?

[awoimbee]

Environment

SaaS (https://sentry.io/)

What are you trying to accomplish?

I want to make Sentry with gunicorn work (see benoitc/gunicorn#1855), with the additional complexity that I want the Sentry <> OTel integration.

How are you getting stuck?

Following the python OTel docs, I setup OTel in gunicorn.conf.py, in post_fork().

But I can't import sentry_sdk and sentry_sdk.init() in post_fork() because sentry uses sockets and gunicorn patches sockets with patch_all() after post_fork(), right before post_worker_init(). So I need to put sentry_sdk.init() in post_worker_init().

But I can't move my OTLPSpanExporter() to post_worker_init() because then, when the OTel endpoint is down, my API stops responding (because of Transient error StatusCode.UNAVAILABLE encountered while exporting traces to <endpoint>, retrying in <many>s.).

All in all, my current gunicorn.conf.py is:

import os
from importlib.metadata import version

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.propagate import set_global_textmap
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

bind = "0.0.0.0:8080"
wsgi_app = "myapp.entrypoint"

worker_class = "gevent"
workers = int(os.getenv("GUNICORN_WORKERS", "2"))
worker_connections = int(os.getenv("GUNICORN_WORKER_CONNECTIONS", "1000"))
max_requests = 5000
max_requests_jitter = 1000
preload_app = False
timeout = 120
keepalive = 60
graceful_timeout = 75

errorlog = "-"


# Setup tracing
# https://opentelemetry-python.readthedocs.io/en/latest/examples/fork-process-model/README.html?highlight=gunicorn#working-with-fork-process-models
# https://docs.sentry.io/platforms/python/tracing/instrumentation/opentelemetry/
def post_fork(server, worker):
    resource = Resource.create({SERVICE_NAME: "myapp", "worker": worker.pid})
    tracer_provider = TracerProvider(resource=resource)

    if otlp_endpoint := os.getenv("TRACING_OTLP_ENDPOINT"):
        otlp_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)
        span_processor = BatchSpanProcessor(otlp_exporter)
        tracer_provider.add_span_processor(span_processor)

    print(f"Sentry enabled: {os.getenv('SENTRY_DSN') is not None}")  # noqa: T201
    print(f"OTel export enabled: {otlp_endpoint is not None}")  # noqa: T201
    trace.set_tracer_provider(tracer_provider)


def post_worker_init(worker):
    import sentry_sdk
    from sentry_sdk.integrations.opentelemetry import (
        SentryPropagator,
        SentrySpanProcessor,
    )

    if sentry_dsn := os.getenv("SENTRY_DSN"):
        sentry_env = os.getenv("SENTRY_ENVIRONMENT")
        release = version("myapp")
        sentry_sdk.init(
            dsn=sentry_dsn,
            environment=sentry_env,
            traces_sample_rate=1.0,
            instrumenter="otel",
            release=f"{release}@{os.environ.get('GIT_REF', '(none)')}",
        )
        tracer_provider = trace.get_tracer_provider()
        tracer_provider.add_span_processor(SentrySpanProcessor())
        set_global_textmap(SentryPropagator())

The final issue here is that I keep adding span processors without removing them...

Where in the product are you?

Issues

Link

No response

DSN

https://[email protected]/5465401

Version

No response

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[awoimbee]

But I can't import sentry_sdk and sentry_sdk.init() in post_fork() because sentry uses sockets and gunicorn patches sockets

Here is the stack trace

/usr/local/lib/python3.12/site-packages/gunicorn/workers/ggevent.py:38: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['urllib3.util (/usr/local/lib/python3.12/site-packages/urllib3/util/__init__.py)', 'urllib3.util.ssl_ (/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py)'].
  monkey.patch_all()
[2024-11-13 18:46:39 +0000] [7] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/gunicorn/arbiter.py", line 608, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.12/site-packages/gunicorn/workers/ggevent.py", line 146, in init_process
    super().init_process()
  File "/usr/local/lib/python3.12/site-packages/gunicorn/workers/base.py", line 135, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.12/site-packages/gunicorn/workers/base.py", line 147, in load_wsgi
    self.wsgi = self.app.wsgi()
                ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gunicorn/app/base.py", line 66, in wsgi
    self.callable = self.load()
                    ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 57, in load
    return self.load_wsgiapp()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 47, in load_wsgiapp
    return util.import_app(self.app_uri)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gunicorn/util.py", line 370, in import_app
    mod = importlib.import_module(module)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/usr/local/lib/python3.12/site-packages/myapp/entrypoint.py", line 4, in <module>
    import requests
  File "/usr/local/lib/python3.12/site-packages/requests/__init__.py", line 164, in <module>
    from .api import delete, get, head, options, patch, post, put, request
  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 11, in <module>
    from . import sessions
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 15, in <module>
    from .adapters import HTTPAdapter
  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 80, in <module>
    _preloaded_ssl_context = create_urllib3_context()
                             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 290, in create_urllib3_context
    context.minimum_version = TLSVersion.TLSv1_2
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/ssl.py", line 546, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  File "/usr/local/lib/python3.12/ssl.py", line 546, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  File "/usr/local/lib/python3.12/ssl.py", line 546, in minimum_version
    super(SSLContext, SSLContext).minimum_version.__set__(self, value)
  [Previous line repeated 925 more times]
  File "/usr/local/lib/python3.12/ssl.py", line 544, in minimum_version
    if value == TLSVersion.SSLv3:
                ^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded
[2024-11-13 18:46:39 +0000] [7] [INFO] Worker exiting (pid: 7)
Exception ignored in thread started by: <bound method Thread._bootstrap of <Thread(OtelBatchSpanProcessor, started daemon 140064713598656)>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/threading.py", line 1030, in _bootstrap
    self._bootstrap_inner()
  File "/usr/local/lib/python3.12/threading.py", line 1077, in _bootstrap_inner
    self._delete()
  File "/usr/local/lib/python3.12/threading.py", line 1109, in _delete
    del _active[get_ident()]
        ~~~~~~~^^^^^^^^^^^^^
KeyError: 140065124234112

[getsantry]

Routing to @getsentry/product-owners-settings-integrations for triage ⏲️

[szokeasaurusrex]

Hi @awoimbee, could you please provide a complete minimal reproduction (including your project's dependencies, e.g. as a requirements.txt), and any other files needed to run the app? If you could share a GitHub repository, along with the exact commands I need to run to observe the bug, that would be ideal. You can also share your actual source code repo if it is public.

[awoimbee]

I made that: https://github.com/Extrality/flask-gevent-sentry-otel
Not sure its super useful

[antonpirker]

Thanks for the linked project @awoimbee! I will try to have a look today

[antonpirker]

Hey @awoimbee

This is not a Sentry issue, but rather a problem that grpc has with gevent.

I have this simple example project that resembles your setup: https://github.com/antonpirker/testing-sentry/tree/main/test-gunicorn-gevent-otel

It does not use the grpc span exporter, but the http one. Because grpc has its problems when using gevent.

This way it works and if the exporter endpoint is not available it raises an error, but the spans are still sent to Sentry.

So you probably should swith to an http exporter or if you need to use grpc than somehow make it work with gevent.

Hope this helps you move forward!

[awoimbee]

It does not use the grpc span exporter, but the http one. Because grpc has its problems when using gevent.

You're just not seeing the issue because in this case requests raises an exception so you don't go into the logic for "retriable" requests with the exponential backoff.
I think I'll just fix open-telemetry/opentelemetry-python#4331 and it will (mostly?) resolve my issue.