Fix pyroscope.write issues with pyroscope.receive_http #2201
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The nodejs Pyroscope SDK sends profiles with a `Connection: close` header. This header was copied to the upstream request, causing connection churn towards Pyroscope, which can be quite bad on the CPU when using TLS. Do not copy the `Connection` header from the incoming request to fix this issue. Additionally, `pyroscope.write` had a single `http.Client` used for forwarding data from `pyroscope.receive_http`, which may not work if multiple endpoints are configured with different options. To fix this, store a `http.Client` for each endpoint.
madaraszg-tulip
force-pushed
the
fix-ingest-keepalive
branch
from
f6cca99 to
f803ffc
Compare
marcsanmi
reviewed
Dec 2, 2024
There was a problem hiding this comment.
Thanks for taking care of this! 👏
From HTTP/2 specification ([RFC 9113]:(https://datatracker.ietf.org/doc/html/rfc9113#name-headers))
HTTP/2 does not use the Connection header field (Section 7.6.1 of [HTTP]) to indicate connection-specific header fields; in this protocol, connection-specific metadata is conveyed by other means. An endpoint MUST NOT generate an HTTP/2 message containing connection-specific header fields. This includes the Connection header field and those listed as having connection-specific semantics in Section 7.6.1 of [HTTP] (that is, Proxy-Connection, Keep-Alive, Transfer-Encoding, and Upgrade).
👆 Could we also exclude the above mentioned hop-by-hop headers?
Added a map to handle all ignored headers. |
marcsanmi
approved these changes
Dec 3, 2024
ptodev
pushed a commit
that referenced
this pull request
Dec 3, 2024
* Fix pyroscope.write issues with pyroscope.receive_http The nodejs Pyroscope SDK sends profiles with a `Connection: close` header. This header was copied to the upstream request, causing connection churn towards Pyroscope, which can be quite bad on the CPU when using TLS. Do not copy the `Connection` header from the incoming request to fix this issue. Additionally, `pyroscope.write` had a single `http.Client` used for forwarding data from `pyroscope.receive_http`, which may not work if multiple endpoints are configured with different options. To fix this, store a `http.Client` for each endpoint.
ptodev
added a commit
that referenced
this pull request
Dec 3, 2024
* Fixed an issue in the `otlp.exporter.prometheus` component (#2102) * Fixed an issue in the `otlp.exporter.prometheus` component * Fixed an issue in the `otlp.exporter.prometheus` component * Fix potential deadlock in import statements (#2129) * Fix potential deadlock in import statements * change * typo * fix: race condition UI remotecfg (#2160) * Refactor ui remtoecfg components to avoid race condition * Fix accidental cast to pointer that should have been struct * Update changelog * fix: fully prevent panic in remotecfg ui (#2164) * Fully prevent panic in remotecfg ui * Address PR feedback * Fix deadlock due to infinite retry (#2174) * Fix deadlock due to infinite retry * changelog * Update ckit to fix memberlist logging issues (#2186) * Upgrade ckit and changelog * go mod tidy * `loki.source.podlogs`: Fix issue which disables clustering unintentionally. (#2187) * Fix issue which disables clustering unintentionally. * prometheus.operator.*: allow setting informer_sync_timeout (#2161) * prometheus.operator.*: allow setting informer_sync_timeout * default to 1m * docs * fix(pyroscope): allow slashes in tag name (#2172) * loki.source.podlogs: For clustering only take into account some labels (#2190) * Only take into account some labels * Reword docs * fix: crash when updating import.http config (#2204) * fix: crash when updating import.http config * fix key/pattern logic for the attribute processor (#2124) * fix: Update postgres exporter (#2019) * Update postgres exporter * Update changelog * Use postgres exporter branch that implements exporter package * Add TODO for future maintainers * Update VERSION file * Add missing changelog entry * Fix pyroscope.write issues with pyroscope.receive_http (#2201) * Fix pyroscope.write issues with pyroscope.receive_http The nodejs Pyroscope SDK sends profiles with a `Connection: close` header. This header was copied to the upstream request, causing connection churn towards Pyroscope, which can be quite bad on the CPU when using TLS. Do not copy the `Connection` header from the incoming request to fix this issue. Additionally, `pyroscope.write` had a single `http.Client` used for forwarding data from `pyroscope.receive_http`, which may not work if multiple endpoints are configured with different options. To fix this, store a `http.Client` for each endpoint. --------- Co-authored-by: YusifAghalar <[email protected]> Co-authored-by: Piotr <[email protected]> Co-authored-by: Sam DeHaan <[email protected]> Co-authored-by: Craig Peterson <[email protected]> Co-authored-by: Marc Sanmiquel <[email protected]> Co-authored-by: Sergei Nikolaev <[email protected]> Co-authored-by: William Dumont <[email protected]> Co-authored-by: Sam DeHaan <[email protected]> Co-authored-by: Gergely Madarász <[email protected]>
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Description
The nodejs Pyroscope SDK sends profiles with a
Connection: closeheader. This header was copied to the upstream request, causing connection churn towards Pyroscope, which can be quite bad on the CPU when using TLS. Do not copy theConnectionheader from the incoming request to fix this issue.Additionally,
pyroscope.writehad a singlehttp.Clientused for forwarding data frompyroscope.receive_http, which may not work if multiple endpoints are configured with different options. To fix this, store ahttp.Clientfor each endpoint.PR Checklist