Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: avoid useless io in large-payload queries #500

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

zghong
Copy link

@zghong zghong commented Dec 25, 2024

Description

When executing large-payload insert queries by POST, the current retry and throughput-limit mechanisms read the POST body, which can be resource-consuming and may even cause CPU/Memory bottlenecks. This MR avoids unnecessary I/O operations in these two mechanisms:

  • For the retry mechanism: If the retry parameter is not set, the step of copying the body can be skipped.
  • For the throughput-limit mechanism: Instead of calculating the length of query, the Content-Length is read directly from POST header.

Closes #499, #428

Pull request type

Please check the type of change your PR introduces:

  • Bugfix
  • Feature
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • Documentation content changes
  • Other (please describe):

Checklist

  • Linter passes correctly
  • Add tests which fail without the change (if possible)
  • All tests passing
  • Extended the README / documentation, if necessary

Does this introduce a breaking change?

  • Yes
  • No

Further comments

For the throughput-limit mechanism: The Content-Length header indicates the size of the request body, so the throughput of GETs will be ignored. This is logical, as GETs typically have minimal impact on throughput compared to POSTs.

@zghong
Copy link
Author

zghong commented Dec 25, 2024

Before:

image
image
image
image

After:

image
image
image
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] High CPU/Memory usage for large-payload insert queries
1 participant