Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DNM] CI run #26446

Closed
wants to merge 2 commits into from
Closed

[DNM] CI run #26446

wants to merge 2 commits into from

Conversation

danhhz
Copy link
Contributor

@danhhz danhhz commented Apr 4, 2024

Motivation

Tips for reviewer

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered.
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • This PR includes the following user-facing behavior changes:

danhhz added 2 commits April 4, 2024 16:03
@danhhz
persist: inline small writes directly in consensus State
90ca07b 
Previously, a compare_and_append operation always required writing the
data to Blob (s3) and then only putting a pointer to it in State (stored
in Consensus/crdb). The latencies of s3 puts are both considerably
higher and with a much fatter tail than crdb latencies. So, inline small
batch parts directly where we would have put the pointer.

- The threshold for small is made a LD param, so we can tune it.
- Skip keeping pushdown stats for these parts.
- Represent them using the new LazyProto abstraction developed for part
  stats. This keeps the inline parts as bytes::Bytes (which is in turn
  just a slice of the encoded ProtoStateRollup/ProtoStateDiff), only
  decoding them when they would have been "fetched". This considerably
  lessens the cost of deserializing these in places that aren't
  interested in the updates themselves (e.g. the controllers on envd).
@danhhz
fixup! persist: inline small writes directly in consensus State
c50e894
@danhhz danhhz closed this Apr 9, 2024
@danhhz danhhz deleted the tmp_persist_inline branch April 9, 2024 18:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@danhhz