Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Support sink coordinator #61

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Conversation

wenym1
Copy link
Contributor

@wenym1 wenym1 commented May 16, 2023

@wenym1 wenym1 marked this pull request as ready for review May 17, 2023 00:37
@wenym1
Copy link
Contributor Author

wenym1 commented May 24, 2023

Summary for the design meeting:

  • In sink coordinator design, there will be multiple concurrent sink writer in each sink parallelism (single sink executor) and a single sink coordinator in the meta store.
  • When checkpoint barrier comes, we call finish_write on all sink writers and get the metadata written by each sink writer. After collecting metadata of all sink writers, the coordinator will commit all metadata to the external sinks.
  • We will implement a simple stateless coordinator for improving the throughput of iceberg sink at first.
  • The implementation logic of two phase commit sink is correct, but complex compared to the simple stateless coordinator. Will implement it when there is more feature demand on exactly-once sink.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants