-
Notifications
You must be signed in to change notification settings - Fork 328
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(remote_wal): impl kafka log store #2971
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## develop #2971 +/- ##
===========================================
- Coverage 85.83% 85.72% -0.11%
===========================================
Files 780 783 +3
Lines 126196 126435 +239
===========================================
+ Hits 108319 108389 +70
- Misses 17877 18046 +169 |
Had an offline discussion with @WenyXu . We all agree that the split/merge of WAL entries should happen inside logstore. It's the logstore that know the exact log entry size limit and how to handle the split/merge. AFAIK rskafka employs some batch mechanism so that we can simply forwrd wal entries to Kafka records in a 1:1 manner and let rskafka to handle the batching stuff. Only one thing to notice: be careful to handle large single wal entries, split those large entries inside logstore and ensure the splited parts get produced sequentially and atomically. |
Splitting a way-too large log entry is in planning. However, it may be implemented in a subsequent PR since we need to investigate how rskafka client handles a way-too large log entry and then design an appropriate splitting strategy. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Almost LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
I hereby agree to the terms of the GreptimeDB CLA
What's changed and what's your intention?
ClientManager
to manage construction and accesses of rskafka client. Through the manager, the kafka log store is able to contact kafka cluster.KafkaLogStore
to provide read/write interfaces for interacting with Kafka remote wal.KafkaLogStore
upon starting the datanode.TODO
Checklist
Refer to a related PR or issue link (optional)