Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Store SourceStates in log and cache #2158

Merged
merged 1 commit into from
Oct 16, 2023
Merged

feat: Store SourceStates in log and cache #2158

merged 1 commit into from
Oct 16, 2023

Conversation

chubei
Copy link
Contributor

@chubei chubei commented Oct 16, 2023

This PR is a preparation for solving the log data loss problem.

  • We store SourceStates in cache, so when cache reconnects to a log, it knows whether it needs to roll back.
  • We keep track of the SourceStates that was persisted when a Log is restored from checkpoint, so a reconnecting client knows where to roll back to if necessary.

@chubei chubei requested a review from Jesse-Bakker October 16, 2023 09:55
Copy link
Contributor

@Jesse-Bakker Jesse-Bakker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my understanding: what's the reason for storing log offset and source states in separate lmdb databases?

@chubei
Copy link
Contributor Author

chubei commented Oct 16, 2023

I planned to store the source states as key value pairs in a database, so log position has to go to another database, otherwise there needs be some magic key for it. Later I found it too troublesome and went with simple bincode serialization but forgot to merge them into the same database.

@chubei chubei added this pull request to the merge queue Oct 16, 2023
Merged via the queue into getdozer:main with commit dfe7931 Oct 16, 2023
6 checks passed
@chubei chubei deleted the feat/source_states branch October 16, 2023 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants