Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data transfers tracking + migrations #209

Merged
merged 11 commits into from
Nov 17, 2023
Merged

Data transfers tracking + migrations #209

merged 11 commits into from
Nov 17, 2023

Conversation

mensfeld
Copy link
Member

@mensfeld mensfeld commented Nov 9, 2023

ref #199

This PR also introduces:

  • the aggregated states migrations, so I can easily ensure states consistency and so I do not have to worry about cases when some of the data would be missing partially and would only start to appear later. I can just migrate missing stuff.
  • It introduces aggregates states version collision detection, allowing us to stop processing and signal if a consumer running older version of karafka would try to work with newer data. We had this for the processes reports, but now it expands to the rest of the data.

obraz

@mensfeld mensfeld self-assigned this Nov 9, 2023
@mensfeld mensfeld changed the base branch from master to 0.8.0 November 9, 2023 14:11
@mensfeld mensfeld changed the title Data transfers Data transfers tracking + migrations Nov 10, 2023
@mensfeld mensfeld added enhancement New feature or request tracking Metrics tracking related stuff migrations PRs related to states migrations labels Nov 10, 2023
@mensfeld mensfeld requested a review from nijikon November 10, 2023 12:26
@mensfeld mensfeld marked this pull request as ready for review November 17, 2023 10:27
@mensfeld mensfeld merged commit 7e84ad1 into 0.8.0 Nov 17, 2023
14 checks passed
@mensfeld mensfeld deleted the data-transfers branch November 17, 2023 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request migrations PRs related to states migrations tracking Metrics tracking related stuff
Development

Successfully merging this pull request may close these issues.

1 participant