Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: kinesis re-shard may cause data loss when scan.startup.mode is latest #18041

Open
tabVersion opened this issue Aug 14, 2024 · 1 comment
Open

bug: kinesis re-shard may cause data loss when scan.startup.mode is latest #18041

tabVersion opened this issue Aug 14, 2024 · 1 comment
Assignees
@stdrc @tabVersion
Labels
type/bug Something isn't working
Milestone
release-2.2

Comments

@tabVersion
Copy link
Contributor 

          @stdrc talked about enumerator should inherent offset settings from source props. But it cannot handle the case if the user set `scan.startup.mode` to `latest`, leading to data loss from the shard starts to the time we actually start reading data.

Originally posted by @tabVersion in #18038 (comment)
@github-actions github-actions bot added this to the release-2.0 milestone Aug 14, 2024
@tabVersion tabVersion self-assigned this Aug 19, 2024
@stdrc stdrc changed the title tracking: more accurate source offset desc kinesis re-shard may cause data loss when scan.startup.mode is latest Oct 10, 2024
@stdrc stdrc changed the title kinesis re-shard may cause data loss when scan.startup.mode is latest bug: kinesis re-shard may cause data loss when scan.startup.mode is latest Oct 10, 2024
@stdrc stdrc added the type/bug Something isn't working label Oct 10, 2024
@tabVersion
Copy link
Contributor Author

Hi @stdrc, can you share more on re-shard and data loss?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stdrc stdrc

@tabVersion tabVersion

Labels
type/bug Something isn't working
Projects
None yet
Milestone
release-2.2
Development

No branches or pull requests
2 participants
@stdrc @tabVersion