Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix][Connector-V2] Add Filter for Partitions to Prevent Blocking in KafkaConsumer #31

Closed
wants to merge 4 commits into from

Conversation

xiaochen-zhou
Copy link
Collaborator

Purpose of this pull request

There may be an issue in the Kafka cluster where the disks containing a partition go offline simultaneously, resulting in the partition's leader being -1. When leader=-1, Kafka APIs like KafkaConsumer.position() will block. Therefore, we should filter out partitions with leader=-1.

image

Does this PR introduce any user-facing change?

no

How was this patch tested?

exist tests

Check list

@xiaochen-zhou xiaochen-zhou changed the title [Bug][Connector-V2] Add Filter for Partitions with to Prevent Blocking in KafkaConsumer. [Bug][Connector-V2] Add Filter for Partitions to Prevent Blocking in KafkaConsumer. Dec 17, 2024
@xiaochen-zhou xiaochen-zhou changed the title [Bug][Connector-V2] Add Filter for Partitions to Prevent Blocking in KafkaConsumer. [Bug][Connector-V2] Add Filter for Partitions to Prevent Blocking in KafkaConsumer Dec 17, 2024
@xiaochen-zhou xiaochen-zhou changed the title [Bug][Connector-V2] Add Filter for Partitions to Prevent Blocking in KafkaConsumer [Fix][Connector-V2] Add Filter for Partitions to Prevent Blocking in KafkaConsumer Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant