Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

does pulsar source will ack the msg? #15492

Open
milkdrink opened this issue Mar 6, 2024 · 4 comments
Open

does pulsar source will ack the msg? #15492

milkdrink opened this issue Mar 6, 2024 · 4 comments
Assignees
Labels
type/bug Something isn't working user-feedback

Comments

@milkdrink
Copy link

Describe the bug

it seems the pulsar source consume message but doesn't ack the message ,so the pulsar backlog size will increase day by day.

image
image

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

@milkdrink milkdrink added the type/bug Something isn't working label Mar 6, 2024
@github-actions github-actions bot added this to the release-1.8 milestone Mar 6, 2024
@fuyufjh fuyufjh modified the milestones: release-1.8, release-1.9 Apr 8, 2024
@tabVersion
Copy link
Contributor

Yes, RW does not commit back the offset back to pulsar broker. Can you share more about your use case and can message be managed by retention time, such as 7 days in kafka?

@tabVersion tabVersion removed this from the release-1.9 milestone May 14, 2024
@NiuBlibing
Copy link
Contributor

Yes, RW does not commit back the offset back to pulsar broker. Can you share more about your use case and can message be managed by retention time, such as 7 days in kafka?

Same issue, there are some trouble when rw does not send ack.

  1. Monitor system is broken: pulsar-manager and dashboard will show all messages is backlog.
  2. We have to set a larger retention policy to avoid message cleaned before consumed due to we don't know whether the messages is consumed correctly, It bring overload for disk space.

@tabVersion
Copy link
Contributor

#16733 this solution can also apply to pulsar 🤔

@NiuBlibing
Copy link
Contributor

#16733 this solution can also apply to pulsar 🤔

Using cumulative_ack ?
https://pulsar.apache.org/docs/3.2.x/concepts-messaging/#acknowledgment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Something isn't working user-feedback
Projects
None yet
Development

No branches or pull requests

5 participants