bug: frontend can't connect to CN sometimes when CNs have been restarted

[chenzl25]

Describe the bug

Frontend can't connect to CN sometimes when CNs have been restarted until we restart the frontend node. Need to find a way to reproduce this issue.

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

[lmatz]

#14030 chaos mesh can reproduce it sometimes

[rotten]

Restarting the frontend after it gets into this state does not help it recover. Restarting the meta node after this happens does seem to resolve it.

[rotten]

... but only if I shutdown the old meta node completely first. Simply bringing up a second meta and tearing down the old one doesn't seem to work.

I could update the "graceful shutdown" script on the compute node to also trigger a meta restart, but that wouldn't cover the scenario when a compute node simply crashes.

[github-actions]

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean.
Don't worry if you think the issue is still valuable to continue in the future.
It's searchable and can be reopened when it's time. 😄