bug: frontend need restart to recover from breakdown

[zwang28]

Describe the bug

We encounter frontend breakdown, as below:

dev=> set visibility_mode to all;
SET_VARIABLE
dev=> select * from xxx limit 1;
ERROR:  QueryError: internal error: error trying to connect: deadline has elapsed
dev=> set visibility_mode to all;
SET_VARIABLE
dev=> select * from xxx limit 1;
ERROR:  QueryError: internal error: error trying to connect: dns error: failed to lookup address information: Name or service not known

After restarting frontend, the breakdown is gone.

To Reproduce

No response

Expected behavior

No response

Additional context

There has been compute node leaving/joining the cluster. Can it be related to frontend's compute node client pool?

[ZENOTME]

Is possible following case happen🤔:
A compute node leaving but the worker_node_manager didn't update yet. And then the query try to get a rpc client using the outdated worker node so that the connect failed.

[zwang28]

Is possible following case happen🤔:
A compute node leaving but the worker_node_manager didn't update yet. And then the query try to get a rpc client using the outdated worker node so that the connect failed.

The error remains for a long time until restart.

[ZENOTME]

Do we have more concrate log to reproduce this bug? Such as what reschedule happen before compute join or leave the cluster.

[hzxa21]

Do we have more concrate log to reproduce this bug? Such as what reschedule happen before compute join or leave the cluster.

cc @zwang28 Any more information can be provided?

Assign to @ZENOTME first. Feel free to reassign.

[zwang28]

Any more information can be provided?

Informed @ZENOTME when it occurred once again recently.
No other information from my side.