Aggressive retry on meta election failure

[arkbriar]

Describe the bug

The meta aggressively retries on another election attempt when the previous one failed. Retry interval is ~50ms.

Error message/log

2023-09-12T07:28:47.536850637Z  INFO risingwave_meta::rpc::election_client: client risingwave-meta-0.risingwave-meta-headless.default.svc:5690 start election
2023-09-12T07:28:47.541331846Z ERROR risingwave_meta::rpc::server: election error happened, Election failed: grpc request error: status: Unavailable, message: "error trying to connect: dns error: failed to lookup address information: Name or service not known", details: [], metadata: MetadataMap { headers: {} }
2023-09-12T07:28:47.541353137Z  INFO risingwave_meta::rpc::election_client: client risingwave-meta-0.risingwave-meta-headless.default.svc:5690 start election
2023-09-12T07:28:47.547203804Z ERROR risingwave_meta::rpc::server: election error happened, Election failed: grpc request error: status: Unavailable, message: "error trying to connect: dns error: failed to lookup address information: Name or service not known", details: [], metadata: MetadataMap { headers: {} }
2023-09-12T07:28:47.547214846Z  INFO risingwave_meta::rpc::election_client: client risingwave-meta-0.risingwave-meta-headless.default.svc:5690 start election
2023-09-12T07:28:47.552142221Z ERROR risingwave_meta::rpc::server: election error happened, Election failed: grpc request error: status: Unavailable, message: "error trying to connect: dns error: failed to lookup address information: Name or service not known", details: [], metadata: MetadataMap { headers: {} }

To Reproduce

Install a RisingWave with helm.

helm repo add risingwavelabs https://risingwavelabs.github.io/helm-charts/
helm repo update
helm install --set wait=true risingwave risingwavelabs/risingwave

You can find the logs by

k logs risingwave-meta-0 -f

Expected behavior

There should be backoff to ensure the CPU won't be exhausted.

How did you deploy RisingWave?

Helm + Kubernetes

The version of RisingWave

dev=> select version();
                                  version
----------------------------------------------------------------------------
 PostgreSQL 9.5-RisingWave-1.2.0 (f27f085e4433eb4b3b9e8df61b3f94406f654d89)
(1 row)

The image is v1.2.0.

Additional context

No response

[fuyufjh]

Ping, any updates?

[neverchanje]

I also encountered this problem. Any solutions?

[fuyufjh]

Does this problem recur now?

[arkbriar]

I think it still exists.

risingwave/src/meta/node/src/server.rs

Lines 258 to 265 in ca8d8ed

	let handle = tokio::spawn(async move {
	while let Err(e) = election_client
	.run_once(lease_interval_secs as i64, stop_rx.clone())
	.await
	{
	tracing::error!(error = %e.as_report(), "election error happened");
	}
	});