defensive: verify permit channel configuration on start-up to avoid potential stuck

[BugenZhao]

In #6170, we introduced permit-based back-pressure to resolve the imbalance throughput between local exchanges and remote exchanges. As an optimization, the message for giving back the permits will be batched before sending back to the upstream.

risingwave/src/stream/src/executor/exchange/input.rs

Lines 171 to 182 in e25ebd3

	if let Some(add_back_permits) = match permits.unwrap().value {
	// For records, batch the permits we received to reduce the backward
	// `AddPermits` messages.
	Some(permits::Value::Record(p)) => {
	batched_permits_accumulated += p;
	if batched_permits_accumulated >= batched_permits_limit as u32 {
	let permits = std::mem::take(&mut batched_permits_accumulated);
	Some(permits::Value::Record(permits))
	} else {
	None
	}
	}

On the other side, if a chunk is too large, it will clamp its required permit to total - batched. The batched is subtracted to prevent deadlock caused by permit batching.

risingwave/src/stream/src/executor/exchange/permit.rs

Line 47 in e25ebd3

let max_chunk_permits: usize = initial_permits - batched_permits;

Here comes the problem: if the upstream and the downstream does not reach consensus on these configuration, the behavior might be unexpected and there might be potential stuck. While it is highly unlikely, it is still possible if different compute nodes utilize varying configurations.

[github-actions]

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.