grpc request may hang when error message is too large after bumping to tonic v0.12

[BugenZhao]

e2e-sink-test now consistently hangs here:

risingwave/e2e_test/sink/kafka/avro.slt

Lines 189 to 196 in 8d8c136

	statement error test-rw-sink-upsert-avro-err-key
	create sink sink_err from into_kafka with (
	connector = 'kafka',
	topic = 'test-rw-sink-upsert-avro-err',
	properties.bootstrap.server = 'message_queue:29092',
	primary_key = 'int32_field,string_field')
	format upsert encode avro (
	schema.registry = 'http://schemaregistry:8082');

I find that by disabling SCHEMA_REGISTRY_DEBUG here, this issue is gone.

risingwave/ci/docker-compose.yml

Line 269 in 8d8c136

SCHEMA_REGISTRY_DEBUG: 'true'

The only difference is that there won't be backtraces from the schema registry in the error message.

failed to validate sink: config error: all request confluent registry all timeout, req path ["subjects", "test-rw-sink-upsert-avro-err-key", "versions", "latest"], urls http://schemaregistry:8082/
	confluent schema registry error 40401: Subject 'test-rw-sink-upsert-avro-err-key' not found. io.confluent.rest.exceptions.RestNotFoundException: Subject 'test-rw-sink-upsert-avro-err-key' not found.
- io.confluent.rest.exceptions.RestNotFoundException: Subject 'test-rw-sink-upsert-avro-err-key' not found.
-	at io.confluent.kafka.schemaregistry.rest.exceptions.Errors.subjectNotFoundException(Errors.java:78)
-	at io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource.getSchemaByVersion(SubjectVersionsResource.java:154)
-	at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)

Following this insight, I suppose it's because we always encode the ServerError in gRPC (HTTP2) headers (#13282), and there's an outstanding issue where tonic 0.12 will hang forever when the header size exceeds some limit.

risingwave/src/error/src/tonic.rs

Lines 68 to 71 in 8d8c136

	let serialized = bincode::serialize(&source).unwrap();
	let mut metadata = MetadataMap::new();
	metadata.insert_bin(ERROR_KEY, MetadataValue::from_bytes(&serialized));

Upstream issues:

• long headers from the server hang tonic hyperium/tonic#1834
• hang when client receives header response that's too big in http2 hyperium/hyper#3724
• streams: notify recv task upon reset hyperium/h2#791

ATM there seems to be no fix. I'll disable SCHEMA_REGISTRY_DEBUG now as a workaround and open an issue for this.

Originally posted by @BugenZhao in #17889 (comment)

[BugenZhao]

With this configuration exposed, we're able to workaround this issue:

hyperium/tonic#1835

Waiting for a new version to be released.

[BugenZhao]

Workarounded with #18639

[github-actions]

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean.
Don't worry if you think the issue is still valuable to continue in the future.
It's searchable and can be reopened when it's time. 😄