massive amount of OpenTelemetry trace error occurred. cannot send message to batch processor as the channel is full in the log

[lmatz]

https://grafana.test.risingwave-cloud.xyz/explore?panes=%7B%22D6x%22:%7B%22datasource%22:%22PE59595AED52CF917%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22sysbench-pg-cdc-daily-20240110%5C%22,%20pod%3D%5C%22benchmark-risingwave-compute-c-0%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22PE59595AED52CF917%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221704869389627%22,%22to%22:%221704912589627%22%7D%7D%7D&schemaVersion=1&orgId=1

several pages actually

[lmatz]

https://buildkite.com/risingwave-test/sysbench-cdc/builds/147#018cf489-68b1-4c27-8ffe-22a6df2da07f

[BugenZhao]

This could happen when there's a huge amount of data to be exported to the opentelemetry service (either embedded in meta or external), especially for serving queries with high concurrency. There might be several ways to reduce the number of traces, like enable sampling.

Since #14528, however, the tracing feature was disabled by default and designed to be only enabled when we're going to diagnose the performance issues in the system. So the issue seems not a big deal anymore.