log: helpful logs may be suppressed

[zwang28]

Describe the bug

Helpful logs that shows large rows matched for join key may be suppressed. If so, developer/user cannot identify the key data that's causing the problems.

e.g. I found 1000k rows matched for join key in grafana, but 10k at max from logs at the same time.

@kwannoel @fuyufjh

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

[zwang28]

e.g. I found 1000k rows matched for join key in grafana, but 10k at max from logs at the same time.

The actual value behind grafana should be at least 500k, because the exponential factor of the histogram is 2. @st1page

let join_matched_join_keys_opts = histogram_opts!(
    "stream_join_matched_join_keys",
    "The number of keys matched in the opposite side",
    exponential_buckets(16.0, 2.0, 28).unwrap() // max 2^31
);

[kwannoel]

I suppose we can do a bucket based log suppression instead, so we can capture better distribution of high join amp keys.

The workaround for now is to tweak:

streaming.developer.stream_high_join_amplification_threshold = 2048

To a higher number. It will then ignore the lower join amplification logs, avoiding the suppression.

After triggering recovery the high join amp key of 1M should show up.

[fuyufjh]

After some thoughts, I tend to simply remove the Log Suppression introduced in #16840 along with the warning log itself. Because a join key with more than 2048 matched rows won't be very frequent in normal situations. Even saying if it's frequent, the problem itself would be much serious than the logs.