feat(dashboard): visualize average backpressure rather than spot backpressure

[kwannoel]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

See the demo: https://www.notion.so/risingwave-labs/Backpressure-Graph-improvements-a11b0c5c74d54202922acb071aca72a0?pvs=4

Currently dashboard calculates the backpressure with the following mechanism.

1. Poll the actor_buffer_output_blocking_ns metric from meta / prometheus at a fixed duration (5s).
2. After polling, get the difference between previous and current blocking_duration.
3. Then backpressure rate is calculated as:

delta_blocking_duration / fixed_duration

4. This means if the blocking duration is a few minutes, it will show up on the graph for 5s, but in the subsequent epoch, it shows up 0, since actor_buffer_output_blocking_ns will ONLY be incremented after the chunk has been yielded downstream.
5. It is very easy to miss as a result, and does not reflect the stuck state of the actor. This becomes worse if the stream job contains a large number of fragments.

We can just compute average BP to solve this problem. cr @fuyufjh.

Further PRs:
see: #18176

Other changes

I changed the suggested steps to run the dashboard in the readme, so there's actually some live workload running. Previously it's just a series of DDLs with no data running through it.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

[fuyufjh]

The calculation is obscure to explain or understand. As a monitoring component, I'd like to keep it straight-forward.

I am thinking about a simpler way to address the issue. Instead of use the diff of delta_blocking_duration time, how about just use current value - first value, which means the accumulated one since the web page opens. In other words, the web page shows the average backpressure of 0-5s, 0-10s, 0-15s, ... as the user staying in the page. When severe backpressure is happening, the longer the user stay, the average number will be more accurate.

Your approach is also acceptable to me, but it's a bit hard to understand. Particularly, the duration spilled here is from a collected barrier i.e. it happened in the past, but it's spilled into the future.

Approved. Please pick the one you like the most. :)

[kwannoel]

The calculation is obscure to explain or understand. As a monitoring component, I'd like to keep it straight-forward.

I am thinking about a simpler way to address the issue. Instead of use the diff of delta_blocking_duration time, how about just use current value - first value, which means the accumulated one since the web page opens. In other words, the web page shows the average backpressure of 0-5s, 0-10s, 0-15s, ... as the user staying in the page. When severe backpressure is happening, the longer the user stay, the average number will be more accurate.

Your approach is also acceptable to me, but it's a bit hard to understand. Particularly, the duration spilled here is from a collected barrier i.e. it happened in the past, but it's spilled into the future.

Approved. Please pick the one you like the most. :)

Average BP makes sense to me. That's a very good solution, because of this:

When severe backpressure is happening, the longer the user stay, the average number will be more accurate.

Let me try it.

[kwannoel]

Stays like this for a prolonged period, which easily identifies the bottleneck of the cluster. This approach works well.