chore(grafana): relative CPU usage

[CAJan93]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

We get alerts that are relative to the CPU limit of the container. Let's add a panel that reflects this CPU usage

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

[tabVersion]

I guess update means there already has been one grafana running 🤔 no need to call risedev full to start a new cluster.

[CAJan93]

Will undo this change. I added it here, because I did not have a cluster running 😁

[Li0k]

What's the difference with the old CPU usage metrics?

[Nebulazhang]

It is depend on k8s metrics, that means we need a flag to ensure it only can be used in Cloud ENV

[CAJan93]

Let me look into that... So far I was fine with the way it is. In the risedev env it will simply show no data

[Nebulazhang]

What's the difference with the old CPU usage metrics?

This panel is the CPU usage utilization. For example, we allocated 4C for the CN, but CN used 3C, that means the utilization is 75%.

[CAJan93]

What's the difference with the old CPU usage metrics?

@Nebulazhang is right here. We also define the high CPU usage alerts (see here) as

(
  sum(rate(container_cpu_usage_seconds_total{namespace=~"rwc-.*", pod=~"risingwave-compute-default.*", container="compute"}[5m])) by (namespace, pod)
) / 
(
  sum(kube_pod_container_resource_limits{namespace=~"rwc-.*", pod=~"risingwave-compute-default.*", container="compute", resource="cpu"}) by(namespace, pod) 
) * 100 > 80

I think it would be helpful to see the same numbers in the dev dashboard when debugging.

[Nebulazhang]

LGTM

[CAJan93]

@tabVersion The panel will not work in non-k8s environments (e.g. risedev). In those cases it will show that no data is available. I added a description that points that out. Are you fine with that? If not we would need some mechanism to hide panels without data, but I feel like this is overkill 😆