Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(dashboard): back-pressure rate on stream graph in Grafana #13422

Closed
wants to merge 3 commits into from

Conversation

BugenZhao
Copy link
Member

@BugenZhao BugenZhao commented Nov 14, 2023

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

So that we can quickly find the bottleneck of a stream job. Inter-table flow chart is coming soon.

Preview:

image

Compared with #12397:

Pros:

  • Embedded in Grafana. Intuitive and easy to access.
  • Benefit from the Grafana(Prometheus)'s capability of handling timeseries. For example, inspect the back-pressure and throughput in a time-travelling way and show different indicators (mean vs p90 vs last) with familiar operations.

Cons:

  • The flowchart is provided by a plugin. Provisioning requires extra work.
  • The dev dashboard has much more details that allows us to inspect the original protobuf definition of job, internal table, or plan node body. So it may not be easily replaced.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

@BugenZhao BugenZhao changed the title feat(dashboard): back-pressure rate on stream graph in Grafana dashboard feat(dashboard): back-pressure rate on stream graph in Grafana Nov 14, 2023
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

license-eye has totally checked 4441 files.

Valid Invalid Ignored Fixed
1951 2 2488 0
Click to see the invalid file list
  • src/utils/flowchart/src/lib.rs
  • src/utils/flowchart/src/main.rs

@@ -0,0 +1,201 @@
use std::collections::{BTreeMap, BTreeSet, HashMap};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
use std::collections::{BTreeMap, BTreeSet, HashMap};
// Copyright 2023 RisingWave Labs
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::collections::{BTreeMap, BTreeSet, HashMap};

@@ -0,0 +1,3 @@
fn main() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fn main() {
// Copyright 2023 RisingWave Labs
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
fn main() {

"show": False,
"stats": ["mean", "last", "min", "max", "sum"],
},
"contentUrl": "http://localhost:5691/api/flowchart/table/${table}",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! In this way the Grafana can display a dynamic graph, generated by the Rust code

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but I find limitations. 😕 The requests seem to be sent from the browser, so the connectivity might be a problem.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JSON data source might help to fetch the flowchart content, which supports sending requests from the server:

https://github.com/simPod/GrafanaJsonDatasource

Copy link
Contributor

This PR has been open for 60 days with no activity. Could you please update the status? Feel free to ping a reviewer if you are waiting for review.

@BugenZhao
Copy link
Member Author

Superseded by #14446.

@BugenZhao BugenZhao closed this Jan 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants