Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: A system table for showing the parallelism of streaming jobs #14256

Closed
neverchanje opened this issue Dec 28, 2023 · 0 comments · Fixed by #14261
Closed

feat: A system table for showing the parallelism of streaming jobs #14256

neverchanje opened this issue Dec 28, 2023 · 0 comments · Fixed by #14261
Assignees
Milestone

Comments

@neverchanje
Copy link
Contributor

neverchanje commented Dec 28, 2023

Is your feature request related to a problem? Please describe.

We currently need to join several low-level system tables rw_parallel_units, rw_materilaized_views and rw_sinks to get a final view for parallelisms.

Describe the solution you'd like

We should provide a view called rw_streaming_parallelism for this purpose. In addition to parallelism number, we also need a column indicating the scaling policy of the object.

Describe alternatives you've considered

WITH all_objects AS (
    SELECT id, name FROM rw_tables
    UNION ALL
    SELECT id, name FROM rw_materialized_views
    UNION ALL
    SELECT id, name FROM rw_sinks
    UNION ALL
    SELECT id, name FROM rw_indexes
),
fragment_parallelism AS (
    SELECT
    f.fragment_id,
    COUNT(a.actor_id) AS parallelism
    FROM rw_actors a
    INNER JOIN rw_fragments f ON a.fragment_id = f.fragment_id
    GROUP BY f.fragment_id
)
SELECT
    f.*,
    fp.parallelism
FROM all_objects ao
INNER JOIN rw_fragments f ON ao.id = f.table_id
INNER JOIN fragment_parallelism fp ON f.fragment_id = fp.fragment_id
ORDER BY ao.name;

Here is the query I used to write for our previous scaling manual. https://docs.risingwave.com/docs/current/k8s-cluster-scaling/

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants