Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(frontend): rename streaming_rate_limit to backfill_rate_limit for mv backfilling and source_rate_limit for source updates #17796

Merged
merged 7 commits into from
Jul 30, 2024

Conversation

kwannoel
Copy link
Contributor

@kwannoel kwannoel commented Jul 24, 2024

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Closes #16466

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added test labels as necessary. See details.
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

After this PR we refactor streaming_rate_limit into:

  • source_rate_limit. This will be applied to TABLE WITH SOURCE, TABLE.
  • backfill_rate_limit. This will be applied to MV, SINK, INDEX for their backfill.

@graphite-app graphite-app bot requested a review from a team July 24, 2024 08:06
@xxchan xxchan removed the request for review from a team July 24, 2024 08:11
@kwannoel kwannoel requested a review from st1page July 24, 2024 09:02
@graphite-app graphite-app bot requested a review from xxchan July 26, 2024 02:51
Copy link
Member

@stdrc stdrc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO the difference between streaming_rate_limit and backfill_rate_limit can be very confusing after this...I propose another approach:

streaming_rate_limit -- still have the same meaning as before, controls both backfill and source/upstream
streaming_backfill_rate_limit -- only controls backfill
streaming_source_rate_limit -- only controls source/upstream

Is this doable?

src/sqlparser/src/keywords.rs Outdated Show resolved Hide resolved
src/sqlparser/src/parser.rs Outdated Show resolved Hide resolved
src/sqlparser/src/parser.rs Show resolved Hide resolved
@kwannoel
Copy link
Contributor Author

IMO the difference between streaming_rate_limit and backfill_rate_limit can be very confusing after this...I propose another approach:

streaming_rate_limit -- still have the same meaning as before, controls both backfill and source/upstream
streaming_backfill_rate_limit -- only controls backfill
streaming_source_rate_limit -- only controls source/upstream

Is this doable?

Seems more complex to have 3 variables to control it, both in terms of ux and our own maintenance.

This LGTM:

> streaming_backfill_rate_limit -- only controls backfill
> streaming_source_rate_limit -- only controls source/upstream

@kwannoel
Copy link
Contributor Author

Pause this PR for a while for some prereq work. See #17796 (comment)

@stdrc
Copy link
Member

stdrc commented Jul 29, 2024

IMO the difference between streaming_rate_limit and backfill_rate_limit can be very confusing after this...I propose another approach:

streaming_rate_limit -- still have the same meaning as before, controls both backfill and source/upstream
streaming_backfill_rate_limit -- only controls backfill
streaming_source_rate_limit -- only controls source/upstream

Is this doable?

Seems more complex to have 3 variables to control it, both in terms of ux and our own maintenance.

This LGTM:

> streaming_backfill_rate_limit -- only controls backfill
> streaming_source_rate_limit -- only controls source/upstream

I was suggesting to keep it for backward-compatibility purpose, as a fallback value of the following two. Either keeping it or removing it looks good to me actually.

@kwannoel kwannoel changed the title refactor(frontend): add backfill_rate_limit for mv backfilling refactor(frontend): rename streaming_rate_limit to backfill_rate_limit for mv backfilling Jul 29, 2024
@kwannoel kwannoel changed the title refactor(frontend): rename streaming_rate_limit to backfill_rate_limit for mv backfilling refactor(frontend): rename streaming_rate_limit to backfill_rate_limit for mv backfilling and source_rate_limit for source updates Jul 29, 2024
@kwannoel
Copy link
Contributor Author

Done further refactoring as per @stdrc's suggestion. Renamed streaming_rate_limit to source_rate_limit to be more precise, and remove any ambiguity.

This PR will introduce a breaking change, and I have included the details in the PR description.

Copy link
Member

@stdrc stdrc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lmatz
Copy link
Contributor

lmatz commented Jul 29, 2024

it seems searching backfill_rate_limit in the documentation produce no results, is it documented before?

@stdrc
Copy link
Member

stdrc commented Jul 29, 2024

it seems searching backfill_rate_limit in the documentation produce no results, is it documented before?

It's added in this PR.

@kwannoel kwannoel added user-facing-changes Contains changes that are visible to users breaking-change labels Jul 29, 2024
@kwannoel
Copy link
Contributor Author

kwannoel commented Jul 29, 2024

it seems searching backfill_rate_limit in the documentation produce no results, is it documented before?

We just make streaming_rate_limit more fine-grained. Instead of setting all kinds of rate limit via streaming_rate_limit, we use backfill_rate_limit, source_rate_limit to set the different types of rate limit.

@kwannoel kwannoel added this pull request to the merge queue Jul 30, 2024
Merged via the queue into main with commit cf30276 Jul 30, 2024
39 of 40 checks passed
@kwannoel kwannoel deleted the kwannoel/rename-rate-limit branch July 30, 2024 04:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change type/refactor user-facing-changes Contains changes that are visible to users
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Separate source and streaming rate limit
3 participants