feat: pause shared SourceExecutor until a downstream actor is created #16348

xxchan · 2024-04-16T18:18:47Z

Signed-off-by: xxchan [email protected]I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

As title. Without this optimization, shared source will waste a lot work at the beginning.
This applies to shared Kafka source, and also shared CDC source.

The implementation of testing whether there are downstreams is similar to MView's optimization for conflict check
- When a SourceExecutor is created, check actor_context.dispatch_num to pause.
- Wait for barrier to resume the stream.
A behavior change: now rw_enable_shared_source only applies to CREATE SOURCE, but not CREATE MV
Also added some additional logic tests for shared source.

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added test labels as necessary. See details.
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

xxchan · 2024-04-20T09:53:03Z

src/frontend/src/optimizer/plan_node/logical_source.rs

-                let use_shared_source = self.source_catalog().is_some_and(|c| c.info.is_shared())
-                    && self.ctx().session_ctx().config().rw_enable_shared_source();
+                // We only check rw_enable_shared_source is true when `CREATE SOURCE`.
+                // The value does not affect the behavior of `CREATE MATERIALIZED VIEW` here.
+                let use_shared_source = self.source_catalog().is_some_and(|c| c.info.is_shared());


I changed this behavior because I found otherwise it's not easy to tell whether the SourceExecutor is shared. I guess this flexibility isn't very useful, so it's fine to be simpler.

e2e_test/source_inline/cdc/mysql/mysql_create_drop.slt

e2e_test/source_inline/kafka/shared_source.slt

e2e_test/source/basic/kafka_shared_source.slt

e2e_test/source_inline/kafka/shared_source.slt

tabVersion

LGTM

chenzl25

LGTM

src/stream/src/executor/mview/materialize.rs

commit ea28be3cc7c6186f93a86a3fb8bb3c8123074f8c Author: xxchan <[email protected]> Date: Thu Apr 25 21:08:32 2024 +0800 fix Signed-off-by: xxchan <[email protected]> commit 0bcc42a7ca96bdb6de9f3ee2308ee86cc1dfcd61 Author: xxchan <[email protected]> Date: Thu Apr 25 20:17:40 2024 +0800 fix Signed-off-by: xxchan <[email protected]> commit 5de9bdce1ec5f1773107f550cd4b4667877721f2 Author: xxchan <[email protected]> Date: Thu Apr 25 17:04:56 2024 +0800 resolve comment Signed-off-by: xxchan <[email protected]> commit b0f5dff1db795ff852090a572e73910874b89d1d Author: xxchan <[email protected]> Date: Wed Apr 24 08:06:41 2024 +0800 update commit f38cb387d6254ff40aac0d02d9fcbaa212ce7b32 Author: xxchan <[email protected]> Date: Wed Apr 24 08:04:04 2024 +0800 update cdc test Signed-off-by: xxchan <[email protected]> commit d5afeb8756a4c029e441b7983d069dcc8767c29e Author: xxchan <[email protected]> Date: Mon Apr 22 15:33:11 2024 +0800 fix: only resume for shared Signed-off-by: xxchan <[email protected]> commit 21163412cb1201dc2190c3b0e285e88f8e443f17 Author: xxchan <[email protected]> Date: Mon Apr 22 15:17:33 2024 +0800 clippy Signed-off-by: xxchan <[email protected]> commit 0d0a304623fff4732126b61f9a977bcb18e56dd9 Author: xxchan <[email protected]> Date: Mon Apr 22 15:00:02 2024 +0800 update tests Signed-off-by: xxchan <[email protected]> commit a0e9670e6002602fd7c508e6f79b6a1f57d291b9 Author: xxchan <[email protected]> Date: Sat Apr 20 17:48:51 2024 +0800 fix Signed-off-by: xxchan <[email protected]> commit 572eab4b35a1d4c0a2352deabd9fa307843cd2c5 Author: xxchan <[email protected]> Date: Sat Apr 20 17:10:58 2024 +0800 clean Signed-off-by: xxchan <[email protected]> commit eee0f6c78822b3d937725bd2a76af23b3b15909a Author: xxchan <[email protected]> Date: Sat Apr 20 17:08:32 2024 +0800 cleanup state table changes Signed-off-by: xxchan <[email protected]> commit ad90536448a2bd16b95b0dab629a428592a19f7d Author: xxchan <[email protected]> Date: Sat Apr 20 17:01:21 2024 +0800 refactor Signed-off-by: xxchan <[email protected]> commit 017e90ffc73993a5f88e3a02b4e78156a6b92b9a Author: xxchan <[email protected]> Date: Fri Apr 19 16:33:05 2024 +0800 add doc Signed-off-by: xxchan <[email protected]> commit b893b915997481fe61c9c6963b75b4769e5f238c Author: xxchan <[email protected]> Date: Wed Apr 17 02:18:00 2024 +0800 feat: pause shared source until a MV is created Signed-off-by: xxchan <[email protected]> Signed-off-by: xxchan <[email protected]>

Signed-off-by: xxchan <[email protected]>

github-actions bot added the type/style label Apr 16, 2024

xxchan changed the title ~~style: improve source test and risedev-env style~~ feat: pause shared source until a MV is created Apr 16, 2024

github-actions bot added type/feature and removed type/style labels Apr 16, 2024

xxchan mentioned this pull request Apr 16, 2024

Tracking issue for shared source #16003

Closed

21 tasks

xxchan force-pushed the xxchan/share-source-04-17 branch from e2401f5 to 302266f Compare April 17, 2024 01:43

Base automatically changed from xxchan/source-test to main April 18, 2024 06:06

xxchan force-pushed the xxchan/share-source-04-17 branch 3 times, most recently from 8220645 to 7774406 Compare April 19, 2024 07:47

xxchan changed the title ~~feat: pause shared source until a MV is created~~ feat: pause shared SourceExecutor until a downstream MV is created Apr 20, 2024

xxchan force-pushed the xxchan/share-source-04-17 branch 4 times, most recently from 48b1ee4 to afe5279 Compare April 20, 2024 09:51

xxchan commented Apr 20, 2024

View reviewed changes

xxchan force-pushed the xxchan/share-source-04-17 branch 2 times, most recently from 57733ab to af290a2 Compare April 22, 2024 07:11

xxchan changed the title ~~feat: pause shared SourceExecutor until a downstream MV is created~~ feat: pause shared SourceExecutor until a downstream actor is created Apr 22, 2024

xxchan mentioned this pull request Apr 23, 2024

test: integrate mysql (user-managed) into risedev and inline source tests #16449

Merged

9 tasks

xxchan force-pushed the xxchan/share-source-04-17 branch from 7ce930f to 01ba723 Compare April 24, 2024 00:04

xxchan changed the base branch from main to xxchan/mysql April 24, 2024 00:04

xxchan commented Apr 24, 2024

View reviewed changes

e2e_test/source_inline/cdc/mysql/mysql_create_drop.slt Outdated Show resolved Hide resolved

xxchan commented Apr 24, 2024

View reviewed changes

e2e_test/source_inline/kafka/shared_source.slt Show resolved Hide resolved

xxchan commented Apr 24, 2024

View reviewed changes

e2e_test/source/basic/kafka_shared_source.slt Show resolved Hide resolved

xxchan requested review from stdrc, chenzl25, fuyufjh, tabVersion and BugenZhao April 24, 2024 00:09

tabVersion requested review from shanicky and StrikeW April 24, 2024 07:26

tabVersion reviewed Apr 24, 2024

View reviewed changes

e2e_test/source_inline/kafka/shared_source.slt Show resolved Hide resolved

e2e_test/source_inline/kafka/shared_source.slt Show resolved Hide resolved

e2e_test/source_inline/kafka/shared_source.slt Show resolved Hide resolved

tabVersion approved these changes Apr 25, 2024

View reviewed changes

chenzl25 approved these changes Apr 25, 2024

View reviewed changes

src/stream/src/executor/mview/materialize.rs Outdated Show resolved Hide resolved

xxchan force-pushed the xxchan/share-source-04-17 branch from ffe69f8 to e18e050 Compare April 25, 2024 11:55

xxchan mentioned this pull request Apr 25, 2024

refactor(state-table): make consistent op an enum in state table #16471

Merged

9 tasks

xxchan force-pushed the xxchan/share-source-04-17 branch from e18e050 to 0d49ccb Compare April 25, 2024 13:25

Base automatically changed from xxchan/mysql to main April 25, 2024 13:59

xxchan force-pushed the xxchan/share-source-04-17 branch from 0d49ccb to cd275e0 Compare April 25, 2024 14:12

xxchan enabled auto-merge April 25, 2024 14:13

sleep longer

3a7e4ad

Signed-off-by: xxchan <[email protected]>

xxchan force-pushed the xxchan/share-source-04-17 branch from cd275e0 to 3a7e4ad Compare April 25, 2024 14:51

xxchan added 5 commits April 25, 2024 23:08

sleepy

b4e42e3

Signed-off-by: xxchan <[email protected]>

fix

164e2a7

Signed-off-by: xxchan <[email protected]>

fix

9f099b3

Signed-off-by: xxchan <[email protected]>

test

358c84e

Signed-off-by: xxchan <[email protected]>

test

691ba99

Signed-off-by: xxchan <[email protected]>

xxchan disabled auto-merge April 25, 2024 16:17

fix

4696cda

Signed-off-by: xxchan <[email protected]>

xxchan force-pushed the xxchan/share-source-04-17 branch from e69fc65 to 4696cda Compare April 25, 2024 17:01

xxchan enabled auto-merge April 25, 2024 17:05

xxchan added this pull request to the merge queue Apr 25, 2024

Merged via the queue into main with commit af9ac6d Apr 25, 2024
27 of 28 checks passed

xxchan deleted the xxchan/share-source-04-17 branch April 25, 2024 17:35

BugenZhao mentioned this pull request Apr 28, 2024

feat(meta): support inject barrier to enable and disable writing old value for subscription #16460

Merged

9 tasks

xxchan mentioned this pull request May 3, 2024

shared source: don't backfill for the first MV #16576

Closed

xxchan mentioned this pull request Nov 19, 2024

fix: fix potential data loss for shared source #19443

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: pause shared SourceExecutor until a downstream actor is created #16348

feat: pause shared SourceExecutor until a downstream actor is created #16348

xxchan commented Apr 16, 2024 •

edited

Loading

xxchan Apr 20, 2024

tabVersion left a comment

chenzl25 left a comment

feat: pause shared SourceExecutor until a downstream actor is created #16348

feat: pause shared SourceExecutor until a downstream actor is created #16348

Conversation

xxchan commented Apr 16, 2024 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

xxchan Apr 20, 2024

Choose a reason for hiding this comment

tabVersion left a comment

Choose a reason for hiding this comment

chenzl25 left a comment

Choose a reason for hiding this comment

xxchan commented Apr 16, 2024 •

edited

Loading