feat(meta): support drop creating materialized views for v2 backend #17503

kwannoel · 2024-06-28T16:24:01Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Notify catalog to fe in the following places:

Similar to #17484. See that PR's description for further details.

Main difference between v1 and v2 is that in v2 we report all catalogs to fe. Because all catalogs are immediately committed.
But we don't do the same for v1 yet, since for v1, only MVs' catalogs are immediately committed.

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added test labels as necessary. See details.
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

… drop mv

This reverts commit 2dfb3e5.

…bscribe + only show created tables in mv catalog

This reverts commit b9789a9.

This reverts commit 33d94ce.

This reverts commit 99427d2.

src/config/ci.toml

kwannoel · 2024-07-08T06:34:21Z

Bump

yezizp2012

Rest LGTM.

src/meta/src/controller/streaming_job.rs

src/meta/src/barrier/mod.rs

BugenZhao · 2024-07-09T06:00:54Z

e2e_test/ddl/drop/drop_creating_mv.slt

+############## Test drop foreground mv
+onlyif can-use-recover
+system ok
+psql -h localhost -p 4566 -d dev -U root -c 'create materialized view m1 as select * from t;' &


This doesn't work under parallelism execution since the database name won't be dev anymore. 😕

Parallel execution won't run this test. Because trigger recovery in one test thread will affect the execution of another test thread.

Can use ./risedev psql -c

src/meta/src/barrier/mod.rs

kwannoel · 2024-07-15T07:17:16Z

Bump

BugenZhao

Not quite familiar with the meta part, may forward to @yezizp2012 for better insights. 🥺 Rest LGTM

BugenZhao · 2024-07-15T08:23:48Z

e2e_test/ddl/drop/drop_creating_mv.slt

+############## Test drop foreground mv
+onlyif can-use-recover
+system ok
+psql -h localhost -p 4566 -d dev -U root -c 'create materialized view m1 as select * from t;' &


Can use ./risedev psql -c

yezizp2012

LGTM, seems like risedev-env is not set correctly.

[cargo-make] INFO - Execute Command: "/usr/bin/env" "bash" "/tmp/fsio_0TY8YHlAuc.sh" "-c" "create materialized view m1 as select * from t;"
risedev-env file not found. Did you start cluster using ./risedev d or ./risedev p?

BugenZhao · 2024-07-16T09:58:18Z

ci/scripts/single-node-utils.sh

+  mkdir -p .risingwave/config
+  cat <<EOF > .risingwave/config/risedev-env
+RW_META_ADDR="http://127.0.0.1:5690"
+RISEDEV_RW_FRONTEND_LISTEN_ADDRESS="127.0.0.1"
+RISEDEV_RW_FRONTEND_PORT="4566"
+EOF


I would suggest doing this in the playground binary so that developers can also benefit from it locally. This is to address #17090. Can do it in next PRs.

…g-v2

kwannoel added 19 commits June 27, 2024 21:50

update creating catalogs

6397ec9

decouple finish create table and finish create mv

2b825e3

refactor cancel

0082554

fix

fd6ff98

handle cancel path in drop

3714af9

handle delete catalog to fe after cancel

d3d4605

filter out creating tables by default

ba91e6c

refactor created table getters + only make creating tables visible in…

cbd78a6

… drop mv

report catalog in v2, left internal tables

b9789a9

only commit internal tables

2dfb3e5

fix v1

044525f

Revert "only commit internal tables"

7f51256

This reverts commit 2dfb3e5.

add fixme for clean up stream jobs + return all tables on frontend su…

6a47d92

…bscribe + only show created tables in mv catalog

fix clean dirty tables for v1

293f034

notify internal table catalog for v2

33d94ce

provide create job obj in finish_streaming_job

99427d2

Revert "report catalog in v2, left internal tables"

884793c

This reverts commit b9789a9.

Revert "notify internal table catalog for v2"

339f089

This reverts commit 33d94ce.

Revert "provide create job obj in finish_streaming_job"

c71f181

This reverts commit 99427d2.

kwannoel changed the base branch from main to kwannoel/mark-creating June 28, 2024 16:24

github-actions bot added type/feature ci/run-e2e-single-node-tests labels Jun 28, 2024

kwannoel added ci/main-cron/run-selected ci/run-backwards-compat-tests Run backwards compatibility tests in your PR. ci/run-backfill-tests labels Jul 1, 2024

kwannoel mentioned this pull request Jul 2, 2024

feat(meta): support drop creating materialized views for v1 backend #17484

Merged

9 tasks

kwannoel force-pushed the kwannoel/mark-creating branch from ad984d3 to c71f181 Compare July 2, 2024 07:54

kwannoel added ci/run-e2e-test ci/run-recovery-test-deterministic-simulation labels Jul 2, 2024

kwannoel force-pushed the kwannoel/mark-creating-v2 branch from aa94636 to f45f741 Compare July 2, 2024 08:38

kwannoel force-pushed the kwannoel/mark-creating-v2 branch from eaa467c to db45d28 Compare July 4, 2024 06:33

Base automatically changed from kwannoel/mark-creating to main July 4, 2024 11:45

kwannoel commented Jul 4, 2024

View reviewed changes

src/config/ci.toml Outdated Show resolved Hide resolved

kwannoel and others added 4 commits July 4, 2024 22:42

Merge branch 'main' into kwannoel/mark-creating-v2

f88a5b4

fix

cb1325c

Merge branch 'main' into kwannoel/mark-creating-v2

60f4317

allow adhoc recovery when disable_recovery set

e2b75b3

kwannoel requested a review from chenzl25 July 8, 2024 06:35

yezizp2012 reviewed Jul 8, 2024

View reviewed changes

src/meta/src/controller/streaming_job.rs Outdated Show resolved Hide resolved

src/meta/src/barrier/mod.rs Show resolved Hide resolved

revert refactor of oid to object

fcf50c2

kwannoel requested a review from yezizp2012 July 8, 2024 08:24

BugenZhao reviewed Jul 9, 2024

View reviewed changes

kwannoel requested a review from BugenZhao July 15, 2024 07:17

BugenZhao reviewed Jul 15, 2024

View reviewed changes

kwannoel added 2 commits July 15, 2024 18:10

make single node compat with risedev psql

ef85fde

fix path

f62d7e6

yezizp2012 approved these changes Jul 16, 2024

View reviewed changes

fix

3eb1c6e

BugenZhao reviewed Jul 16, 2024

View reviewed changes

kwannoel mentioned this pull request Jul 16, 2024

Initialize risedev-env inside single node #17700

Closed

kwannoel added this pull request to the merge queue Jul 16, 2024

kwannoel removed this pull request from the merge queue due to a manual request Jul 16, 2024

Merge remote-tracking branch 'origin/main' into kwannoel/mark-creatin…

99186bf

…g-v2

kwannoel added this pull request to the merge queue Jul 16, 2024

Merged via the queue into main with commit 5a56574 Jul 16, 2024
32 of 34 checks passed

kwannoel deleted the kwannoel/mark-creating-v2 branch July 16, 2024 12:09

BugenZhao mentioned this pull request Oct 23, 2024

Make state table visible for creating MV #19031

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(meta): support drop creating materialized views for v2 backend #17503

feat(meta): support drop creating materialized views for v2 backend #17503

kwannoel commented Jun 28, 2024 •

edited

Loading

kwannoel commented Jul 8, 2024

yezizp2012 left a comment

BugenZhao Jul 9, 2024

kwannoel Jul 15, 2024 •

edited

Loading

BugenZhao Jul 15, 2024

kwannoel commented Jul 15, 2024

BugenZhao left a comment •

edited

Loading

BugenZhao Jul 15, 2024

yezizp2012 left a comment •

edited

Loading

BugenZhao Jul 16, 2024

kwannoel Jul 16, 2024

feat(meta): support drop creating materialized views for v2 backend #17503

feat(meta): support drop creating materialized views for v2 backend #17503

Conversation

kwannoel commented Jun 28, 2024 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

kwannoel commented Jul 8, 2024

yezizp2012 left a comment

Choose a reason for hiding this comment

BugenZhao Jul 9, 2024

Choose a reason for hiding this comment

kwannoel Jul 15, 2024 • edited Loading

Choose a reason for hiding this comment

BugenZhao Jul 15, 2024

Choose a reason for hiding this comment

kwannoel commented Jul 15, 2024

BugenZhao left a comment • edited Loading

Choose a reason for hiding this comment

BugenZhao Jul 15, 2024

Choose a reason for hiding this comment

yezizp2012 left a comment • edited Loading

Choose a reason for hiding this comment

BugenZhao Jul 16, 2024

Choose a reason for hiding this comment

kwannoel Jul 16, 2024

Choose a reason for hiding this comment

kwannoel commented Jun 28, 2024 •

edited

Loading

kwannoel Jul 15, 2024 •

edited

Loading

BugenZhao left a comment •

edited

Loading

yezizp2012 left a comment •

edited

Loading