-
Notifications
You must be signed in to change notification settings - Fork 592
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(cdc): auto schema change for mysql cdc #17876
Merged
Merged
Changes from 1 commit
Commits
Show all changes
93 commits
Select commit
Hold shift + click to select a range
606b1d0
enable ddl event
StrikeW 99c0785
split schema event to a chunk
StrikeW 245ddbe
Merge remote-tracking branch 'origin/main' into siyuan/cdc-schema-change
StrikeW dbb18a5
skeleton to parse cdc message
StrikeW bd123ff
Merge branch 'siyuan/cdc-schema-change' of github.com:risingwavelabs/…
StrikeW 5971ea2
minor
StrikeW 1d60a6a
impl mysql schema event parser
StrikeW da9d689
Merge remote-tracking branch 'origin/main' into siyuan/cdc-schema-change
StrikeW 7d2a408
meta->frontend rpc skeleton
StrikeW e1cc54a
call frontend to generate new table plan
StrikeW 58e1817
minor
StrikeW 35f9b4f
launch rpc service
StrikeW 532c31e
refactor frontend service
StrikeW f507196
add cdc table name to PbTable
StrikeW 9b85887
wip: meta->frontend rpc implementation
StrikeW 33857bb
wip: rpc source parser->meta
StrikeW 850f1d0
refine source -> meta -> frontend rpc
StrikeW 6ffeb3b
submit schema change to spawned task in source exec
StrikeW 44f736f
Merge remote-tracking branch 'origin/main' into siyuan/cdc-schema-change
StrikeW 266354b
meta->frontend rpc skeleton
StrikeW 32b9377
call frontend to generate new table plan
StrikeW 253ea41
minor
StrikeW e6cfb50
launch rpc service
StrikeW f1c41b0
refactor frontend service
StrikeW dcb359e
add cdc table name to PbTable
StrikeW 0539172
wip: meta->frontend rpc implementation
StrikeW f1fff64
wip: rpc source parser->meta
StrikeW 5d97d73
refine source -> meta -> frontend rpc
StrikeW ead027b
submit schema change to spawned task in source exec
StrikeW 7d1fe85
Merge branch 'siyuan/cdc-handle-schema-event' of github.com:risingwav…
StrikeW 1295374
wip: debuging rpc
StrikeW 82815d1
finish auto replace workflow
StrikeW 7be0555
register frontend rpc addr to meta
StrikeW 78088c9
add e2e test
StrikeW a2b2a2b
minor
StrikeW bd3cbaf
minor
StrikeW 73878f4
refine
StrikeW 09b7979
refine parsing
StrikeW 950a09d
add
StrikeW 27a7017
clean
StrikeW 158ff10
format
StrikeW a7c00d6
clippy
StrikeW 7cd2788
clean code
StrikeW 702d146
minor
StrikeW 12778aa
clean again
StrikeW 6452745
fix comment
StrikeW c50856a
minor
StrikeW a8f52e4
fix comments
StrikeW 4ce2c47
Merge branch 'siyuan/cdc-schema-change' into siyuan/cdc-handle-schema…
StrikeW b85a604
remove fullTableName
StrikeW 3532e74
Merge remote-tracking branch 'origin/siyuan/cdc-schema-change' into s…
StrikeW 22e67e8
minor
StrikeW 70c9299
fix type cast
StrikeW 2c435d9
Merge remote-tracking branch 'origin/siyuan/cdc-schema-change' into s…
StrikeW 3012b57
Merge remote-tracking branch 'origin/siyuan/cdc-handle-schema-event' …
StrikeW af8bca9
clean
StrikeW 5b1d8d6
clippy
StrikeW da9ee1b
Merge remote-tracking branch 'origin/main' into siyuan/cdc-handle-sch…
StrikeW 6d18dce
fix comments
StrikeW d1711c9
fix check
StrikeW 3038bf2
add license check
StrikeW 717f1af
Merge branch 'main' into siyuan/cdc-handle-schema-event
StrikeW 6ac586b
fix ut
StrikeW 331409a
Merge branch 'siyuan/cdc-handle-schema-event' of github.com:risingwav…
StrikeW 0dee439
minor
StrikeW 90bc49e
Merge remote-tracking branch 'origin/main' into siyuan/cdc-handle-sch…
StrikeW f17d1c2
fix sim test
StrikeW 40ccdd4
fix sim test
StrikeW ed91f7e
refactor frontend rpc addr impl
StrikeW 259c055
fix
StrikeW df65235
add feature guard flag
StrikeW 766313c
retry with grpc errors
StrikeW df80dd6
Merge remote-tracking branch 'origin/main' into siyuan/cdc-handle-sch…
StrikeW 756d840
Revert "chore: bump `tonic` to v0.12 (#17889)"
StrikeW 219a77e
minor
StrikeW ddd667a
Reapply "chore: bump `tonic` to v0.12 (#17889)"
StrikeW deefc9a
fix
StrikeW 08d355b
fix
StrikeW 5d238d2
fix
StrikeW 1b3ee00
fix
StrikeW 864fae2
rename cdc_table_name -> cdc_table_id
StrikeW a7fc383
fix
StrikeW 7609748
fix
StrikeW 52ddfeb
Merge remote-tracking branch 'origin/main' into siyuan/cdc-handle-sch…
StrikeW 844cbf0
fix ut
StrikeW b853f20
Merge remote-tracking branch 'origin/main' into siyuan/cdc-handle-sch…
StrikeW 27fa578
minor
StrikeW e0fd50c
minor
StrikeW ec742de
fix
StrikeW 31162c8
try fix
StrikeW d68aca5
init logger for test_sink_scale
StrikeW d003d2b
Merge remote-tracking branch 'origin/main' into siyuan/cdc-handle-sch…
StrikeW 533cf22
unblock ci
StrikeW File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
wip: debuging rpc
commit 129537451241a677bcbd3053513d283197010f8f
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
35 changes: 35 additions & 0 deletions
35
src/meta/model_v2/migration/src/m20240726_063833_auto_schema_change.rs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
use sea_orm_migration::prelude::{Table as MigrationTable, *}; | ||
|
||
#[derive(DeriveMigrationName)] | ||
pub struct Migration; | ||
|
||
#[async_trait::async_trait] | ||
impl MigrationTrait for Migration { | ||
async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { | ||
manager | ||
.alter_table( | ||
MigrationTable::alter() | ||
.table(Table::Table) | ||
.add_column(ColumnDef::new(Table::CdcTableName).string()) | ||
.to_owned(), | ||
) | ||
.await | ||
} | ||
|
||
async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { | ||
manager | ||
.alter_table( | ||
MigrationTable::alter() | ||
.table(Table::Table) | ||
.drop_column(Table::CdcTableName) | ||
.to_owned(), | ||
) | ||
.await | ||
} | ||
} | ||
|
||
#[derive(DeriveIden)] | ||
enum Table { | ||
Table, | ||
CdcTableName, | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
here spawn a task in source executor to handle schema change from parser