Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(connector): remove WriteGuard and fulfill_meta_column from parser #12542

Merged
merged 7 commits into from
Sep 27, 2023

Conversation

BugenZhao
Copy link
Member

@BugenZhao BugenZhao commented Sep 26, 2023

Signed-off-by: Bugen Zhao [email protected]I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

  • Since fix(parser): allow insert multiple rows in one record #6541, the WriteGuard does not make any sense in the interface. So we may totally remove this to avoid confusion.
  • For each source message, parse then fulfill_meta_column is not that clear for readability. We directly embed the metadata in the RowWriter and wrap f in do_action to handle columns with types other than Normal.

Preparation for #12366. Also some minor refactors on the JSON parser.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
@codecov
Copy link

codecov bot commented Sep 26, 2023

Codecov Report

Merging #12542 (f9b07af) into main (ea9b94c) will decrease coverage by 0.01%.
Report is 17 commits behind head on main.
The diff coverage is 54.71%.

@@            Coverage Diff             @@
##             main   #12542      +/-   ##
==========================================
- Coverage   69.40%   69.40%   -0.01%     
==========================================
  Files        1429     1429              
  Lines      238724   238709      -15     
==========================================
- Hits       165696   165680      -16     
- Misses      73028    73029       +1     
Flag Coverage Δ
rust 69.40% <54.71%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
src/connector/src/lib.rs 81.25% <ø> (ø)
src/connector/src/parser/common.rs 100.00% <100.00%> (ø)
src/connector/src/parser/canal/simd_json_parser.rs 84.74% <50.00%> (ø)
src/connector/src/parser/csv_parser.rs 87.09% <66.66%> (ø)
...connector/src/parser/debezium/mongo_json_parser.rs 81.10% <50.00%> (ø)
src/connector/src/parser/maxwell/maxwell_parser.rs 55.81% <50.00%> (ø)
src/connector/src/parser/plain_parser.rs 56.41% <50.00%> (ø)
src/connector/src/parser/unified/json.rs 63.70% <75.00%> (ø)
src/connector/src/parser/unified/util.rs 70.11% <83.33%> (ø)
src/connector/src/parser/upsert_parser.rs 67.46% <50.00%> (ø)
... and 4 more

... and 5 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

} else {
f(desc)?
) -> Result<()> {
let mut f = |desc: &SourceColumnDesc| {
Copy link
Contributor

@StrikeW StrikeW Sep 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The input parameter is also named f, I am confused which is which. You may rename the input f to column_func or something.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed wrapped f with f_with_meta.

Copy link
Contributor

@StrikeW StrikeW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest lgtm!

@BugenZhao BugenZhao force-pushed the bz/refactor-source-parser-row-writer branch from 08dd5fe to f9b07af Compare September 27, 2023 04:17
@BugenZhao BugenZhao added this pull request to the merge queue Sep 27, 2023
Merged via the queue into main with commit 590faea Sep 27, 2023
6 of 7 checks passed
@BugenZhao BugenZhao deleted the bz/refactor-source-parser-row-writer branch September 27, 2023 05:39
BugenZhao pushed a commit that referenced this pull request Sep 27, 2023
…2123)

feat(java-binding): generate jni method signature with macro (#12487)

feat(streaming): recover `no_shuffle_backfill` (#12493)

feat(test): support customized source logic in deterministic test (#12456)

refactor(storage): reorg selector (#12392)

feat(frontend): add rw_catalog.rw_hummock_sstables (#12532)

chore: bump risingwave version in docker compose to 1.2.0 (#12530)

Signed-off-by: Bugen Zhao <[email protected]>

refactor(sink): decouple sink formatter and payload writer (#12515)

chore(java-binding): refine stream chunk benchmark (#12504)

feat(meta): add create_type to `Table` (#12529)

refactor(sink): refine sink trait and macro (#12478)

refactor(backup): refine error message (#12388)

feat(sink): turn sink writer into higher level log sinker (#12152)

fix(over window): don't expect stream keys in UpdateDelete and UpdateInsert the same (#12536)

Signed-off-by: Richard Chien <[email protected]>

refactor(expr): separate function implementations into a new crate (#12485)

Signed-off-by: Runji Wang <[email protected]>

fix(config): make metrics_level opt backward compatible (#12541)

refactor(connector): replace sink writer rpc with jni (#12480)

refactor(connector): remove `WriteGuard` and `fulfill_meta_column` from parser (#12542)

Signed-off-by: Bugen Zhao <[email protected]>

feat(sink): handle stream key sink pk mismatch (#12458)

feat(optimizer): support apply hop window transpose rule (#12338)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants