-
Notifications
You must be signed in to change notification settings - Fork 594
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(stream): materialize should not compact input when handling conflict #13351
Conversation
Some(OwnedRow::new(vec![Some(8_i32.into()), Some(3_i32.into())])) | ||
Some(OwnedRow::new(vec![Some(8_i32.into()), Some(2_i32.into())])) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
U- 8 1
U+ 8 2
+ 8 3
The conflict behavior is ignore. Quesiton is if we need to consider the update as a whole operation. The U- 8 1
is trying to delete a non-existent record, so it is ignored. But if the following U+ 8 2
should be ignored too?
Some(OwnedRow::new(vec![Some(1_i32.into()), Some(4_i32.into())])) | ||
Some(OwnedRow::new(vec![Some(1_i32.into()), Some(3_i32.into())])) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
" i i
+ 1 3
+ 1 4
I think it should be 3 for a ignored behavior?
Codecov Report
@@ Coverage Diff @@
## main #13351 +/- ##
==========================================
- Coverage 67.81% 67.74% -0.07%
==========================================
Files 1524 1524
Lines 259436 259469 +33
==========================================
- Hits 175934 175787 -147
- Misses 83502 83682 +180
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 21 files with indirect coverage changes 📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would you mind writing some tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thanks for the quick fix~
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we leverage the snapshot test here?
risingwave/src/stream/tests/integration_tests/snapshot.rs
Lines 85 to 88 in b7c57d0
/// Similar to [`check_until_pending`], but use a DSL test script as input. | |
/// | |
/// For each input event, it drives the executor until it is pending. | |
pub async fn check_with_script<F, Fut>( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I will take a look later
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
fix #13346
INSERT 1, DELETE 1
can be compacted into No-op. butUPSERT 1, DELETE 1
can not because we can not know if the record that should be deleted is inserted by the upsert operation.I think we should remove the
MaterializeBuffer
structure but just leave a HashSet to help to get all the keys in the streamChunk and fetch them into the cache. And then we need to do all real operations in the state cache.Originally posted by @st1page in #13346 (comment)
Checklist
./risedev check
(or alias,./risedev c
)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.