feat(streaming): Support in memory backfill executor for mv on mv #6341
Conversation
| let executor = RearrangedChainExecutor::new( | ||
| ) | ||
| .boxed(), | ||
| ChainType::Rearrange => RearrangedChainExecutor::new( |
There was a problem hiding this comment.
Just to confirm: Will ChainExecutor or RearrangedChainExecutor be used in any cases?
There was a problem hiding this comment.
ChainExecutor is still used in StreamIndexScan as before, while RearrangedChainExecutor will be replaced by BackfillExecutor.
|
Will take a look later. 🥵 |
Codecov Report
@@ Coverage Diff @@
## main #6341 +/- ##
==========================================
- Coverage 73.94% 73.93% -0.01%
==========================================
Files 981 982 +1
Lines 159159 159928 +769
==========================================
+ Hits 117690 118245 +555
- Misses 41469 41683 +214
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
| value_indices, | ||
| ); | ||
|
|
||
| BackfillExecutor::new( |
There was a problem hiding this comment.
Previously we expected the BatchQueryExecutor to be more general to support some simple pruning, but it seems to be unused by the Backfill. I'm not sure what the future plan is, should we keep that, or totally remove that and try something like what we did in BatchLookupJoin?
There was a problem hiding this comment.
I think currently BatchQueryExecutor is only used by the StreamIndexScan. I haven't figured out why it use ChainExecutor instead of RearrangedChainExecutor before. Once we can replace all of them by BackfillExecutor I think we don't need the BatchQueryExecutor . However, once we remove the BatchQuery from the ChainNode, it seems like ChainNode is not a proper name.
I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.
What's changed and what's your intention?
PLEASE DO NOT LEAVE THIS EMPTY !!!
Please explain IN DETAIL what the changes are in this PR and why they are needed:
BackfillExecutorbased on the RFC: RFC: Use Backfill To Let Mv On Mv Stream Again rfcs#13Also we only read committed data from the upstream mv in this PR, but we can support uncommitted read later.BackfillExecutortogether with the upstream mv.ChainNodeand add achain_typefield to decide which executor will be used to create mv on mv. So at this moment,BackfillExecutoris an implementation of theChainNode.BackfillExecutorhas a special read pattern which is it will read the storage through a stream, but likely stop the reading and drop the stream without consuming all the data of the stream. This behavior has never met by storage before, so it is also some kind of testing for the storage.BackfillExecutor.Checklist
./risedev check(or alias,./risedev c)Documentation
If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.
Types of user-facing changes
Please keep the types that apply to your changes, and remove those that do not apply.
Release note
Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.
Refer to a related PR or issue link (optional)
#6275