-
Notifications
You must be signed in to change notification settings - Fork 598
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor(meta): improve create mview / scheduler readability #3748
Conversation
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
Codecov Report
@@ Coverage Diff @@
## main #3748 +/- ##
==========================================
+ Coverage 74.20% 74.22% +0.02%
==========================================
Files 797 797
Lines 112204 112191 -13
==========================================
+ Hits 83256 83275 +19
+ Misses 28948 28916 -32
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
We might distinguish different types of mappings in a clearer way. For example, when we say |
for actor in &fragment.actors { | ||
actor_to_vnode_mapping.insert(actor.actor_id, fragment.vnode_mapping.clone()); | ||
// Record vnode to parallel unit mapping for actors. | ||
let actor_to_vnode_parallel_unit_mapping = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let actor_to_vnode_parallel_unit_mapping = { | |
let actor_to_vnode_mapping = { |
vnode_mapping
might be enough, for vnode_mapping
stands for the mapping from vnode to parallel unit elsewhere in the code as well (e.g. hash_mapping.rs).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually I didn't realize the difference between ParallelUnitMapping
and ActorMapping
below and had been confused for a while. 🤣
https://github.com/singularity-data/risingwave/blob/a6449facc7a1380aab8ac41b642a1d28724a5b6e/src/meta/src/stream/stream_manager.rs#L404-L416
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is proposed by chi actually. It is designed to set apart
- mapping from vnode to parallel unit
- mapping from vnode to actor
just for clarity.
I think this could be deferred. |
Really agree with this! I'd like to refine it continuously while working on #3750. |
Signed-off-by: Bugen Zhao <[email protected]>
commit 048587d Author: lmatz <[email protected]> Date: Thu Jul 14 11:03:48 2022 +0800 fix(test): pass user name for regress tests (#3854) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 9b0d3a1 Author: Noel Kwan <[email protected]> Date: Thu Jul 14 10:13:51 2022 +0800 feat(sqlsmith): generate group by (#3805) * implement sqlsmith group by * fmt * add required cols * pass valid cols to gen_col * gen col from valid_cols if available * override bound columns for group by * capture local column context * fix case where no FROM * fmt * cleanup * clean Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 8231cf1 Author: congyi <[email protected]> Date: Thu Jul 14 00:23:16 2022 +0800 feat(row-based encoding): introduce row-based encoding by using value encoding (#3835) * introduce row-based encoding 1.0 commit 1966c93 Author: xxchan <[email protected]> Date: Wed Jul 13 17:08:09 2022 +0200 chore(proto): remove unused FilterScanNode (#3846) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 23388b9 Author: Wenzhuo Liu <[email protected]> Date: Wed Jul 13 22:48:40 2022 +0800 feat(batch): support selective aggregation (#3683) * feat(batch): support selective aggregation for count * add e2e test for selective count * add group by test for selective count * remove option * add filter to approx count distinct * add general agg and fix approx count * add general distinct agg * format * add e2e test * add selective distinct agg e2e * avoid collect * fix count star * format * fix bug in optimizer Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 2a8a96d Author: xxchan <[email protected]> Date: Wed Jul 13 16:07:03 2022 +0200 style(stream): exhaustive patterm for build_executor (#3848) commit ca49f6d Author: Li0k <[email protected]> Date: Wed Jul 13 20:55:55 2022 +0800 feat(storage): support deserialize_prefix_len for prefix_bloom_filter (#3807) * feat(storage): support deserialize_prefix_len for prefix_bloom_filter * fix(storage): fix deserialize_prefix unit-test * chore(storage): add doc for encoding_data_size * fix(storage): fix encoding_data_size to return truely length in storage_encoding commit edc6660 Author: Dylan <[email protected]> Date: Wed Jul 13 20:38:23 2022 +0800 feat(frontend): support explain verbose (#3798) * feat(frontend) support explain verbose * prefix each field name with table_name * do-apply-planner-test * minor improvement * fix batch two phase agg which should satisfy all agg-call contain no distinct and order by * fix stream two phase agg which should satisfy all agg-call contain no distinct and order by * move explain_verbose flag from session config to optimizer context * fmt * fix test case commit 4a5503e Author: Li0k <[email protected]> Date: Wed Jul 13 18:59:48 2022 +0800 fix(hash): fix hash_key_data_size (#3843) * fix(hash): fix hash_key_data_size * fix(hash): support Key8 for Boolean commit d0027c6 Author: Bugen Zhao <[email protected]> Date: Wed Jul 13 17:53:04 2022 +0800 fix(test): datagen parallel e2e (#3840) Signed-off-by: Bugen Zhao <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit aa09e44 Author: Huangjw <[email protected]> Date: Wed Jul 13 17:40:49 2022 +0800 chore(risedev): add external_labels and remote_write in prometeus config (#3825) * add remote write in prometheus config * fix config * add env PROMETHEUS_REMOTE_WRITE * cargo fmt * cargo fmt * update risedev yml * fix risedev yml * fix risedev yml Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 6e45426 Author: Alex Chi <[email protected]> Date: Wed Jul 13 17:08:41 2022 +0800 chore(streaming): check epoch continuous (#3839) * chore(streaming): check epoch continuous Signed-off-by: Alex Chi <[email protected]> * fix unit test Signed-off-by: Alex Chi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4fadf48 Author: Kaige Li <[email protected]> Date: Wed Jul 13 16:36:10 2022 +0800 feat(optimizer): distinct agg. (#3717) * feat(optimizer): distinct agg. * add comments. * add comments and tests. * correct planner test. * refactor. * improvement. * small fix. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 0f8148d Author: wzzzzd <[email protected]> Date: Wed Jul 13 16:15:17 2022 +0800 feat(common): support generating `Datum` in field generator (#3827) * add generate_datum to generator traits * add MIN and MAX to OrderedF32 and OrderedF64 * make OrderedFloat serializable and expose impl_rand in common configs * implement generate_datum * add unit tests and fix format * fix bug Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4b511e8 Author: William Wen <[email protected]> Date: Wed Jul 13 15:36:04 2022 +0800 feat(meta): support BTreeMapTransaction that avoids copying the whole BTreeMap (#3680) * feat(meta): support BTreeMapTransaction that avoids copying the whole BTreeMap * add docs Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 369b57e Author: Bohan Zhang <[email protected]> Date: Wed Jul 13 14:42:22 2022 +0800 refactor: move source connectors to source dir (#3826) move source connectors to source/ Signed-off-by: tabVersion <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 3e72311 Author: Alex Chi <[email protected]> Date: Wed Jul 13 14:19:09 2022 +0800 feat(cli): add prof and tokio env config (#3824) Signed-off-by: Alex Chi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit b16c25a Author: Richard Chien <[email protected]> Date: Wed Jul 13 13:42:40 2022 +0800 refactor(batch,agg): merge `Aggregator::update*` and refactor `SortAggExecutor` (#3794) * feed Aggregator at most one group at a time * impl new Aggregator functions for existing aggregators * impl `nth` for ArrayImplIterator to speed up `iter.skip(n)` * fix clippy warnings * add unittest for ArrayImplIterator::nth Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 3f03b2e Author: Li0k <[email protected]> Date: Wed Jul 13 12:39:16 2022 +0800 fix(type): fix decimal from_str and add e2e test (#3823) * chore(e2e): add e2e test for decimal special values * fix(type): fix decimal from_str and add e2e test * fix(type): add unit-test for decimal from_str commit eb1d49e Author: Richard Chien <[email protected]> Date: Wed Jul 13 10:33:28 2022 +0800 feat(frontend): support order by clause of agg call (#3720) * support binding agg order by clause * support planning agg call with order by clause * print order by clause of agg call * correct the generation of BatchProject * publicly expose AggOrderBy::sort_exprs * remove a useless log * fix debug format of PlanAggCall * revert changes to PlanAggCall::filter * add plan tests for order by clause * return error for some invalid syntax * use Option<OrderType> instead of Option<bool> * move ORDER BY clause in debug output * fix plan tests * use Direction instead of OrderType * remove unused derive * change default value of AggOrderByExpr::direction to Direction::Asc * add a todo comment * change nulls_first field to bool without Option * fix plan tests Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ff4c0d7 Author: Steven Chua <[email protected]> Date: Wed Jul 13 09:56:01 2022 +0800 feat(frontend): Add planning and option in frontend for Lookup Join (#3763) * feat(frontend): Add plan node for batch lookup join * feat(frontend): Add config for using lookup join * feat(frontend): Change batch lookup join to unary node * feat(frontend): Add sources for lookup join node * feat(frontend): Fix bugs in Lookup Join * feat(frontend): Add error handling when data types of two sides of equality predicate are different * feat(frontend): Fix formatting and remove extraneous code * feat(frontend): fix bug * feat(frontend): Add e2e tests * feat(frontend): Fix e2e tests * feat(frontend): Better handling for when lookup join's conditions are not met * feat(frontend): Fix bug with comparing order key and predicate indices * feat(frontend): Refactored code and renamed config to rw_batch_enable_lookup_join * feat(frontend): Fix bug with output indices * feat(frontend): Fix test * feat(frontend): Fix another test * feat(frontend): Fix tests * feat(frontend): Change name and add more e2e tests * feat(frontend): Fix formatting * feat(frontend): Fix test bug Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 7a1331b Author: Li0k <[email protected]> Date: Tue Jul 12 21:19:00 2022 +0800 fix(utils): fix decimal flag bound (#3808) commit 6141051 Author: William Wen <[email protected]> Date: Tue Jul 12 17:53:21 2022 +0800 feat(meta): add metrics in hummock manager lock and real process time (#3802) * feat(meta): add metrics in hummock manager lock and real process time * fix timer use * add lock type in metrics label * add grafana panel Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 6dadbe0 Author: Noel Kwan <[email protected]> Date: Tue Jul 12 17:13:08 2022 +0800 feat(StorageTableIter): parameterize on decoding (#3800) * refactor deserializer constructor * parameterize on deserializer * rerun ci Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 27a7ea8 Author: Noel Kwan <[email protected]> Date: Tue Jul 12 16:48:12 2022 +0800 feat(sqlsmith): Generate EquiJoin (#3613) * generalize gen table * initial skeleton * pass columns * gen nested join factor * generate join_on expr * initialize join on binop * fmt * use inner join * clean * gen qualified column names * fmt * clean * clean * add planner test for join on and cross join * add expected logical plan * fix typo * disable equijoin * revert join.yaml Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ead67f7 Author: wzzzzd <[email protected]> Date: Tue Jul 12 16:36:24 2022 +0800 test(batch): add benchmark for sort merge join (#3797) * add gen_sorted_data * modify the visibility of modules to allow benchmarking * remove data_type of gen_data and gen_sorted_data * add benchmark for sort merge join Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 7b544ad Author: William Wen <[email protected]> Date: Tue Jul 12 16:09:05 2022 +0800 refactor(storage): move hummock tests to a new crate to break crate cyclic dependency (#3795) * refactor(storage): move hummock tests to a new crate to break crate cyclic dependency * remove meta dependency for failpoint test * Update src/storage/hummock_test/src/vacuum_tests.rs Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * add workspace-hack to the new craet Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 6170ff2 Author: congyi <[email protected]> Date: Tue Jul 12 15:51:20 2022 +0800 refactor(streaming): ensure correct storage table constructing for all streaming operators (#3765) * refactor * refactor look_up * fix * fix lookup * fix lookup * remove new_for_test * fix distribution of arrangement Signed-off-by: Alex Chi <[email protected]> Co-authored-by: Alex Chi <[email protected]> commit b618974 Author: jon-chuang <[email protected]> Date: Tue Jul 12 15:32:31 2022 +0800 feat(executor): `DynamicFilter` write right table state exactly once - only if actor owns vnode = 0 (#3783) * initial * stash * do-appply-planner-test * fmt * clippy * license * minor * stash * fix required dist, output indices * fmt * stash * revert * do-apply-planner-test * success! * more e2e * minor cleanup * stash * fix * fix * fix * minor * minor * madsim * clippy * fmt * minor * fix * fix, apply code review suggestions * fix * fix, improve * simplify * clippy * refactor into sub functions * clippy * minor * fix * fix: rowsort * fmt * use literal in expression * clippy * fmt * use madsim btreemap * minor * fmt * respect previous behaviour - return empty message * bump-ci * use correct enum * add state table * fmt * prepare for cache policy * fmt * minor * minor * minor * Update src/stream/src/executor/managed_state/dynamic_filter.rs add license Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fmt * add state table for RHS * minor * fmt * add right_table to executor * minor * fmt * fix * clippy * fmt * fix double write * use correct * try to write on every msg * fix weird omissions * bump * minor * revert q18, q20 * revert q4 * remove inserts * only insert for right table * try to insert at barrier only * add asserts * debug * fix * fmt * only write on vnode = 0 * minor * minor * use bool as we need to read from RHS in the future... (for recovery)... Co-authored-by: TennyZhuang <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> commit fd0b7e2 Author: Bohan Zhang <[email protected]> Date: Tue Jul 12 14:26:12 2022 +0800 feat: kafka json append only sink (#3682) * stage Signed-off-by: tabVersion <[email protected]> * add datum_to_json_object Signed-off-by: Peng Chen <[email protected]> * stage Signed-off-by: tabVersion <[email protected]> * finishing Signed-off-by: tabVersion <[email protected]> * make risedev happy Signed-off-by: tabVersion <[email protected]> * add ut Signed-off-by: tabVersion <[email protected]> * pending error: Transaction error: Operation not valid in state Ready Signed-off-by: tabVersion <[email protected]> * fix by changing epoch number Signed-off-by: tabVersion <[email protected]> * add doc for sink trait Signed-off-by: tabVersion <[email protected]> * add comment in exec Signed-off-by: tabVersion <[email protected]> * remove take_snapshot Signed-off-by: tabVersion <[email protected]> * fix warning Signed-off-by: tabVersion <[email protected]> * fix table_id Signed-off-by: tabVersion <[email protected]> Co-authored-by: Peng Chen <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 0b65e68 Author: Alex Chi <[email protected]> Date: Tue Jul 12 14:00:53 2022 +0800 fix(dashboardv1): adapt new dispatcher design and add dynamic filter node (#3793) Signed-off-by: Alex Chi <[email protected]> commit 17bdf88 Author: TennyZhuang <[email protected]> Date: Tue Jul 12 10:21:58 2022 +0800 build(deps): use official pulsar-rs instead of fork (#3787) Signed-off-by: TennyZhuang <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 8bd5f63 Author: TennyZhuang <[email protected]> Date: Tue Jul 12 01:33:53 2022 +0800 fix(connector): support parsing non-string nexmark properties (#3784) * fix(connector): support parse non-string nexmark properties Signed-off-by: TennyZhuang <[email protected]> * still use serde_json currently Signed-off-by: TennyZhuang <[email protected]> * rollback to Box Signed-off-by: TennyZhuang <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit f93d8e3 Author: xiangjinwu <[email protected]> Date: Tue Jul 12 01:07:05 2022 +0800 refactor: remove stale AggKind::RowCount (#3774) * refactor: remove stale AggKind::RowCount * update doc Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 2ee98b5 Author: StrikeW <[email protected]> Date: Mon Jul 11 23:53:31 2022 +0800 fix(batch): fix order_by with struct table (#3782) * fix(batch): order_by with struct table * add more case Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit b7a969c Author: jon-chuang <[email protected]> Date: Mon Jul 11 22:05:53 2022 +0800 feat(executor): Use `TableCatalog` and `StateTable` in `DynamicFilter` (only writes) (#3757) * initial * stash * do-appply-planner-test * fmt * clippy * license * minor * stash * fix required dist, output indices * fmt * stash * revert * do-apply-planner-test * success! * more e2e * minor cleanup * stash * fix * fix * fix * minor * minor * madsim * clippy * fmt * minor * fix * fix, apply code review suggestions * fix * fix, improve * simplify * clippy * refactor into sub functions * clippy * minor * fix * fix: rowsort * fmt * use literal in expression * clippy * fmt * use madsim btreemap * minor * fmt * respect previous behaviour - return empty message * bump-ci * use correct enum * add state table * fmt * prepare for cache policy * fmt * minor * minor * minor * Update src/stream/src/executor/managed_state/dynamic_filter.rs add license Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fmt * add state table for RHS * minor * fmt * add right_table to executor * minor * fmt * fix * clippy * fmt * fix double write * use correct * try to write on every msg * fix weird omissions Co-authored-by: TennyZhuang <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> commit 5430e23 Author: Li0k <[email protected]> Date: Mon Jul 11 20:13:30 2022 +0800 chore(storage): remove value meta (#3779) commit 0608f0a Author: StrikeW <[email protected]> Date: Mon Jul 11 17:36:36 2022 +0800 refactor(stream): refactor state of topN executor to one state table (#3747) commit 097e3eb Author: Bugen Zhao <[email protected]> Date: Mon Jul 11 17:16:17 2022 +0800 feat(streaming): use multiple dispatcher for mview creation (#3758) * move mutation and barrier to stream_plan.proto Signed-off-by: Bugen Zhao <[email protected]> * add dispatcher when creating mview Signed-off-by: Bugen Zhao <[email protected]> * from to proto of add dispatcher Signed-off-by: Bugen Zhao <[email protected]> * put actor info to shared context Signed-off-by: Bugen Zhao <[email protected]> * add dispatcher on compute node Signed-off-by: Bugen Zhao <[email protected]> * clean up empty dispatcher Signed-off-by: Bugen Zhao <[email protected]> * cleanup Signed-off-by: Bugen Zhao <[email protected]> * fix clippy Signed-off-by: Bugen Zhao <[email protected]> * fix tests Signed-off-by: Bugen Zhao <[email protected]> * use add dispatcher for source executor Signed-off-by: Bugen Zhao <[email protected]> * rename dispatch to dispatcher Signed-off-by: Bugen Zhao <[email protected]> * add comments Signed-off-by: Bugen Zhao <[email protected]> * fix unique dispatcher id Signed-off-by: Bugen Zhao <[email protected]> * fix it Signed-off-by: Bugen Zhao <[email protected]> * use expect dead code Signed-off-by: Bugen Zhao <[email protected]> commit 59579ce Author: wzzzzd <[email protected]> Date: Mon Jul 11 16:58:54 2022 +0800 fix(batch): add first-match optimization for left semi join (#3773) add first-match optimization for left semi join commit 7d21848 Author: wzzzzd <[email protected]> Date: Mon Jul 11 16:09:01 2022 +0800 feat(common): add is_set to BitmapBuilder (#3768) add is_set to BitmapBuilder commit 78e5b66 Author: wzzzzd <[email protected]> Date: Mon Jul 11 15:33:31 2022 +0800 refactor(batch): say goodbye to the state machine in NLJ (#3724) * refactor(batch): say goodbye to the state machine * add some utilities and move right table to the outer loop * refactor * modify the unit test for left outer join to adapt to the semantical change * fix utility usage * remove unused imports * add some comments * more comments * fix format commit 7715827 Author: Zhanxiang (Patrick) Huang <[email protected]> Date: Mon Jul 11 12:51:00 2022 +0800 refactor: remove vnode_mappings in CompactTask and GroupedSstableBuilder (#3764) * refactor: remove vnode_mappings in CompactTask * remove GroupedSstableBuilder Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit a8d0c57 Author: Xinhao Xu <[email protected]> Date: Mon Jul 11 12:10:40 2022 +0800 feat(common): move observer_manager to common service (#3754) * mv observer_manager to common_service * mv observer_manager to common_service * rename commit 33c3611 Author: zwang28 <[email protected]> Date: Mon Jul 11 10:50:59 2022 +0800 fix: sync compaction group correctly (#3743) commit 00422c6 Author: Liang <[email protected]> Date: Mon Jul 11 09:57:42 2022 +0800 perf(hummmock manager): remove unnecessary hummock version deltas (#3689) * perf(hummmock manager): remove unnecessary hummock version deltas * delete multiple version deltas in one trx * do not use vartrx for hummock version * do not use vartrx for hummockversion in commit_epoch * remove checkpoint_version_id Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 208ab51 Author: Alex Chi <[email protected]> Date: Sun Jul 10 16:38:39 2022 +0800 fix(risedev): unexpected rebuilds on Linux (#3761) * fix(risedev): unexpected rebuilds on Linux Signed-off-by: Alex Chi <[email protected]> * update workspace hack Signed-off-by: Alex Chi <[email protected]> * use bash instead of fish Co-authored-by: Bugen Zhao <[email protected]> commit 301499f Author: Lee Zong Yu <[email protected]> Date: Sat Jul 9 14:51:38 2022 +0800 feat(metrics): Add Fragment Level Exchange Metrics (#3696) * Edit update.sh * Edit format error * Add payload to gitignore * Revert risedev.yml modification * Fix wrong gitignore * Fix wrong gitignore * Done coding for Fragment level exchange * Done coding for Fragment level exchange * Edit implementation error * Edit down fragment error * Delete println statement and revert configuration settings * Done coding for Fragment level exchange * Edit update.sh * Edit format error * Add payload to gitignore * Revert risedev.yml modification * Fix wrong gitignore * Done coding for Fragment level exchange * Edit implementation error * Edit down fragment error * Delete println statement and revert configuration settings * Edit Format and Resolve conflict * Resolve Test error * Resolve some error * generate dashboard * use rate_interval instead of 15s Signed-off-by: Alex Chi <[email protected]> Co-authored-by: Alex Chi <[email protected]> commit 92ee6d9 Author: Bugen Zhao <[email protected]> Date: Fri Jul 8 23:25:12 2022 +0800 refactor(meta): improve create mview / scheduler readability (#3748) * refactor(meta): improve create mview / scheduler readability Signed-off-by: Bugen Zhao <[email protected]> * add some comments for hash mapping Signed-off-by: Bugen Zhao <[email protected]> * refine docs Signed-off-by: Bugen Zhao <[email protected]> * make yuanxin happy Signed-off-by: Bugen Zhao <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 8fbe3f9 Author: congyi <[email protected]> Date: Fri Jul 8 19:03:34 2022 +0800 refactor(streaming): use table catalog for materialize executor (#3742) * use catalog for materialize executor commit 23f41ab Author: August <[email protected]> Date: Fri Jul 8 18:31:01 2022 +0800 feat: deprecate catalog ref id (#3745) feat: deprecate database/schema/table_ref_id Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit fa9e66c Author: Dylan <[email protected]> Date: Fri Jul 8 16:39:44 2022 +0800 feat(optimizer): support two phase topn and limit in batch mode (#3719) * feat(optimizer): support two phase topn and limit in batch mode * fix limit distribution in local batch mode * add test Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4b50016 Author: xxchan <[email protected]> Date: Fri Jul 8 10:23:47 2022 +0200 fix(batch): fix batch scan distribution & handle more than one table scans in one stage (#3736) * fix batch scan distribution * handle more than one table scans in one stage Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 17c0417 Author: Yuhao Su <[email protected]> Date: Fri Jul 8 14:25:09 2022 +0800 refactor(streaming): use table catalog in hash join (#3707) * use tablecatalog * fix infer_internal_table_catalog * cargo fix * fix tests * keys -> key * fix build executor * minor * fix incorrect order keys * fix comments * fix typo * add comment commit 2dc7992 Author: TennyZhuang <[email protected]> Date: Fri Jul 8 14:12:58 2022 +0800 test(nexmark): add nexmark q10 q14 e2e test (#3740) * test(nexmark): add nexmark q10 e2e test Signed-off-by: TennyZhuang <[email protected]> * test(nexmark): add nexmark q14 e2e test Signed-off-by: TennyZhuang <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit d21b2b9 Author: congyi <[email protected]> Date: Fri Jul 8 13:58:20 2022 +0800 chore(risedev): update grafana version (#3731) * update grafana version * enable auto update * "retry" commit aea1065 Author: Alex Chi <[email protected]> Date: Fri Jul 8 13:43:59 2022 +0800 fix(ci): use header fix action for license check (#3739) commit 7a86c4d Author: Tao Wu <[email protected]> Date: Fri Jul 8 10:30:39 2022 +0800 fix(sqlsmith): completely cover all exprs (#3737) commit a777217 Author: congyi <[email protected]> Date: Thu Jul 7 22:44:39 2022 +0800 refactor(storage): remove cell_based serialize/deserialize from serde (#3722) * remove cell_based serialize/deserialize from serde commit 8d97480 Author: Tao Wu <[email protected]> Date: Thu Jul 7 22:23:34 2022 +0800 fix: support decimal mul interval (#3734) commit 9b64fca Author: Steven Chua <[email protected]> Date: Thu Jul 7 21:12:14 2022 +0800 fix(risedev): Revert Makefile.toml changes (#3732) Revert Makefile.toml changes Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 19cffce Author: Richard Chien <[email protected]> Date: Thu Jul 7 20:57:15 2022 +0800 refactor(sqlparser): rename `parse_string_agg` to `parse_string_concat` (#3733) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit d767d99 Author: jon-chuang <[email protected]> Date: Thu Jul 7 20:33:06 2022 +0800 feat(executor): Basic in-application-memory `DynamicFilterExecutor`, pass tpch q11, q22 e2e tests (#3681) * initial * stash * do-appply-planner-test * fmt * clippy * license * minor * stash * fix required dist, output indices * fmt * stash * revert * do-apply-planner-test * success! * more e2e * minor cleanup * stash * fix * fix * fix * minor * minor * madsim * clippy * fmt * minor * fix * fix, apply code review suggestions * fix * fix, improve * simplify * clippy * refactor into sub functions * clippy * minor * fix * fix: rowsort * fmt * use literal in expression * clippy * fmt * use madsim btreemap * minor * fmt * respect previous behaviour - return empty message * bump-ci Co-authored-by: TennyZhuang <[email protected]> commit 07f282b Author: Alex Chi <[email protected]> Date: Thu Jul 7 19:53:06 2022 +0800 chore(deps): bump dependencies (#3725) Signed-off-by: Alex Chi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 95586c8 Author: Yuhao Su <[email protected]> Date: Thu Jul 7 19:02:54 2022 +0800 test(streaming): enable streaming e2e test for tpch q4, q18, q20 (#3718) enable e2e commit ca0f431 Author: Alex Chi <[email protected]> Date: Thu Jul 7 18:11:07 2022 +0800 fix(metrics): use current actor id for backpressure (#3721) * fix(metrics): use current actor id for backpressure Signed-off-by: Alex Chi <[email protected]> * fix Signed-off-by: Alex Chi <[email protected]> commit 477a4be Author: August <[email protected]> Date: Thu Jul 7 17:15:27 2022 +0800 fix: fix heartbeat req field and meta client impl (#3716) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit f9362b1 Author: Bowen <[email protected]> Date: Thu Jul 7 17:03:13 2022 +0800 doc: add relational table schema doc + move files (#3712) * doc: add relational table schema doc + move files * fix comment * Update docs/relational_table/relational-table-schema.md Co-authored-by: congyi <[email protected]> Co-authored-by: congyi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4a92eb6 Author: TennyZhuang <[email protected]> Date: Thu Jul 7 16:50:54 2022 +0800 test(nexmark): explain q16 q17 (#3715) Signed-off-by: TennyZhuang <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit e1fd06b Author: xxchan <[email protected]> Date: Thu Jul 7 10:30:57 2022 +0200 feat(batch): prune scan partition according to scan_range (#3698) * prune vnode according to scan_range * disallow accessing other vnodes from batch scan * improve style * add ut * fix doc about dist key * fix Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ed4cb79 Author: Alex Chi <[email protected]> Date: Thu Jul 7 16:11:15 2022 +0800 feat(storage): default block size = 1024 (#3463) Signed-off-by: Alex Chi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 3bb55cf Author: zwang28 <[email protected]> Date: Thu Jul 7 15:55:55 2022 +0800 fix: correctly implement check_state_consistency (#3711) commit 078c2f6 Author: xiangjinwu <[email protected]> Date: Thu Jul 7 15:35:38 2022 +0800 refactor(optimizer): cleanup LogicalAgg::prune_col (#3663) * refactor(optimizer): cleanup LogicalAgg::prune_col * planner test of prune-filter bug Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 82ef7ec Author: Tao Wu <[email protected]> Date: Thu Jul 7 14:54:18 2022 +0800 fix: reduce logging in the critical path (#3704) * fix: reduce logging in the critical path * fix cargo check Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 79eb45b Author: William Wen <[email protected]> Date: Thu Jul 7 14:42:14 2022 +0800 feat(meta): add monitoring meta process metrics (#3706) * feat(meta): add monitoring meta process metrics * update procfs dependency * add libc dependency Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Signed-off-by: Little-Wallace <[email protected]>
* fix do not schedule compact task Signed-off-by: Little-Wallace <[email protected]> * add unit aware compaction Signed-off-by: Little-Wallace <[email protected]> * fix strategy Signed-off-by: Little-Wallace <[email protected]> * fix conflict Signed-off-by: Little-Wallace <[email protected]> * fix strategy Signed-off-by: Little-Wallace <[email protected]> * tmp commit Signed-off-by: Little-Wallace <[email protected]> * remove unit id Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix meta Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix fmt Signed-off-by: Little-Wallace <[email protected]> * fix config Signed-off-by: Little-Wallace <[email protected]> * fix a large compaction Signed-off-by: Little-Wallace <[email protected]> * remove unit id Signed-off-by: Little-Wallace <[email protected]> * refactor overlap compaction picker Signed-off-by: Little-Wallace <[email protected]> * support sub level Signed-off-by: Little-Wallace <[email protected]> * refactor strategy Signed-off-by: Little-Wallace <[email protected]> * Squashed commit of the following: commit 048587d Author: lmatz <[email protected]> Date: Thu Jul 14 11:03:48 2022 +0800 fix(test): pass user name for regress tests (#3854) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 9b0d3a1 Author: Noel Kwan <[email protected]> Date: Thu Jul 14 10:13:51 2022 +0800 feat(sqlsmith): generate group by (#3805) * implement sqlsmith group by * fmt * add required cols * pass valid cols to gen_col * gen col from valid_cols if available * override bound columns for group by * capture local column context * fix case where no FROM * fmt * cleanup * clean Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 8231cf1 Author: congyi <[email protected]> Date: Thu Jul 14 00:23:16 2022 +0800 feat(row-based encoding): introduce row-based encoding by using value encoding (#3835) * introduce row-based encoding 1.0 commit 1966c93 Author: xxchan <[email protected]> Date: Wed Jul 13 17:08:09 2022 +0200 chore(proto): remove unused FilterScanNode (#3846) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 23388b9 Author: Wenzhuo Liu <[email protected]> Date: Wed Jul 13 22:48:40 2022 +0800 feat(batch): support selective aggregation (#3683) * feat(batch): support selective aggregation for count * add e2e test for selective count * add group by test for selective count * remove option * add filter to approx count distinct * add general agg and fix approx count * add general distinct agg * format * add e2e test * add selective distinct agg e2e * avoid collect * fix count star * format * fix bug in optimizer Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 2a8a96d Author: xxchan <[email protected]> Date: Wed Jul 13 16:07:03 2022 +0200 style(stream): exhaustive patterm for build_executor (#3848) commit ca49f6d Author: Li0k <[email protected]> Date: Wed Jul 13 20:55:55 2022 +0800 feat(storage): support deserialize_prefix_len for prefix_bloom_filter (#3807) * feat(storage): support deserialize_prefix_len for prefix_bloom_filter * fix(storage): fix deserialize_prefix unit-test * chore(storage): add doc for encoding_data_size * fix(storage): fix encoding_data_size to return truely length in storage_encoding commit edc6660 Author: Dylan <[email protected]> Date: Wed Jul 13 20:38:23 2022 +0800 feat(frontend): support explain verbose (#3798) * feat(frontend) support explain verbose * prefix each field name with table_name * do-apply-planner-test * minor improvement * fix batch two phase agg which should satisfy all agg-call contain no distinct and order by * fix stream two phase agg which should satisfy all agg-call contain no distinct and order by * move explain_verbose flag from session config to optimizer context * fmt * fix test case commit 4a5503e Author: Li0k <[email protected]> Date: Wed Jul 13 18:59:48 2022 +0800 fix(hash): fix hash_key_data_size (#3843) * fix(hash): fix hash_key_data_size * fix(hash): support Key8 for Boolean commit d0027c6 Author: Bugen Zhao <[email protected]> Date: Wed Jul 13 17:53:04 2022 +0800 fix(test): datagen parallel e2e (#3840) Signed-off-by: Bugen Zhao <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit aa09e44 Author: Huangjw <[email protected]> Date: Wed Jul 13 17:40:49 2022 +0800 chore(risedev): add external_labels and remote_write in prometeus config (#3825) * add remote write in prometheus config * fix config * add env PROMETHEUS_REMOTE_WRITE * cargo fmt * cargo fmt * update risedev yml * fix risedev yml * fix risedev yml Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 6e45426 Author: Alex Chi <[email protected]> Date: Wed Jul 13 17:08:41 2022 +0800 chore(streaming): check epoch continuous (#3839) * chore(streaming): check epoch continuous Signed-off-by: Alex Chi <[email protected]> * fix unit test Signed-off-by: Alex Chi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4fadf48 Author: Kaige Li <[email protected]> Date: Wed Jul 13 16:36:10 2022 +0800 feat(optimizer): distinct agg. (#3717) * feat(optimizer): distinct agg. * add comments. * add comments and tests. * correct planner test. * refactor. * improvement. * small fix. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 0f8148d Author: wzzzzd <[email protected]> Date: Wed Jul 13 16:15:17 2022 +0800 feat(common): support generating `Datum` in field generator (#3827) * add generate_datum to generator traits * add MIN and MAX to OrderedF32 and OrderedF64 * make OrderedFloat serializable and expose impl_rand in common configs * implement generate_datum * add unit tests and fix format * fix bug Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4b511e8 Author: William Wen <[email protected]> Date: Wed Jul 13 15:36:04 2022 +0800 feat(meta): support BTreeMapTransaction that avoids copying the whole BTreeMap (#3680) * feat(meta): support BTreeMapTransaction that avoids copying the whole BTreeMap * add docs Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 369b57e Author: Bohan Zhang <[email protected]> Date: Wed Jul 13 14:42:22 2022 +0800 refactor: move source connectors to source dir (#3826) move source connectors to source/ Signed-off-by: tabVersion <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 3e72311 Author: Alex Chi <[email protected]> Date: Wed Jul 13 14:19:09 2022 +0800 feat(cli): add prof and tokio env config (#3824) Signed-off-by: Alex Chi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit b16c25a Author: Richard Chien <[email protected]> Date: Wed Jul 13 13:42:40 2022 +0800 refactor(batch,agg): merge `Aggregator::update*` and refactor `SortAggExecutor` (#3794) * feed Aggregator at most one group at a time * impl new Aggregator functions for existing aggregators * impl `nth` for ArrayImplIterator to speed up `iter.skip(n)` * fix clippy warnings * add unittest for ArrayImplIterator::nth Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 3f03b2e Author: Li0k <[email protected]> Date: Wed Jul 13 12:39:16 2022 +0800 fix(type): fix decimal from_str and add e2e test (#3823) * chore(e2e): add e2e test for decimal special values * fix(type): fix decimal from_str and add e2e test * fix(type): add unit-test for decimal from_str commit eb1d49e Author: Richard Chien <[email protected]> Date: Wed Jul 13 10:33:28 2022 +0800 feat(frontend): support order by clause of agg call (#3720) * support binding agg order by clause * support planning agg call with order by clause * print order by clause of agg call * correct the generation of BatchProject * publicly expose AggOrderBy::sort_exprs * remove a useless log * fix debug format of PlanAggCall * revert changes to PlanAggCall::filter * add plan tests for order by clause * return error for some invalid syntax * use Option<OrderType> instead of Option<bool> * move ORDER BY clause in debug output * fix plan tests * use Direction instead of OrderType * remove unused derive * change default value of AggOrderByExpr::direction to Direction::Asc * add a todo comment * change nulls_first field to bool without Option * fix plan tests Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ff4c0d7 Author: Steven Chua <[email protected]> Date: Wed Jul 13 09:56:01 2022 +0800 feat(frontend): Add planning and option in frontend for Lookup Join (#3763) * feat(frontend): Add plan node for batch lookup join * feat(frontend): Add config for using lookup join * feat(frontend): Change batch lookup join to unary node * feat(frontend): Add sources for lookup join node * feat(frontend): Fix bugs in Lookup Join * feat(frontend): Add error handling when data types of two sides of equality predicate are different * feat(frontend): Fix formatting and remove extraneous code * feat(frontend): fix bug * feat(frontend): Add e2e tests * feat(frontend): Fix e2e tests * feat(frontend): Better handling for when lookup join's conditions are not met * feat(frontend): Fix bug with comparing order key and predicate indices * feat(frontend): Refactored code and renamed config to rw_batch_enable_lookup_join * feat(frontend): Fix bug with output indices * feat(frontend): Fix test * feat(frontend): Fix another test * feat(frontend): Fix tests * feat(frontend): Change name and add more e2e tests * feat(frontend): Fix formatting * feat(frontend): Fix test bug Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 7a1331b Author: Li0k <[email protected]> Date: Tue Jul 12 21:19:00 2022 +0800 fix(utils): fix decimal flag bound (#3808) commit 6141051 Author: William Wen <[email protected]> Date: Tue Jul 12 17:53:21 2022 +0800 feat(meta): add metrics in hummock manager lock and real process time (#3802) * feat(meta): add metrics in hummock manager lock and real process time * fix timer use * add lock type in metrics label * add grafana panel Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 6dadbe0 Author: Noel Kwan <[email protected]> Date: Tue Jul 12 17:13:08 2022 +0800 feat(StorageTableIter): parameterize on decoding (#3800) * refactor deserializer constructor * parameterize on deserializer * rerun ci Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 27a7ea8 Author: Noel Kwan <[email protected]> Date: Tue Jul 12 16:48:12 2022 +0800 feat(sqlsmith): Generate EquiJoin (#3613) * generalize gen table * initial skeleton * pass columns * gen nested join factor * generate join_on expr * initialize join on binop * fmt * use inner join * clean * gen qualified column names * fmt * clean * clean * add planner test for join on and cross join * add expected logical plan * fix typo * disable equijoin * revert join.yaml Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ead67f7 Author: wzzzzd <[email protected]> Date: Tue Jul 12 16:36:24 2022 +0800 test(batch): add benchmark for sort merge join (#3797) * add gen_sorted_data * modify the visibility of modules to allow benchmarking * remove data_type of gen_data and gen_sorted_data * add benchmark for sort merge join Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 7b544ad Author: William Wen <[email protected]> Date: Tue Jul 12 16:09:05 2022 +0800 refactor(storage): move hummock tests to a new crate to break crate cyclic dependency (#3795) * refactor(storage): move hummock tests to a new crate to break crate cyclic dependency * remove meta dependency for failpoint test * Update src/storage/hummock_test/src/vacuum_tests.rs Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * add workspace-hack to the new craet Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 6170ff2 Author: congyi <[email protected]> Date: Tue Jul 12 15:51:20 2022 +0800 refactor(streaming): ensure correct storage table constructing for all streaming operators (#3765) * refactor * refactor look_up * fix * fix lookup * fix lookup * remove new_for_test * fix distribution of arrangement Signed-off-by: Alex Chi <[email protected]> Co-authored-by: Alex Chi <[email protected]> commit b618974 Author: jon-chuang <[email protected]> Date: Tue Jul 12 15:32:31 2022 +0800 feat(executor): `DynamicFilter` write right table state exactly once - only if actor owns vnode = 0 (#3783) * initial * stash * do-appply-planner-test * fmt * clippy * license * minor * stash * fix required dist, output indices * fmt * stash * revert * do-apply-planner-test * success! * more e2e * minor cleanup * stash * fix * fix * fix * minor * minor * madsim * clippy * fmt * minor * fix * fix, apply code review suggestions * fix * fix, improve * simplify * clippy * refactor into sub functions * clippy * minor * fix * fix: rowsort * fmt * use literal in expression * clippy * fmt * use madsim btreemap * minor * fmt * respect previous behaviour - return empty message * bump-ci * use correct enum * add state table * fmt * prepare for cache policy * fmt * minor * minor * minor * Update src/stream/src/executor/managed_state/dynamic_filter.rs add license Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fmt * add state table for RHS * minor * fmt * add right_table to executor * minor * fmt * fix * clippy * fmt * fix double write * use correct * try to write on every msg * fix weird omissions * bump * minor * revert q18, q20 * revert q4 * remove inserts * only insert for right table * try to insert at barrier only * add asserts * debug * fix * fmt * only write on vnode = 0 * minor * minor * use bool as we need to read from RHS in the future... (for recovery)... Co-authored-by: TennyZhuang <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> commit fd0b7e2 Author: Bohan Zhang <[email protected]> Date: Tue Jul 12 14:26:12 2022 +0800 feat: kafka json append only sink (#3682) * stage Signed-off-by: tabVersion <[email protected]> * add datum_to_json_object Signed-off-by: Peng Chen <[email protected]> * stage Signed-off-by: tabVersion <[email protected]> * finishing Signed-off-by: tabVersion <[email protected]> * make risedev happy Signed-off-by: tabVersion <[email protected]> * add ut Signed-off-by: tabVersion <[email protected]> * pending error: Transaction error: Operation not valid in state Ready Signed-off-by: tabVersion <[email protected]> * fix by changing epoch number Signed-off-by: tabVersion <[email protected]> * add doc for sink trait Signed-off-by: tabVersion <[email protected]> * add comment in exec Signed-off-by: tabVersion <[email protected]> * remove take_snapshot Signed-off-by: tabVersion <[email protected]> * fix warning Signed-off-by: tabVersion <[email protected]> * fix table_id Signed-off-by: tabVersion <[email protected]> Co-authored-by: Peng Chen <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 0b65e68 Author: Alex Chi <[email protected]> Date: Tue Jul 12 14:00:53 2022 +0800 fix(dashboardv1): adapt new dispatcher design and add dynamic filter node (#3793) Signed-off-by: Alex Chi <[email protected]> commit 17bdf88 Author: TennyZhuang <[email protected]> Date: Tue Jul 12 10:21:58 2022 +0800 build(deps): use official pulsar-rs instead of fork (#3787) Signed-off-by: TennyZhuang <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 8bd5f63 Author: TennyZhuang <[email protected]> Date: Tue Jul 12 01:33:53 2022 +0800 fix(connector): support parsing non-string nexmark properties (#3784) * fix(connector): support parse non-string nexmark properties Signed-off-by: TennyZhuang <[email protected]> * still use serde_json currently Signed-off-by: TennyZhuang <[email protected]> * rollback to Box Signed-off-by: TennyZhuang <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit f93d8e3 Author: xiangjinwu <[email protected]> Date: Tue Jul 12 01:07:05 2022 +0800 refactor: remove stale AggKind::RowCount (#3774) * refactor: remove stale AggKind::RowCount * update doc Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 2ee98b5 Author: StrikeW <[email protected]> Date: Mon Jul 11 23:53:31 2022 +0800 fix(batch): fix order_by with struct table (#3782) * fix(batch): order_by with struct table * add more case Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit b7a969c Author: jon-chuang <[email protected]> Date: Mon Jul 11 22:05:53 2022 +0800 feat(executor): Use `TableCatalog` and `StateTable` in `DynamicFilter` (only writes) (#3757) * initial * stash * do-appply-planner-test * fmt * clippy * license * minor * stash * fix required dist, output indices * fmt * stash * revert * do-apply-planner-test * success! * more e2e * minor cleanup * stash * fix * fix * fix * minor * minor * madsim * clippy * fmt * minor * fix * fix, apply code review suggestions * fix * fix, improve * simplify * clippy * refactor into sub functions * clippy * minor * fix * fix: rowsort * fmt * use literal in expression * clippy * fmt * use madsim btreemap * minor * fmt * respect previous behaviour - return empty message * bump-ci * use correct enum * add state table * fmt * prepare for cache policy * fmt * minor * minor * minor * Update src/stream/src/executor/managed_state/dynamic_filter.rs add license Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fmt * add state table for RHS * minor * fmt * add right_table to executor * minor * fmt * fix * clippy * fmt * fix double write * use correct * try to write on every msg * fix weird omissions Co-authored-by: TennyZhuang <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> commit 5430e23 Author: Li0k <[email protected]> Date: Mon Jul 11 20:13:30 2022 +0800 chore(storage): remove value meta (#3779) commit 0608f0a Author: StrikeW <[email protected]> Date: Mon Jul 11 17:36:36 2022 +0800 refactor(stream): refactor state of topN executor to one state table (#3747) commit 097e3eb Author: Bugen Zhao <[email protected]> Date: Mon Jul 11 17:16:17 2022 +0800 feat(streaming): use multiple dispatcher for mview creation (#3758) * move mutation and barrier to stream_plan.proto Signed-off-by: Bugen Zhao <[email protected]> * add dispatcher when creating mview Signed-off-by: Bugen Zhao <[email protected]> * from to proto of add dispatcher Signed-off-by: Bugen Zhao <[email protected]> * put actor info to shared context Signed-off-by: Bugen Zhao <[email protected]> * add dispatcher on compute node Signed-off-by: Bugen Zhao <[email protected]> * clean up empty dispatcher Signed-off-by: Bugen Zhao <[email protected]> * cleanup Signed-off-by: Bugen Zhao <[email protected]> * fix clippy Signed-off-by: Bugen Zhao <[email protected]> * fix tests Signed-off-by: Bugen Zhao <[email protected]> * use add dispatcher for source executor Signed-off-by: Bugen Zhao <[email protected]> * rename dispatch to dispatcher Signed-off-by: Bugen Zhao <[email protected]> * add comments Signed-off-by: Bugen Zhao <[email protected]> * fix unique dispatcher id Signed-off-by: Bugen Zhao <[email protected]> * fix it Signed-off-by: Bugen Zhao <[email protected]> * use expect dead code Signed-off-by: Bugen Zhao <[email protected]> commit 59579ce Author: wzzzzd <[email protected]> Date: Mon Jul 11 16:58:54 2022 +0800 fix(batch): add first-match optimization for left semi join (#3773) add first-match optimization for left semi join commit 7d21848 Author: wzzzzd <[email protected]> Date: Mon Jul 11 16:09:01 2022 +0800 feat(common): add is_set to BitmapBuilder (#3768) add is_set to BitmapBuilder commit 78e5b66 Author: wzzzzd <[email protected]> Date: Mon Jul 11 15:33:31 2022 +0800 refactor(batch): say goodbye to the state machine in NLJ (#3724) * refactor(batch): say goodbye to the state machine * add some utilities and move right table to the outer loop * refactor * modify the unit test for left outer join to adapt to the semantical change * fix utility usage * remove unused imports * add some comments * more comments * fix format commit 7715827 Author: Zhanxiang (Patrick) Huang <[email protected]> Date: Mon Jul 11 12:51:00 2022 +0800 refactor: remove vnode_mappings in CompactTask and GroupedSstableBuilder (#3764) * refactor: remove vnode_mappings in CompactTask * remove GroupedSstableBuilder Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit a8d0c57 Author: Xinhao Xu <[email protected]> Date: Mon Jul 11 12:10:40 2022 +0800 feat(common): move observer_manager to common service (#3754) * mv observer_manager to common_service * mv observer_manager to common_service * rename commit 33c3611 Author: zwang28 <[email protected]> Date: Mon Jul 11 10:50:59 2022 +0800 fix: sync compaction group correctly (#3743) commit 00422c6 Author: Liang <[email protected]> Date: Mon Jul 11 09:57:42 2022 +0800 perf(hummmock manager): remove unnecessary hummock version deltas (#3689) * perf(hummmock manager): remove unnecessary hummock version deltas * delete multiple version deltas in one trx * do not use vartrx for hummock version * do not use vartrx for hummockversion in commit_epoch * remove checkpoint_version_id Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 208ab51 Author: Alex Chi <[email protected]> Date: Sun Jul 10 16:38:39 2022 +0800 fix(risedev): unexpected rebuilds on Linux (#3761) * fix(risedev): unexpected rebuilds on Linux Signed-off-by: Alex Chi <[email protected]> * update workspace hack Signed-off-by: Alex Chi <[email protected]> * use bash instead of fish Co-authored-by: Bugen Zhao <[email protected]> commit 301499f Author: Lee Zong Yu <[email protected]> Date: Sat Jul 9 14:51:38 2022 +0800 feat(metrics): Add Fragment Level Exchange Metrics (#3696) * Edit update.sh * Edit format error * Add payload to gitignore * Revert risedev.yml modification * Fix wrong gitignore * Fix wrong gitignore * Done coding for Fragment level exchange * Done coding for Fragment level exchange * Edit implementation error * Edit down fragment error * Delete println statement and revert configuration settings * Done coding for Fragment level exchange * Edit update.sh * Edit format error * Add payload to gitignore * Revert risedev.yml modification * Fix wrong gitignore * Done coding for Fragment level exchange * Edit implementation error * Edit down fragment error * Delete println statement and revert configuration settings * Edit Format and Resolve conflict * Resolve Test error * Resolve some error * generate dashboard * use rate_interval instead of 15s Signed-off-by: Alex Chi <[email protected]> Co-authored-by: Alex Chi <[email protected]> commit 92ee6d9 Author: Bugen Zhao <[email protected]> Date: Fri Jul 8 23:25:12 2022 +0800 refactor(meta): improve create mview / scheduler readability (#3748) * refactor(meta): improve create mview / scheduler readability Signed-off-by: Bugen Zhao <[email protected]> * add some comments for hash mapping Signed-off-by: Bugen Zhao <[email protected]> * refine docs Signed-off-by: Bugen Zhao <[email protected]> * make yuanxin happy Signed-off-by: Bugen Zhao <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 8fbe3f9 Author: congyi <[email protected]> Date: Fri Jul 8 19:03:34 2022 +0800 refactor(streaming): use table catalog for materialize executor (#3742) * use catalog for materialize executor commit 23f41ab Author: August <[email protected]> Date: Fri Jul 8 18:31:01 2022 +0800 feat: deprecate catalog ref id (#3745) feat: deprecate database/schema/table_ref_id Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit fa9e66c Author: Dylan <[email protected]> Date: Fri Jul 8 16:39:44 2022 +0800 feat(optimizer): support two phase topn and limit in batch mode (#3719) * feat(optimizer): support two phase topn and limit in batch mode * fix limit distribution in local batch mode * add test Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4b50016 Author: xxchan <[email protected]> Date: Fri Jul 8 10:23:47 2022 +0200 fix(batch): fix batch scan distribution & handle more than one table scans in one stage (#3736) * fix batch scan distribution * handle more than one table scans in one stage Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 17c0417 Author: Yuhao Su <[email protected]> Date: Fri Jul 8 14:25:09 2022 +0800 refactor(streaming): use table catalog in hash join (#3707) * use tablecatalog * fix infer_internal_table_catalog * cargo fix * fix tests * keys -> key * fix build executor * minor * fix incorrect order keys * fix comments * fix typo * add comment commit 2dc7992 Author: TennyZhuang <[email protected]> Date: Fri Jul 8 14:12:58 2022 +0800 test(nexmark): add nexmark q10 q14 e2e test (#3740) * test(nexmark): add nexmark q10 e2e test Signed-off-by: TennyZhuang <[email protected]> * test(nexmark): add nexmark q14 e2e test Signed-off-by: TennyZhuang <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit d21b2b9 Author: congyi <[email protected]> Date: Fri Jul 8 13:58:20 2022 +0800 chore(risedev): update grafana version (#3731) * update grafana version * enable auto update * "retry" commit aea1065 Author: Alex Chi <[email protected]> Date: Fri Jul 8 13:43:59 2022 +0800 fix(ci): use header fix action for license check (#3739) commit 7a86c4d Author: Tao Wu <[email protected]> Date: Fri Jul 8 10:30:39 2022 +0800 fix(sqlsmith): completely cover all exprs (#3737) commit a777217 Author: congyi <[email protected]> Date: Thu Jul 7 22:44:39 2022 +0800 refactor(storage): remove cell_based serialize/deserialize from serde (#3722) * remove cell_based serialize/deserialize from serde commit 8d97480 Author: Tao Wu <[email protected]> Date: Thu Jul 7 22:23:34 2022 +0800 fix: support decimal mul interval (#3734) commit 9b64fca Author: Steven Chua <[email protected]> Date: Thu Jul 7 21:12:14 2022 +0800 fix(risedev): Revert Makefile.toml changes (#3732) Revert Makefile.toml changes Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 19cffce Author: Richard Chien <[email protected]> Date: Thu Jul 7 20:57:15 2022 +0800 refactor(sqlparser): rename `parse_string_agg` to `parse_string_concat` (#3733) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit d767d99 Author: jon-chuang <[email protected]> Date: Thu Jul 7 20:33:06 2022 +0800 feat(executor): Basic in-application-memory `DynamicFilterExecutor`, pass tpch q11, q22 e2e tests (#3681) * initial * stash * do-appply-planner-test * fmt * clippy * license * minor * stash * fix required dist, output indices * fmt * stash * revert * do-apply-planner-test * success! * more e2e * minor cleanup * stash * fix * fix * fix * minor * minor * madsim * clippy * fmt * minor * fix * fix, apply code review suggestions * fix * fix, improve * simplify * clippy * refactor into sub functions * clippy * minor * fix * fix: rowsort * fmt * use literal in expression * clippy * fmt * use madsim btreemap * minor * fmt * respect previous behaviour - return empty message * bump-ci Co-authored-by: TennyZhuang <[email protected]> commit 07f282b Author: Alex Chi <[email protected]> Date: Thu Jul 7 19:53:06 2022 +0800 chore(deps): bump dependencies (#3725) Signed-off-by: Alex Chi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 95586c8 Author: Yuhao Su <[email protected]> Date: Thu Jul 7 19:02:54 2022 +0800 test(streaming): enable streaming e2e test for tpch q4, q18, q20 (#3718) enable e2e commit ca0f431 Author: Alex Chi <[email protected]> Date: Thu Jul 7 18:11:07 2022 +0800 fix(metrics): use current actor id for backpressure (#3721) * fix(metrics): use current actor id for backpressure Signed-off-by: Alex Chi <[email protected]> * fix Signed-off-by: Alex Chi <[email protected]> commit 477a4be Author: August <[email protected]> Date: Thu Jul 7 17:15:27 2022 +0800 fix: fix heartbeat req field and meta client impl (#3716) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit f9362b1 Author: Bowen <[email protected]> Date: Thu Jul 7 17:03:13 2022 +0800 doc: add relational table schema doc + move files (#3712) * doc: add relational table schema doc + move files * fix comment * Update docs/relational_table/relational-table-schema.md Co-authored-by: congyi <[email protected]> Co-authored-by: congyi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4a92eb6 Author: TennyZhuang <[email protected]> Date: Thu Jul 7 16:50:54 2022 +0800 test(nexmark): explain q16 q17 (#3715) Signed-off-by: TennyZhuang <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit e1fd06b Author: xxchan <[email protected]> Date: Thu Jul 7 10:30:57 2022 +0200 feat(batch): prune scan partition according to scan_range (#3698) * prune vnode according to scan_range * disallow accessing other vnodes from batch scan * improve style * add ut * fix doc about dist key * fix Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ed4cb79 Author: Alex Chi <[email protected]> Date: Thu Jul 7 16:11:15 2022 +0800 feat(storage): default block size = 1024 (#3463) Signed-off-by: Alex Chi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 3bb55cf Author: zwang28 <[email protected]> Date: Thu Jul 7 15:55:55 2022 +0800 fix: correctly implement check_state_consistency (#3711) commit 078c2f6 Author: xiangjinwu <[email protected]> Date: Thu Jul 7 15:35:38 2022 +0800 refactor(optimizer): cleanup LogicalAgg::prune_col (#3663) * refactor(optimizer): cleanup LogicalAgg::prune_col * planner test of prune-filter bug Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 82ef7ec Author: Tao Wu <[email protected]> Date: Thu Jul 7 14:54:18 2022 +0800 fix: reduce logging in the critical path (#3704) * fix: reduce logging in the critical path * fix cargo check Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 79eb45b Author: William Wen <[email protected]> Date: Thu Jul 7 14:42:14 2022 +0800 feat(meta): add monitoring meta process metrics (#3706) * feat(meta): add monitoring meta process metrics * update procfs dependency * add libc dependency Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Signed-off-by: Little-Wallace <[email protected]> * fix format Signed-off-by: Little-Wallace <[email protected]> * fix large compaction Signed-off-by: Little-Wallace <[email protected]> * merge e2e_test from main Signed-off-by: Little-Wallace <[email protected]> * simple compact Signed-off-by: Little-Wallace <[email protected]> * refactor splits Signed-off-by: Little-Wallace <[email protected]> * fix splits Signed-off-by: Little-Wallace <[email protected]> * fix bug Signed-off-by: Little-Wallace <[email protected]> * fix unit test Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * tmp commit Signed-off-by: Little-Wallace <[email protected]> * fix conflict Signed-off-by: Little-Wallace <[email protected]> * fix test bug Signed-off-by: Little-Wallace <[email protected]> * fix version Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix bug Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix clippy warn Signed-off-by: Little-Wallace <[email protected]> * fix conflict Signed-off-by: Little-Wallace <[email protected]> * fix format Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * split tier compaction file Signed-off-by: Little-Wallace <[email protected]> * fix license Signed-off-by: Little-Wallace <[email protected]> * fix trivial move Signed-off-by: Little-Wallace <[email protected]> * fix compact Signed-off-by: Little-Wallace <[email protected]> * fix conflict Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix compactor Signed-off-by: Little-Wallace <[email protected]> * fix commit Signed-off-by: Little-Wallace <[email protected]> * fix conflict Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…avelabs#3748) * refactor(meta): improve create mview / scheduler readability Signed-off-by: Bugen Zhao <[email protected]> * add some comments for hash mapping Signed-off-by: Bugen Zhao <[email protected]> * refine docs Signed-off-by: Bugen Zhao <[email protected]> * make yuanxin happy Signed-off-by: Bugen Zhao <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* fix do not schedule compact task Signed-off-by: Little-Wallace <[email protected]> * add unit aware compaction Signed-off-by: Little-Wallace <[email protected]> * fix strategy Signed-off-by: Little-Wallace <[email protected]> * fix conflict Signed-off-by: Little-Wallace <[email protected]> * fix strategy Signed-off-by: Little-Wallace <[email protected]> * tmp commit Signed-off-by: Little-Wallace <[email protected]> * remove unit id Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix meta Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix fmt Signed-off-by: Little-Wallace <[email protected]> * fix config Signed-off-by: Little-Wallace <[email protected]> * fix a large compaction Signed-off-by: Little-Wallace <[email protected]> * remove unit id Signed-off-by: Little-Wallace <[email protected]> * refactor overlap compaction picker Signed-off-by: Little-Wallace <[email protected]> * support sub level Signed-off-by: Little-Wallace <[email protected]> * refactor strategy Signed-off-by: Little-Wallace <[email protected]> * Squashed commit of the following: commit 048587d Author: lmatz <[email protected]> Date: Thu Jul 14 11:03:48 2022 +0800 fix(test): pass user name for regress tests (risingwavelabs#3854) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 9b0d3a1 Author: Noel Kwan <[email protected]> Date: Thu Jul 14 10:13:51 2022 +0800 feat(sqlsmith): generate group by (risingwavelabs#3805) * implement sqlsmith group by * fmt * add required cols * pass valid cols to gen_col * gen col from valid_cols if available * override bound columns for group by * capture local column context * fix case where no FROM * fmt * cleanup * clean Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 8231cf1 Author: congyi <[email protected]> Date: Thu Jul 14 00:23:16 2022 +0800 feat(row-based encoding): introduce row-based encoding by using value encoding (risingwavelabs#3835) * introduce row-based encoding 1.0 commit 1966c93 Author: xxchan <[email protected]> Date: Wed Jul 13 17:08:09 2022 +0200 chore(proto): remove unused FilterScanNode (risingwavelabs#3846) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 23388b9 Author: Wenzhuo Liu <[email protected]> Date: Wed Jul 13 22:48:40 2022 +0800 feat(batch): support selective aggregation (risingwavelabs#3683) * feat(batch): support selective aggregation for count * add e2e test for selective count * add group by test for selective count * remove option * add filter to approx count distinct * add general agg and fix approx count * add general distinct agg * format * add e2e test * add selective distinct agg e2e * avoid collect * fix count star * format * fix bug in optimizer Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 2a8a96d Author: xxchan <[email protected]> Date: Wed Jul 13 16:07:03 2022 +0200 style(stream): exhaustive patterm for build_executor (risingwavelabs#3848) commit ca49f6d Author: Li0k <[email protected]> Date: Wed Jul 13 20:55:55 2022 +0800 feat(storage): support deserialize_prefix_len for prefix_bloom_filter (risingwavelabs#3807) * feat(storage): support deserialize_prefix_len for prefix_bloom_filter * fix(storage): fix deserialize_prefix unit-test * chore(storage): add doc for encoding_data_size * fix(storage): fix encoding_data_size to return truely length in storage_encoding commit edc6660 Author: Dylan <[email protected]> Date: Wed Jul 13 20:38:23 2022 +0800 feat(frontend): support explain verbose (risingwavelabs#3798) * feat(frontend) support explain verbose * prefix each field name with table_name * do-apply-planner-test * minor improvement * fix batch two phase agg which should satisfy all agg-call contain no distinct and order by * fix stream two phase agg which should satisfy all agg-call contain no distinct and order by * move explain_verbose flag from session config to optimizer context * fmt * fix test case commit 4a5503e Author: Li0k <[email protected]> Date: Wed Jul 13 18:59:48 2022 +0800 fix(hash): fix hash_key_data_size (risingwavelabs#3843) * fix(hash): fix hash_key_data_size * fix(hash): support Key8 for Boolean commit d0027c6 Author: Bugen Zhao <[email protected]> Date: Wed Jul 13 17:53:04 2022 +0800 fix(test): datagen parallel e2e (risingwavelabs#3840) Signed-off-by: Bugen Zhao <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit aa09e44 Author: Huangjw <[email protected]> Date: Wed Jul 13 17:40:49 2022 +0800 chore(risedev): add external_labels and remote_write in prometeus config (risingwavelabs#3825) * add remote write in prometheus config * fix config * add env PROMETHEUS_REMOTE_WRITE * cargo fmt * cargo fmt * update risedev yml * fix risedev yml * fix risedev yml Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 6e45426 Author: Alex Chi <[email protected]> Date: Wed Jul 13 17:08:41 2022 +0800 chore(streaming): check epoch continuous (risingwavelabs#3839) * chore(streaming): check epoch continuous Signed-off-by: Alex Chi <[email protected]> * fix unit test Signed-off-by: Alex Chi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4fadf48 Author: Kaige Li <[email protected]> Date: Wed Jul 13 16:36:10 2022 +0800 feat(optimizer): distinct agg. (risingwavelabs#3717) * feat(optimizer): distinct agg. * add comments. * add comments and tests. * correct planner test. * refactor. * improvement. * small fix. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 0f8148d Author: wzzzzd <[email protected]> Date: Wed Jul 13 16:15:17 2022 +0800 feat(common): support generating `Datum` in field generator (risingwavelabs#3827) * add generate_datum to generator traits * add MIN and MAX to OrderedF32 and OrderedF64 * make OrderedFloat serializable and expose impl_rand in common configs * implement generate_datum * add unit tests and fix format * fix bug Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4b511e8 Author: William Wen <[email protected]> Date: Wed Jul 13 15:36:04 2022 +0800 feat(meta): support BTreeMapTransaction that avoids copying the whole BTreeMap (risingwavelabs#3680) * feat(meta): support BTreeMapTransaction that avoids copying the whole BTreeMap * add docs Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 369b57e Author: Bohan Zhang <[email protected]> Date: Wed Jul 13 14:42:22 2022 +0800 refactor: move source connectors to source dir (risingwavelabs#3826) move source connectors to source/ Signed-off-by: tabVersion <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 3e72311 Author: Alex Chi <[email protected]> Date: Wed Jul 13 14:19:09 2022 +0800 feat(cli): add prof and tokio env config (risingwavelabs#3824) Signed-off-by: Alex Chi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit b16c25a Author: Richard Chien <[email protected]> Date: Wed Jul 13 13:42:40 2022 +0800 refactor(batch,agg): merge `Aggregator::update*` and refactor `SortAggExecutor` (risingwavelabs#3794) * feed Aggregator at most one group at a time * impl new Aggregator functions for existing aggregators * impl `nth` for ArrayImplIterator to speed up `iter.skip(n)` * fix clippy warnings * add unittest for ArrayImplIterator::nth Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 3f03b2e Author: Li0k <[email protected]> Date: Wed Jul 13 12:39:16 2022 +0800 fix(type): fix decimal from_str and add e2e test (risingwavelabs#3823) * chore(e2e): add e2e test for decimal special values * fix(type): fix decimal from_str and add e2e test * fix(type): add unit-test for decimal from_str commit eb1d49e Author: Richard Chien <[email protected]> Date: Wed Jul 13 10:33:28 2022 +0800 feat(frontend): support order by clause of agg call (risingwavelabs#3720) * support binding agg order by clause * support planning agg call with order by clause * print order by clause of agg call * correct the generation of BatchProject * publicly expose AggOrderBy::sort_exprs * remove a useless log * fix debug format of PlanAggCall * revert changes to PlanAggCall::filter * add plan tests for order by clause * return error for some invalid syntax * use Option<OrderType> instead of Option<bool> * move ORDER BY clause in debug output * fix plan tests * use Direction instead of OrderType * remove unused derive * change default value of AggOrderByExpr::direction to Direction::Asc * add a todo comment * change nulls_first field to bool without Option * fix plan tests Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ff4c0d7 Author: Steven Chua <[email protected]> Date: Wed Jul 13 09:56:01 2022 +0800 feat(frontend): Add planning and option in frontend for Lookup Join (risingwavelabs#3763) * feat(frontend): Add plan node for batch lookup join * feat(frontend): Add config for using lookup join * feat(frontend): Change batch lookup join to unary node * feat(frontend): Add sources for lookup join node * feat(frontend): Fix bugs in Lookup Join * feat(frontend): Add error handling when data types of two sides of equality predicate are different * feat(frontend): Fix formatting and remove extraneous code * feat(frontend): fix bug * feat(frontend): Add e2e tests * feat(frontend): Fix e2e tests * feat(frontend): Better handling for when lookup join's conditions are not met * feat(frontend): Fix bug with comparing order key and predicate indices * feat(frontend): Refactored code and renamed config to rw_batch_enable_lookup_join * feat(frontend): Fix bug with output indices * feat(frontend): Fix test * feat(frontend): Fix another test * feat(frontend): Fix tests * feat(frontend): Change name and add more e2e tests * feat(frontend): Fix formatting * feat(frontend): Fix test bug Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 7a1331b Author: Li0k <[email protected]> Date: Tue Jul 12 21:19:00 2022 +0800 fix(utils): fix decimal flag bound (risingwavelabs#3808) commit 6141051 Author: William Wen <[email protected]> Date: Tue Jul 12 17:53:21 2022 +0800 feat(meta): add metrics in hummock manager lock and real process time (risingwavelabs#3802) * feat(meta): add metrics in hummock manager lock and real process time * fix timer use * add lock type in metrics label * add grafana panel Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 6dadbe0 Author: Noel Kwan <[email protected]> Date: Tue Jul 12 17:13:08 2022 +0800 feat(StorageTableIter): parameterize on decoding (risingwavelabs#3800) * refactor deserializer constructor * parameterize on deserializer * rerun ci Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 27a7ea8 Author: Noel Kwan <[email protected]> Date: Tue Jul 12 16:48:12 2022 +0800 feat(sqlsmith): Generate EquiJoin (risingwavelabs#3613) * generalize gen table * initial skeleton * pass columns * gen nested join factor * generate join_on expr * initialize join on binop * fmt * use inner join * clean * gen qualified column names * fmt * clean * clean * add planner test for join on and cross join * add expected logical plan * fix typo * disable equijoin * revert join.yaml Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ead67f7 Author: wzzzzd <[email protected]> Date: Tue Jul 12 16:36:24 2022 +0800 test(batch): add benchmark for sort merge join (risingwavelabs#3797) * add gen_sorted_data * modify the visibility of modules to allow benchmarking * remove data_type of gen_data and gen_sorted_data * add benchmark for sort merge join Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 7b544ad Author: William Wen <[email protected]> Date: Tue Jul 12 16:09:05 2022 +0800 refactor(storage): move hummock tests to a new crate to break crate cyclic dependency (risingwavelabs#3795) * refactor(storage): move hummock tests to a new crate to break crate cyclic dependency * remove meta dependency for failpoint test * Update src/storage/hummock_test/src/vacuum_tests.rs Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * add workspace-hack to the new craet Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 6170ff2 Author: congyi <[email protected]> Date: Tue Jul 12 15:51:20 2022 +0800 refactor(streaming): ensure correct storage table constructing for all streaming operators (risingwavelabs#3765) * refactor * refactor look_up * fix * fix lookup * fix lookup * remove new_for_test * fix distribution of arrangement Signed-off-by: Alex Chi <[email protected]> Co-authored-by: Alex Chi <[email protected]> commit b618974 Author: jon-chuang <[email protected]> Date: Tue Jul 12 15:32:31 2022 +0800 feat(executor): `DynamicFilter` write right table state exactly once - only if actor owns vnode = 0 (risingwavelabs#3783) * initial * stash * do-appply-planner-test * fmt * clippy * license * minor * stash * fix required dist, output indices * fmt * stash * revert * do-apply-planner-test * success! * more e2e * minor cleanup * stash * fix * fix * fix * minor * minor * madsim * clippy * fmt * minor * fix * fix, apply code review suggestions * fix * fix, improve * simplify * clippy * refactor into sub functions * clippy * minor * fix * fix: rowsort * fmt * use literal in expression * clippy * fmt * use madsim btreemap * minor * fmt * respect previous behaviour - return empty message * bump-ci * use correct enum * add state table * fmt * prepare for cache policy * fmt * minor * minor * minor * Update src/stream/src/executor/managed_state/dynamic_filter.rs add license Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fmt * add state table for RHS * minor * fmt * add right_table to executor * minor * fmt * fix * clippy * fmt * fix double write * use correct * try to write on every msg * fix weird omissions * bump * minor * revert q18, q20 * revert q4 * remove inserts * only insert for right table * try to insert at barrier only * add asserts * debug * fix * fmt * only write on vnode = 0 * minor * minor * use bool as we need to read from RHS in the future... (for recovery)... Co-authored-by: TennyZhuang <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> commit fd0b7e2 Author: Bohan Zhang <[email protected]> Date: Tue Jul 12 14:26:12 2022 +0800 feat: kafka json append only sink (risingwavelabs#3682) * stage Signed-off-by: tabVersion <[email protected]> * add datum_to_json_object Signed-off-by: Peng Chen <[email protected]> * stage Signed-off-by: tabVersion <[email protected]> * finishing Signed-off-by: tabVersion <[email protected]> * make risedev happy Signed-off-by: tabVersion <[email protected]> * add ut Signed-off-by: tabVersion <[email protected]> * pending error: Transaction error: Operation not valid in state Ready Signed-off-by: tabVersion <[email protected]> * fix by changing epoch number Signed-off-by: tabVersion <[email protected]> * add doc for sink trait Signed-off-by: tabVersion <[email protected]> * add comment in exec Signed-off-by: tabVersion <[email protected]> * remove take_snapshot Signed-off-by: tabVersion <[email protected]> * fix warning Signed-off-by: tabVersion <[email protected]> * fix table_id Signed-off-by: tabVersion <[email protected]> Co-authored-by: Peng Chen <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 0b65e68 Author: Alex Chi <[email protected]> Date: Tue Jul 12 14:00:53 2022 +0800 fix(dashboardv1): adapt new dispatcher design and add dynamic filter node (risingwavelabs#3793) Signed-off-by: Alex Chi <[email protected]> commit 17bdf88 Author: TennyZhuang <[email protected]> Date: Tue Jul 12 10:21:58 2022 +0800 build(deps): use official pulsar-rs instead of fork (risingwavelabs#3787) Signed-off-by: TennyZhuang <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 8bd5f63 Author: TennyZhuang <[email protected]> Date: Tue Jul 12 01:33:53 2022 +0800 fix(connector): support parsing non-string nexmark properties (risingwavelabs#3784) * fix(connector): support parse non-string nexmark properties Signed-off-by: TennyZhuang <[email protected]> * still use serde_json currently Signed-off-by: TennyZhuang <[email protected]> * rollback to Box Signed-off-by: TennyZhuang <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit f93d8e3 Author: xiangjinwu <[email protected]> Date: Tue Jul 12 01:07:05 2022 +0800 refactor: remove stale AggKind::RowCount (risingwavelabs#3774) * refactor: remove stale AggKind::RowCount * update doc Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 2ee98b5 Author: StrikeW <[email protected]> Date: Mon Jul 11 23:53:31 2022 +0800 fix(batch): fix order_by with struct table (risingwavelabs#3782) * fix(batch): order_by with struct table * add more case Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit b7a969c Author: jon-chuang <[email protected]> Date: Mon Jul 11 22:05:53 2022 +0800 feat(executor): Use `TableCatalog` and `StateTable` in `DynamicFilter` (only writes) (risingwavelabs#3757) * initial * stash * do-appply-planner-test * fmt * clippy * license * minor * stash * fix required dist, output indices * fmt * stash * revert * do-apply-planner-test * success! * more e2e * minor cleanup * stash * fix * fix * fix * minor * minor * madsim * clippy * fmt * minor * fix * fix, apply code review suggestions * fix * fix, improve * simplify * clippy * refactor into sub functions * clippy * minor * fix * fix: rowsort * fmt * use literal in expression * clippy * fmt * use madsim btreemap * minor * fmt * respect previous behaviour - return empty message * bump-ci * use correct enum * add state table * fmt * prepare for cache policy * fmt * minor * minor * minor * Update src/stream/src/executor/managed_state/dynamic_filter.rs add license Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fmt * add state table for RHS * minor * fmt * add right_table to executor * minor * fmt * fix * clippy * fmt * fix double write * use correct * try to write on every msg * fix weird omissions Co-authored-by: TennyZhuang <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> commit 5430e23 Author: Li0k <[email protected]> Date: Mon Jul 11 20:13:30 2022 +0800 chore(storage): remove value meta (risingwavelabs#3779) commit 0608f0a Author: StrikeW <[email protected]> Date: Mon Jul 11 17:36:36 2022 +0800 refactor(stream): refactor state of topN executor to one state table (risingwavelabs#3747) commit 097e3eb Author: Bugen Zhao <[email protected]> Date: Mon Jul 11 17:16:17 2022 +0800 feat(streaming): use multiple dispatcher for mview creation (risingwavelabs#3758) * move mutation and barrier to stream_plan.proto Signed-off-by: Bugen Zhao <[email protected]> * add dispatcher when creating mview Signed-off-by: Bugen Zhao <[email protected]> * from to proto of add dispatcher Signed-off-by: Bugen Zhao <[email protected]> * put actor info to shared context Signed-off-by: Bugen Zhao <[email protected]> * add dispatcher on compute node Signed-off-by: Bugen Zhao <[email protected]> * clean up empty dispatcher Signed-off-by: Bugen Zhao <[email protected]> * cleanup Signed-off-by: Bugen Zhao <[email protected]> * fix clippy Signed-off-by: Bugen Zhao <[email protected]> * fix tests Signed-off-by: Bugen Zhao <[email protected]> * use add dispatcher for source executor Signed-off-by: Bugen Zhao <[email protected]> * rename dispatch to dispatcher Signed-off-by: Bugen Zhao <[email protected]> * add comments Signed-off-by: Bugen Zhao <[email protected]> * fix unique dispatcher id Signed-off-by: Bugen Zhao <[email protected]> * fix it Signed-off-by: Bugen Zhao <[email protected]> * use expect dead code Signed-off-by: Bugen Zhao <[email protected]> commit 59579ce Author: wzzzzd <[email protected]> Date: Mon Jul 11 16:58:54 2022 +0800 fix(batch): add first-match optimization for left semi join (risingwavelabs#3773) add first-match optimization for left semi join commit 7d21848 Author: wzzzzd <[email protected]> Date: Mon Jul 11 16:09:01 2022 +0800 feat(common): add is_set to BitmapBuilder (risingwavelabs#3768) add is_set to BitmapBuilder commit 78e5b66 Author: wzzzzd <[email protected]> Date: Mon Jul 11 15:33:31 2022 +0800 refactor(batch): say goodbye to the state machine in NLJ (risingwavelabs#3724) * refactor(batch): say goodbye to the state machine * add some utilities and move right table to the outer loop * refactor * modify the unit test for left outer join to adapt to the semantical change * fix utility usage * remove unused imports * add some comments * more comments * fix format commit 7715827 Author: Zhanxiang (Patrick) Huang <[email protected]> Date: Mon Jul 11 12:51:00 2022 +0800 refactor: remove vnode_mappings in CompactTask and GroupedSstableBuilder (risingwavelabs#3764) * refactor: remove vnode_mappings in CompactTask * remove GroupedSstableBuilder Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit a8d0c57 Author: Xinhao Xu <[email protected]> Date: Mon Jul 11 12:10:40 2022 +0800 feat(common): move observer_manager to common service (risingwavelabs#3754) * mv observer_manager to common_service * mv observer_manager to common_service * rename commit 33c3611 Author: zwang28 <[email protected]> Date: Mon Jul 11 10:50:59 2022 +0800 fix: sync compaction group correctly (risingwavelabs#3743) commit 00422c6 Author: Liang <[email protected]> Date: Mon Jul 11 09:57:42 2022 +0800 perf(hummmock manager): remove unnecessary hummock version deltas (risingwavelabs#3689) * perf(hummmock manager): remove unnecessary hummock version deltas * delete multiple version deltas in one trx * do not use vartrx for hummock version * do not use vartrx for hummockversion in commit_epoch * remove checkpoint_version_id Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 208ab51 Author: Alex Chi <[email protected]> Date: Sun Jul 10 16:38:39 2022 +0800 fix(risedev): unexpected rebuilds on Linux (risingwavelabs#3761) * fix(risedev): unexpected rebuilds on Linux Signed-off-by: Alex Chi <[email protected]> * update workspace hack Signed-off-by: Alex Chi <[email protected]> * use bash instead of fish Co-authored-by: Bugen Zhao <[email protected]> commit 301499f Author: Lee Zong Yu <[email protected]> Date: Sat Jul 9 14:51:38 2022 +0800 feat(metrics): Add Fragment Level Exchange Metrics (risingwavelabs#3696) * Edit update.sh * Edit format error * Add payload to gitignore * Revert risedev.yml modification * Fix wrong gitignore * Fix wrong gitignore * Done coding for Fragment level exchange * Done coding for Fragment level exchange * Edit implementation error * Edit down fragment error * Delete println statement and revert configuration settings * Done coding for Fragment level exchange * Edit update.sh * Edit format error * Add payload to gitignore * Revert risedev.yml modification * Fix wrong gitignore * Done coding for Fragment level exchange * Edit implementation error * Edit down fragment error * Delete println statement and revert configuration settings * Edit Format and Resolve conflict * Resolve Test error * Resolve some error * generate dashboard * use rate_interval instead of 15s Signed-off-by: Alex Chi <[email protected]> Co-authored-by: Alex Chi <[email protected]> commit 92ee6d9 Author: Bugen Zhao <[email protected]> Date: Fri Jul 8 23:25:12 2022 +0800 refactor(meta): improve create mview / scheduler readability (risingwavelabs#3748) * refactor(meta): improve create mview / scheduler readability Signed-off-by: Bugen Zhao <[email protected]> * add some comments for hash mapping Signed-off-by: Bugen Zhao <[email protected]> * refine docs Signed-off-by: Bugen Zhao <[email protected]> * make yuanxin happy Signed-off-by: Bugen Zhao <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 8fbe3f9 Author: congyi <[email protected]> Date: Fri Jul 8 19:03:34 2022 +0800 refactor(streaming): use table catalog for materialize executor (risingwavelabs#3742) * use catalog for materialize executor commit 23f41ab Author: August <[email protected]> Date: Fri Jul 8 18:31:01 2022 +0800 feat: deprecate catalog ref id (risingwavelabs#3745) feat: deprecate database/schema/table_ref_id Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit fa9e66c Author: Dylan <[email protected]> Date: Fri Jul 8 16:39:44 2022 +0800 feat(optimizer): support two phase topn and limit in batch mode (risingwavelabs#3719) * feat(optimizer): support two phase topn and limit in batch mode * fix limit distribution in local batch mode * add test Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4b50016 Author: xxchan <[email protected]> Date: Fri Jul 8 10:23:47 2022 +0200 fix(batch): fix batch scan distribution & handle more than one table scans in one stage (risingwavelabs#3736) * fix batch scan distribution * handle more than one table scans in one stage Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 17c0417 Author: Yuhao Su <[email protected]> Date: Fri Jul 8 14:25:09 2022 +0800 refactor(streaming): use table catalog in hash join (risingwavelabs#3707) * use tablecatalog * fix infer_internal_table_catalog * cargo fix * fix tests * keys -> key * fix build executor * minor * fix incorrect order keys * fix comments * fix typo * add comment commit 2dc7992 Author: TennyZhuang <[email protected]> Date: Fri Jul 8 14:12:58 2022 +0800 test(nexmark): add nexmark q10 q14 e2e test (risingwavelabs#3740) * test(nexmark): add nexmark q10 e2e test Signed-off-by: TennyZhuang <[email protected]> * test(nexmark): add nexmark q14 e2e test Signed-off-by: TennyZhuang <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit d21b2b9 Author: congyi <[email protected]> Date: Fri Jul 8 13:58:20 2022 +0800 chore(risedev): update grafana version (risingwavelabs#3731) * update grafana version * enable auto update * "retry" commit aea1065 Author: Alex Chi <[email protected]> Date: Fri Jul 8 13:43:59 2022 +0800 fix(ci): use header fix action for license check (risingwavelabs#3739) commit 7a86c4d Author: Tao Wu <[email protected]> Date: Fri Jul 8 10:30:39 2022 +0800 fix(sqlsmith): completely cover all exprs (risingwavelabs#3737) commit a777217 Author: congyi <[email protected]> Date: Thu Jul 7 22:44:39 2022 +0800 refactor(storage): remove cell_based serialize/deserialize from serde (risingwavelabs#3722) * remove cell_based serialize/deserialize from serde commit 8d97480 Author: Tao Wu <[email protected]> Date: Thu Jul 7 22:23:34 2022 +0800 fix: support decimal mul interval (risingwavelabs#3734) commit 9b64fca Author: Steven Chua <[email protected]> Date: Thu Jul 7 21:12:14 2022 +0800 fix(risedev): Revert Makefile.toml changes (risingwavelabs#3732) Revert Makefile.toml changes Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 19cffce Author: Richard Chien <[email protected]> Date: Thu Jul 7 20:57:15 2022 +0800 refactor(sqlparser): rename `parse_string_agg` to `parse_string_concat` (risingwavelabs#3733) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit d767d99 Author: jon-chuang <[email protected]> Date: Thu Jul 7 20:33:06 2022 +0800 feat(executor): Basic in-application-memory `DynamicFilterExecutor`, pass tpch q11, q22 e2e tests (risingwavelabs#3681) * initial * stash * do-appply-planner-test * fmt * clippy * license * minor * stash * fix required dist, output indices * fmt * stash * revert * do-apply-planner-test * success! * more e2e * minor cleanup * stash * fix * fix * fix * minor * minor * madsim * clippy * fmt * minor * fix * fix, apply code review suggestions * fix * fix, improve * simplify * clippy * refactor into sub functions * clippy * minor * fix * fix: rowsort * fmt * use literal in expression * clippy * fmt * use madsim btreemap * minor * fmt * respect previous behaviour - return empty message * bump-ci Co-authored-by: TennyZhuang <[email protected]> commit 07f282b Author: Alex Chi <[email protected]> Date: Thu Jul 7 19:53:06 2022 +0800 chore(deps): bump dependencies (risingwavelabs#3725) Signed-off-by: Alex Chi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 95586c8 Author: Yuhao Su <[email protected]> Date: Thu Jul 7 19:02:54 2022 +0800 test(streaming): enable streaming e2e test for tpch q4, q18, q20 (risingwavelabs#3718) enable e2e commit ca0f431 Author: Alex Chi <[email protected]> Date: Thu Jul 7 18:11:07 2022 +0800 fix(metrics): use current actor id for backpressure (risingwavelabs#3721) * fix(metrics): use current actor id for backpressure Signed-off-by: Alex Chi <[email protected]> * fix Signed-off-by: Alex Chi <[email protected]> commit 477a4be Author: August <[email protected]> Date: Thu Jul 7 17:15:27 2022 +0800 fix: fix heartbeat req field and meta client impl (risingwavelabs#3716) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit f9362b1 Author: Bowen <[email protected]> Date: Thu Jul 7 17:03:13 2022 +0800 doc: add relational table schema doc + move files (risingwavelabs#3712) * doc: add relational table schema doc + move files * fix comment * Update docs/relational_table/relational-table-schema.md Co-authored-by: congyi <[email protected]> Co-authored-by: congyi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 4a92eb6 Author: TennyZhuang <[email protected]> Date: Thu Jul 7 16:50:54 2022 +0800 test(nexmark): explain q16 q17 (risingwavelabs#3715) Signed-off-by: TennyZhuang <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit e1fd06b Author: xxchan <[email protected]> Date: Thu Jul 7 10:30:57 2022 +0200 feat(batch): prune scan partition according to scan_range (risingwavelabs#3698) * prune vnode according to scan_range * disallow accessing other vnodes from batch scan * improve style * add ut * fix doc about dist key * fix Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit ed4cb79 Author: Alex Chi <[email protected]> Date: Thu Jul 7 16:11:15 2022 +0800 feat(storage): default block size = 1024 (risingwavelabs#3463) Signed-off-by: Alex Chi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 3bb55cf Author: zwang28 <[email protected]> Date: Thu Jul 7 15:55:55 2022 +0800 fix: correctly implement check_state_consistency (risingwavelabs#3711) commit 078c2f6 Author: xiangjinwu <[email protected]> Date: Thu Jul 7 15:35:38 2022 +0800 refactor(optimizer): cleanup LogicalAgg::prune_col (risingwavelabs#3663) * refactor(optimizer): cleanup LogicalAgg::prune_col * planner test of prune-filter bug Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 82ef7ec Author: Tao Wu <[email protected]> Date: Thu Jul 7 14:54:18 2022 +0800 fix: reduce logging in the critical path (risingwavelabs#3704) * fix: reduce logging in the critical path * fix cargo check Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> commit 79eb45b Author: William Wen <[email protected]> Date: Thu Jul 7 14:42:14 2022 +0800 feat(meta): add monitoring meta process metrics (risingwavelabs#3706) * feat(meta): add monitoring meta process metrics * update procfs dependency * add libc dependency Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Signed-off-by: Little-Wallace <[email protected]> * fix format Signed-off-by: Little-Wallace <[email protected]> * fix large compaction Signed-off-by: Little-Wallace <[email protected]> * merge e2e_test from main Signed-off-by: Little-Wallace <[email protected]> * simple compact Signed-off-by: Little-Wallace <[email protected]> * refactor splits Signed-off-by: Little-Wallace <[email protected]> * fix splits Signed-off-by: Little-Wallace <[email protected]> * fix bug Signed-off-by: Little-Wallace <[email protected]> * fix unit test Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * tmp commit Signed-off-by: Little-Wallace <[email protected]> * fix conflict Signed-off-by: Little-Wallace <[email protected]> * fix test bug Signed-off-by: Little-Wallace <[email protected]> * fix version Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix bug Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix clippy warn Signed-off-by: Little-Wallace <[email protected]> * fix conflict Signed-off-by: Little-Wallace <[email protected]> * fix format Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * split tier compaction file Signed-off-by: Little-Wallace <[email protected]> * fix license Signed-off-by: Little-Wallace <[email protected]> * fix trivial move Signed-off-by: Little-Wallace <[email protected]> * fix compact Signed-off-by: Little-Wallace <[email protected]> * fix conflict Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix compactor Signed-off-by: Little-Wallace <[email protected]> * fix commit Signed-off-by: Little-Wallace <[email protected]> * fix conflict Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao [email protected]
I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.
What's changed and what's your intention?
As title. This PR makes
create_materialized_view
andScheduler
more readable by adding some comments and extracting common snippets.Refer to a related PR or issue link (optional)