panic when executing a flow job

[killme2008]

What type of bug is this?

Crash

What subsystems are affected?

Datanode

Minimal reproduce step

1. Create the input table:

CREATE TABLE numbers_input (
    number INT,
    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY(number),
    TIME INDEX(ts)
);

2. Create the output table:

create table out_num_cnt (
    number INT, 
    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP TIME INDEX);

3. Create the flow to filter numbers which are greater than 10:

CREATE FLOW filter_numbers  SINK TO out_num_cnt AS  SELECT number FROM numbers_input where number > 10;

4. Insert some data, works well:

mysql> insert into numbers_input values(1, now());
Query OK, 1 row affected (0.03 sec)

mysql> insert into numbers_input values(11, now());
Query OK, 1 row affected (0.01 sec)

mysql> select * from out_num_cnt;
+--------+----------------------------+
| number | ts                         |
+--------+----------------------------+
|     11 | 2024-05-21 15:48:46.813000 |
+--------+----------------------------+
1 row in set (0.02 sec)

5. Waits about 10 seconds or more, insert more data but it doesn't work, and the flow job panic:

thread '<unnamed>' panicked at src/datatypes/src/value.rs:558:9:
Cannot compare different values Timestamp(Timestamp { value: 1716306563537, unit: Millisecond }) and Int32(10)
stack backtrace:
   0:        0x10bf6ced8 - std::backtrace_rs::backtrace::libunwind::trace::h8897f742b605706d
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/../../backtrace/src/backtrace/libunwind.rs:105:5
   1:        0x10bf6ced8 - std::backtrace_rs::backtrace::trace_unsynchronized::h7fb2dfc9f56f5c5f
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x10bf6ced8 - std::sys_common::backtrace::_print_fmt::h81954a34720f6d0d
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys_common/backtrace.rs:68:5
   3:        0x10bf6ced8 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::had101fed8b9c5a73
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x10bf91d9c - core::fmt::rt::Argument::fmt::h0abfdce623bca0ef
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/fmt/rt.rs:165:63
   5:        0x10bf91d9c - core::fmt::write::h1021c4d383a98539
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/fmt/mod.rs:1157:21
   6:        0x10bf68b14 - std::io::Write::write_fmt::h504af0366c253c0d
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/io/mod.rs:1832:15
   7:        0x10bf6cd30 - std::sys_common::backtrace::_print::h57de875fe64bfb6c
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys_common/backtrace.rs:47:5
   8:        0x10bf6cd30 - std::sys_common::backtrace::print::h3ec8eda36096518c
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys_common/backtrace.rs:34:9
   9:        0x10bf6e18c - std::panicking::default_hook::{{closure}}::haaa2fe7c524b5a01
  10:        0x10bf6de80 - std::panicking::default_hook::h42c3bca2cc8688e0
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:298:9
  11:        0x10ade2558 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h12d2cbd959814616
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/alloc/src/boxed.rs:2036:9
  12:        0x10adddff0 - common_telemetry::panic_hook::set_panic_hook::{{closure}}::h2444d3384de1ced0
                               at /Users/dennis/programming/rust/greptimedb/src/common/telemetry/src/panic_hook.rs:51:9
  13:        0x10bf6ea6c - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h80203d6bb2ad7a77
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/alloc/src/boxed.rs:2036:9
  14:        0x10bf6ea6c - std::panicking::rust_panic_with_hook::ha7897fb99da481b9
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:799:13
  15:        0x10bf6e474 - std::panicking::begin_panic_handler::{{closure}}::h39b5ce0a48d94b7c
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:664:13
  16:        0x10bf6d360 - std::sys_common::backtrace::__rust_end_short_backtrace::hbd9195b51dc23b49
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys_common/backtrace.rs:171:18
  17:        0x10bf6e1e4 - rust_begin_unwind
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:652:5
  18:        0x10c175160 - core::panicking::panic_fmt::h57724f3c01588279
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/panicking.rs:72:14
  19:        0x107d25204 - <datatypes::value::Value as core::cmp::Ord>::cmp::h61f5f0b42cf33419
                               at /Users/dennis/programming/rust/greptimedb/src/datatypes/src/value.rs:558:9
  20:        0x107d24d34 - <datatypes::value::Value as core::cmp::PartialOrd>::partial_cmp::h4648a3cbf599cd49
                               at /Users/dennis/programming/rust/greptimedb/src/datatypes/src/value.rs:552:14
  21:        0x102a28608 - core::cmp::PartialOrd::gt::h36b27407c0a3fe57
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/cmp.rs:1215:18
  22:        0x102988004 - flow::expr::func::BinaryFunc::eval::h56493f41d560d56f
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/expr/func.rs:655:40
  23:        0x102974cbc - flow::expr::scalar::ScalarExpr::eval::h7fc68c4ff754f98f
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/expr/scalar.rs:244:62
  24:        0x10299abe8 - flow::expr::linear::SafeMfpPlan::evaluate_inner::ha7b20fc4751692e3
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/expr/linear.rs:517:16
  25:        0x102a6b2ac - flow::expr::linear::MfpPlan::evaluate::he9357ba47af52876
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/expr/linear.rs:608:15
  26:        0x102a9d6b0 - flow::compute::render::map::eval_mfp_core::ha629551cb7a83d1d
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/compute/render/map.rs:157:23
  27:        0x102a9d22c - flow::compute::render::map::mfp_subgraph::{{closure}}::he7f18ffebb383f6e
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/compute/render/map.rs:117:27
  28:        0x102a7a960 - flow::compute::types::ErrCollector::run::h73d0f230edc2c238
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/compute/types.rs:175:15
  29:        0x102a9cf78 - flow::compute::render::map::mfp_subgraph::hfa03ab5670838dc2
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/compute/render/map.rs:121:5
  30:        0x102a9ce84 - flow::compute::render::map::<impl flow::compute::render::Context>::render_mfp::{{closure}}::h2097f2bf02cd2749
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/compute/render/map.rs:79:17
  31:        0x102a42484 - <hydroflow::scheduled::graph::Hydroflow as hydroflow::scheduled::graph_ext::GraphExt>::add_subgraph_in_out::{{closure}}::h8e60e5b3506b1a3b
                               at /Users/dennis/.cargo/git/checkouts/hydroflow-070e3c8ef51b7002/b072ee0/hydroflow/src/scheduled/graph_ext.rs:64:21
  32:        0x102998c78 - hydroflow::scheduled::graph::Hydroflow::add_subgraph_stratified::{{closure}}::h82aed3dd2cb4980c
                               at /Users/dennis/.cargo/git/checkouts/hydroflow-070e3c8ef51b7002/b072ee0/hydroflow/src/scheduled/graph.rs:630:13
  33:        0x10299621c - <F as hydroflow::scheduled::subgraph::Subgraph>::run::h7af6aecd3c506a1b
                               at /Users/dennis/.cargo/git/checkouts/hydroflow-070e3c8ef51b7002/b072ee0/hydroflow/src/scheduled/subgraph.rs:14:9
  34:        0x102aaaa5c - hydroflow::scheduled::graph::Hydroflow::run_stratum::{{closure}}::hb808cb171e40e097
                               at /Users/dennis/.cargo/git/checkouts/hydroflow-070e3c8ef51b7002/b072ee0/hydroflow/src/scheduled/graph.rs:302:17
  35:        0x102aa924c - hydroflow::scheduled::graph::Hydroflow::run_stratum::hb8199fe4a3397161
                               at /Users/dennis/.cargo/git/checkouts/hydroflow-070e3c8ef51b7002/b072ee0/hydroflow/src/scheduled/graph.rs:282:5
  36:        0x102aa8c4c - hydroflow::scheduled::graph::Hydroflow::run_available::{{closure}}::hc1c8de8798145e7d
                               at /Users/dennis/.cargo/git/checkouts/hydroflow-070e3c8ef51b7002/b072ee0/hydroflow/src/scheduled/graph.rs:254:13
  37:        0x102aa7ca8 - hydroflow::scheduled::graph::Hydroflow::run_available::h07b39930ef3a8d95
                               at /Users/dennis/.cargo/git/checkouts/hydroflow-070e3c8ef51b7002/b072ee0/hydroflow/src/scheduled/graph.rs:247:5
  38:        0x1029f9cdc - flow::compute::state::DataflowState::run_available_with_schedule::h77ca8b236e18717d
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/compute/state.rs:77:9
  39:        0x102a4949c - flow::adapter::worker::ActiveDataflowState::run_available::h0f04601f53376563
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/adapter/worker.rs:103:9
  40:        0x102a4b5fc - flow::adapter::worker::Worker::run_tick::hc7aff38b43112a0f
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/adapter/worker.rs:307:13
  41:        0x102a4b7fc - flow::adapter::worker::Worker::handle_req::hb13d9c1d4299386a
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/adapter/worker.rs:349:17
  42:        0x102a4a148 - flow::adapter::worker::Worker::run::he96622980bb91c95
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/adapter/worker.rs:283:23
  43:        0x10249ed9c - flow::adapter::FlownodeBuilder::build::{{closure}}::{{closure}}::h133c281f843bb692
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/adapter.rs:134:13
  44:        0x10243c3f4 - std::sys_common::backtrace::__rust_begin_short_backtrace::h4890d63206c3654d
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys_common/backtrace.rs:155:18
  45:        0x1022fc1f4 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hab3b199b1d1936b0
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/thread/mod.rs:542:17
  46:        0x10239439c - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h2b6585e8d503b09c
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/panic/unwind_safe.rs:272:9
  47:        0x1023cb840 - std::panicking::try::do_call::hc7e81191229c826e
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:559:40
  48:        0x1023d2d2c - ___rust_try
  49:        0x1023c5f94 - std::panicking::try::hafec648ace53b312
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:523:19
  50:        0x1022fc024 - std::panic::catch_unwind::hb8e021e5228b1777
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panic.rs:149:14
  51:        0x1022fc024 - std::thread::Builder::spawn_unchecked_::{{closure}}::h181ecb0c48bf4a31
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/thread/mod.rs:541:30
  52:        0x1024c2a2c - core::ops::function::FnOnce::call_once{{vtable.shim}}::hec7a8280f0e72e41
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/ops/function.rs:250:5
  53:        0x10bf74c58 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hae82ec5a7f723514
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/alloc/src/boxed.rs:2022:9
  54:        0x10bf74c58 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h662e6d57208b72b3
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/alloc/src/boxed.rs:2022:9
  55:        0x10bf74c58 - std::sys::pal::unix::thread::Thread::new::thread_start::h4553383436d5a44f
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys/pal/unix/thread.rs:108:17
  56:        0x19ed9af94 - __pthread_joiner_wake
thread 'bg-worker' panicked at src/flow/src/adapter.rs:513:62:
called `Result::unwrap()` on an `Err` value: 0: Internal error, at src/flow/src/adapter/worker.rs:487:6
stack backtrace:
   0:        0x10bf6ced8 - std::backtrace_rs::backtrace::libunwind::trace::h8897f742b605706d
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/../../backtrace/src/backtrace/libunwind.rs:105:5
   1:        0x10bf6ced8 - std::backtrace_rs::backtrace::trace_unsynchronized::h7fb2dfc9f56f5c5f
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x10bf6ced8 - std::sys_common::backtrace::_print_fmt::h81954a34720f6d0d
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys_common/backtrace.rs:68:5
   3:        0x10bf6ced8 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::had101fed8b9c5a73
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x10bf91d9c - core::fmt::rt::Argument::fmt::h0abfdce623bca0ef
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/fmt/rt.rs:165:63
   5:        0x10bf91d9c - core::fmt::write::h1021c4d383a98539
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/fmt/mod.rs:1157:21
   6:        0x10bf68b14 - std::io::Write::write_fmt::h504af0366c253c0d
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/io/mod.rs:1832:15
   7:        0x10bf6cd30 - std::sys_common::backtrace::_print::h57de875fe64bfb6c
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys_common/backtrace.rs:47:5
   8:        0x10bf6cd30 - std::sys_common::backtrace::print::h3ec8eda36096518c
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys_common/backtrace.rs:34:9
   9:        0x10bf6e18c - std::panicking::default_hook::{{closure}}::haaa2fe7c524b5a01
  10:        0x10bf6de80 - std::panicking::default_hook::h42c3bca2cc8688e0
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:298:9
  11:        0x10ade2558 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h12d2cbd959814616
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/alloc/src/boxed.rs:2036:9
  12:        0x10adddff0 - common_telemetry::panic_hook::set_panic_hook::{{closure}}::h2444d3384de1ced0
                               at /Users/dennis/programming/rust/greptimedb/src/common/telemetry/src/panic_hook.rs:51:9
  13:        0x10bf6ea6c - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h80203d6bb2ad7a77
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/alloc/src/boxed.rs:2036:9
  14:        0x10bf6ea6c - std::panicking::rust_panic_with_hook::ha7897fb99da481b9
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:799:13
  15:        0x10bf6e474 - std::panicking::begin_panic_handler::{{closure}}::h39b5ce0a48d94b7c
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:664:13
  16:        0x10bf6d360 - std::sys_common::backtrace::__rust_end_short_backtrace::hbd9195b51dc23b49
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys_common/backtrace.rs:171:18
  17:        0x10bf6e1e4 - rust_begin_unwind
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:652:5
  18:        0x10c175160 - core::panicking::panic_fmt::h57724f3c01588279
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/panicking.rs:72:14
  19:        0x10c175560 - core::result::unwrap_failed::h2966636c0504d6c6
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/result.rs:1654:5
  20:        0x102919134 - core::result::Result<T,E>::unwrap::hd13250c8cfbb6bac
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/result.rs:1077:23
  21:        0x102919134 - flow::adapter::FlownodeManager::run_available::{{closure}}::hd410032c3976f776
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/adapter.rs:513:17
  22:        0x102917ddc - flow::adapter::FlownodeManager::run::{{closure}}::hb2c9f6f5909a9836
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/adapter.rs:494:34
  23:        0x1029163dc - flow::adapter::FlownodeManager::run_background::{{closure}}::h6d2abf9ca1afa8ea
                               at /Users/dennis/programming/rust/greptimedb/src/flow/src/adapter.rs:469:24
  24:        0x102a45c64 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h13fbe21ee0102a8b
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  25:        0x102a455b4 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h404a63e0a5aa13b0
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  26:        0x102a455b4 - tokio::runtime::task::core::Core<T,S>::poll::h070441a588d73a0a
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  27:        0x102a6e870 - tokio::runtime::task::harness::poll_future::{{closure}}::he9910ab2b88eff0a
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  28:        0x10297aa08 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h8c23699517071dc1
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/panic/unwind_safe.rs:272:9
  29:        0x102883bac - std::panicking::try::do_call::h9901174999c5bf92
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:559:40
  30:        0x102884f1c - ___rust_try
  31:        0x102882be8 - std::panicking::try::h7a9270a553c474ed
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:523:19
  32:        0x10299d16c - std::panic::catch_unwind::h77813504c0e1c786
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panic.rs:149:14
  33:        0x102a6db4c - tokio::runtime::task::harness::poll_future::h3e6b4a21ecf13353
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  34:        0x102a6ed3c - tokio::runtime::task::harness::Harness<T,S>::poll_inner::ha86905bef588a15d
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  35:        0x102a700b4 - tokio::runtime::task::harness::Harness<T,S>::poll::h8582f30368011145
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  36:        0x102aa4094 - tokio::runtime::task::raw::poll::h4f841a1afa27c7dd
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
  37:        0x10bc0cc88 - tokio::runtime::task::raw::RawTask::poll::h4c036476cc49bef7
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  38:        0x10bc5998c - tokio::runtime::task::LocalNotified<S>::run::h1022aa56c3b79b9a
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
  39:        0x10bc17f3c - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::ha7786c4ad9d92427
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:576:13
  40:        0x10bc17e20 - tokio::runtime::coop::with_budget::h53f90a72f57e73e3
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
  41:        0x10bc17e20 - tokio::runtime::coop::budget::hfbd0cff6e839987d
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
  42:        0x10bc17e20 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::he3f924921b094fa3
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
  43:        0x10bc1775c - tokio::runtime::scheduler::multi_thread::worker::Context::run::ha4a036854ae0da2a
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
  44:        0x10bc17478 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::h5e7ccfd0394bb62b
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  45:        0x10bc58c54 - tokio::runtime::context::scoped::Scoped<T>::set::h5feff1e50f07f8b4
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
  46:        0x10bc8bf84 - tokio::runtime::context::set_scheduler::{{closure}}::h816ffb85ae6f249a
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26
  47:        0x10bc87d90 - std::thread::local::LocalKey<T>::try_with::h1d5a81ad9f07b58a
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/thread/local.rs:286:12
  48:        0x10bc87a30 - std::thread::local::LocalKey<T>::with::h1bf5f48f9ce34b98
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/thread/local.rs:262:9
  49:        0x10bc8bf50 - tokio::runtime::context::set_scheduler::h9222ef15e97dae90
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:9
  50:        0x10bc1739c - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::hf456d209a8cf586b
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  51:        0x10bc39718 - tokio::runtime::context::runtime::enter_runtime::h7dae1ea4eb42f79c
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
  52:        0x10bc171d4 - tokio::runtime::scheduler::multi_thread::worker::run::h6c2310f74b1bbb76
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  53:        0x10bc1706c - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h829a19df08801833
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  54:        0x10bc1feb0 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::hd300e9220d9da6bd
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
  55:        0x10bc424ec - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h65f6859c4c2556b3
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  56:        0x10bc41980 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hdedecedb3bb574f3
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  57:        0x10bc41980 - tokio::runtime::task::core::Core<T,S>::poll::hc969bfe50383a11b
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  58:        0x10bc040d8 - tokio::runtime::task::harness::poll_future::{{closure}}::hb21f642517bd6c16
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  59:        0x10bc0f6e0 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h89db3b97259ae092
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/panic/unwind_safe.rs:272:9
  60:        0x10bbf66a0 - std::panicking::try::do_call::h5e6b7afc626e5229
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:559:40
  61:        0x10bbf9d0c - ___rust_try
  62:        0x10bbf5870 - std::panicking::try::hefda585590f33d1a
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:523:19
  63:        0x10bc10134 - std::panic::catch_unwind::h0e76411dee65e50d
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panic.rs:149:14
  64:        0x10bc02774 - tokio::runtime::task::harness::poll_future::h642db26f0ffb7c44
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  65:        0x10bbfbf54 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h58f4356c53ffc9eb
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  66:        0x10bbfb7c8 - tokio::runtime::task::harness::Harness<T,S>::poll::hf70fa4f68eacbfbf
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  67:        0x10bc0d104 - tokio::runtime::task::raw::poll::h8feed0d24e548844
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
  68:        0x10bc0cc88 - tokio::runtime::task::raw::RawTask::poll::h4c036476cc49bef7
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  69:        0x10bc59a50 - tokio::runtime::task::UnownedTask<S>::run::h336702b0f5b1e1b8
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
  70:        0x10bc70774 - tokio::runtime::blocking::pool::Task::run::h62a711c52c97ee85
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
  71:        0x10bc749b8 - tokio::runtime::blocking::pool::Inner::run::h16686da72c01eacc
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
  72:        0x10bc74720 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h044b6beb6682010f
                               at /Users/dennis/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13
  73:        0x10bc8e55c - std::sys_common::backtrace::__rust_begin_short_backtrace::h136f0bef0f7855f3
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys_common/backtrace.rs:155:18
  74:        0x10bc78b74 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::he0d410ea77ce5669
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/thread/mod.rs:542:17
  75:        0x10bc0fa24 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hd0eff801916336db
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/panic/unwind_safe.rs:272:9
  76:        0x10bbf74c8 - std::panicking::try::do_call::hf3a3abfa35cf18f2
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:559:40
  77:        0x10bbf9d0c - ___rust_try
  78:        0x10bbf591c - std::panicking::try::hf0b9303982c5ca43
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panicking.rs:523:19
  79:        0x10bc78700 - std::panic::catch_unwind::h17b7351ad0aa2d8a
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/panic.rs:149:14
  80:        0x10bc78700 - std::thread::Builder::spawn_unchecked_::{{closure}}::h77a9f6d888cacaea
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/thread/mod.rs:541:30
  81:        0x10bc78d44 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h8f6ef3f97d8c74e7
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/core/src/ops/function.rs:250:5
  82:        0x10bf74c58 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hae82ec5a7f723514
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/alloc/src/boxed.rs:2022:9
  83:        0x10bf74c58 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h662e6d57208b72b3
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/alloc/src/boxed.rs:2022:9
  84:        0x10bf74c58 - std::sys::pal::unix::thread::Thread::new::thread_start::h4553383436d5a44f
                               at /rustc/f9b16149208c8a8a349c32813312716f6603eb6f/library/std/src/sys/pal/unix/thread.rs:108:17
  85:        0x19ed9af94 - __pthread_joiner_wake

What did you expect to see?

The flow should run well.

What did you see instead?

panic

What operating system did you use?

All os

What version of GreptimeDB did you use?

0.8

Relevant log output and stack trace

No response

[killme2008]

Fixed via #4018 and released v0.8.1