Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HashAgg panic when upgrades from v1.3 to v1.4 #13373

Closed
StrikeW opened this issue Nov 13, 2023 · 5 comments
Closed

HashAgg panic when upgrades from v1.3 to v1.4 #13373

StrikeW opened this issue Nov 13, 2023 · 5 comments
Assignees
@stdrc
Labels
type/bug Something isn't working
Milestone
release-1.5

Comments

@StrikeW
Copy link
Contributor

StrikeW commented Nov 13, 2023 •
edited
Loading

Describe the bug

HashAgg panic

Error message/log

thread 'risingwave-streaming-actor' panicked at /risingwave/src/stream/src/executor/aggregation/agg_group.rs:320:13:
row count should be non-negative
  11:     0x56321664ed37 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h84ffc5e7d19eca23
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/alloc/src/boxed.rs:2021:9
  12:     0x56321664ed37 - risingwave_rt::panic_hook::set_panic_hook::{{closure}}::hdd15ccf322e0c923
                               at /risingwave/src/utils/runtime/src/panic_hook.rs:25:9
  13:     0x56321664ed37 - std::panicking::update_hook::{{closure}}::hf6b9b482c57c2882
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:234:47
  14:     0x5632171fc18a - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h49df07f7cb25728d
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/alloc/src/boxed.rs:2021:9
  15:     0x5632171fc18a - std::panicking::rust_panic_with_hook::h4b1939882c095f9c
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:735:13
  16:     0x5632171fbed9 - std::panicking::begin_panic_handler::{{closure}}::ha8983f1edca62350
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:601:13
  17:     0x5632171fab56 - std::sys_common::backtrace::__rust_end_short_backtrace::h14cba478a9ee1fbd
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/sys_common/backtrace.rs:170:18
  18:     0x5632171fbc82 - rust_begin_unwind
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:597:5
  19:     0x56320ee697e5 - core::panicking::panic_fmt::h6433adef86579fb6
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/panicking.rs:72:14
  20:     0x563212321a42 - risingwave_stream::executor::aggregation::agg_group::AggGroup<S,Strtg>::curr_row_count::hee124175ffdaf2d1
  21:     0x56321231f5eb - risingwave_stream::executor::aggregation::agg_group::AggGroup<S,Strtg>::apply_chunk::{{closure}}::h51a4550fb50c32aa
                               at /risingwave/src/stream/src/executor/aggregation/agg_group.rs:351:12
  22:     0x5632123731da - risingwave_stream::executor::hash_agg::HashAggExecutor<K,S>::apply_chunk::{{closure}}::hfbfaf716b95d31bc
                               at /risingwave/src/stream/src/executor/hash_agg.rs:428:18
  23:     0x5632123731da - risingwave_stream::executor::hash_agg::HashAggExecutor<K,S>::execute_inner::{{closure}}::haaecb38db0f6acdc
                               at /risingwave/src/stream/src/executor/hash_agg.rs:659:68
  24:     0x56321228a60d - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h59ec85e8fba2e82e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  25:     0x563216dab847 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h7668540d1b086471
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  26:     0x563216dab847 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::hcbdd9965839e0df1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  27:     0x563216dab847 - futures_util::stream::stream::StreamExt::poll_next_unpin::h62dcec6df6b6da71
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  28:     0x563216dab847 - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::he011d3f6e1b6ffc4
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  29:     0x563216dab847 - <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll::h258c499e854a6a1b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/future.rs:124:23
  30:     0x563216d745ab - risingwave_stream::executor::wrapper::trace::instrument_await_tree::{{closure}}::h79596790e365d0bb
                               at /risingwave/src/stream/src/executor/wrapper/trace.rs:124:10
  31:     0x563216d745ab - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h1bba7d0e38d6ec15
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  32:     0x563216d745ab - risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}::h1a4b123fae93a2f1
                               at /risingwave/src/stream/src/executor/wrapper/schema_check.rs:24:1
  33:     0x563216d745ab - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h282145ac64771b96
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  34:     0x563216d745ab - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::he9de0af48bb0da24
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  35:     0x563216d745ab - futures_util::stream::stream::StreamExt::poll_next_unpin::hc7f5517d0354a134
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  36:     0x563216d72d3e - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::h0cbdc5b6d51a6251
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  37:     0x563216d72d3e - risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}::hbe4481905fd75e8d
                               at /risingwave/src/stream/src/executor/wrapper/epoch_check.rs:31:44
  38:     0x563216d72d3e - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::hcb1c9e54fb0fb94f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  39:     0x563216d72d3e - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h5593441d692fb71a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  40:     0x563216d72d3e - <S as futures_core::stream::TryStream>::try_poll_next::h27302e16c561f9fa
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9
  41:     0x563216d72d3e - futures_util::stream::try_stream::TryStreamExt::try_poll_next_unpin::h474e976cd75bf368
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/mod.rs:1068:9
  42:     0x563216cd8ff0 - <futures_util::stream::try_stream::try_next::TryNext<St> as core::future::future::Future>::poll::h4ae473f071f49c63
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/try_next.rs:32:9
  43:     0x563216cd8ff0 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}::h2deca66882e91d79
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:347:31
  44:     0x563216cd8ff0 - tokio::task::task_local::LocalKey<T>::scope_inner::h463266c0d941491f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:217:19
  45:     0x563216cd8ff0 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::hf20715f9cbad758f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:343:19
  46:     0x563216d73914 - risingwave_common::util::epoch::task_local::scope::{{closure}}::h0a915a1535355750
                               at /risingwave/src/common/src/util/epoch.rs:192:47
  47:     0x563216d73914 - risingwave_stream::executor::wrapper::epoch_provide::epoch_provide::{{closure}}::h570e6c5e6fc9fdbc
                               at /risingwave/src/stream/src/executor/wrapper/epoch_provide.rs:31:59
  48:     0x563216d73914 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h6dfd56b5729ed9a3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  49:     0x563216d73914 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h1ae025d6451bfc56
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  50:     0x563216d73914 - futures_util::stream::stream::StreamExt::poll_next_unpin::h1a7e3d6e6a720de9
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  51:     0x563216d502ec - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::h63131217c5c2c864
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  52:     0x563216d502ec - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hb9d07a5e8fd4ff41
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  53:     0x563216d6babe - risingwave_stream::executor::wrapper::trace::trace::{{closure}}::he951a11c19010004
                               at /risingwave/src/stream/src/executor/wrapper/trace.rs:50:69
  54:     0x563216d6babe - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h71715fba4c625b9d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  55:     0x563216d91151 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h6027befd70b38012
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  56:     0x563216d12cea - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h7668540d1b086471
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  57:     0x563216d12cea - risingwave_stream::executor::project::Inner::execute::{{closure}}::h6f39f1e9f562138c
                               at /risingwave/src/stream/src/executor/project.rs:173:5
  58:     0x563216d12cea - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h92dab2f5bacb7c16
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  59:     0x563216dab847 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h7668540d1b086471
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  60:     0x563216dab847 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::hcbdd9965839e0df1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  61:     0x563216dab847 - futures_util::stream::stream::StreamExt::poll_next_unpin::h62dcec6df6b6da71
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  62:     0x563216dab847 - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::he011d3f6e1b6ffc4
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  63:     0x563216dab847 - <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll::h258c499e854a6a1b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/future.rs:124:23
  64:     0x563216d745ab - risingwave_stream::executor::wrapper::trace::instrument_await_tree::{{closure}}::h79596790e365d0bb
                               at /risingwave/src/stream/src/executor/wrapper/trace.rs:124:10
  65:     0x563216d745ab - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h1bba7d0e38d6ec15
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  66:     0x563216d745ab - risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}::h1a4b123fae93a2f1
                               at /risingwave/src/stream/src/executor/wrapper/schema_check.rs:24:1
  67:     0x563216d745ab - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h282145ac64771b96
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  68:     0x563216d745ab - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::he9de0af48bb0da24
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  69:     0x563216d745ab - futures_util::stream::stream::StreamExt::poll_next_unpin::hc7f5517d0354a134
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  70:     0x563216d72d3e - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::h0cbdc5b6d51a6251
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  71:     0x563216d72d3e - risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}::hbe4481905fd75e8d
                               at /risingwave/src/stream/src/executor/wrapper/epoch_check.rs:31:44
  72:     0x563216d72d3e - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::hcb1c9e54fb0fb94f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  73:     0x563216d72d3e - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h5593441d692fb71a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  74:     0x563216d72d3e - <S as futures_core::stream::TryStream>::try_poll_next::h27302e16c561f9fa
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9
  75:     0x563216d72d3e - futures_util::stream::try_stream::TryStreamExt::try_poll_next_unpin::h474e976cd75bf368
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/mod.rs:1068:9
  76:     0x563216cd8ff0 - <futures_util::stream::try_stream::try_next::TryNext<St> as core::future::future::Future>::poll::h4ae473f071f49c63
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/try_next.rs:32:9
  77:     0x563216cd8ff0 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}::h2deca66882e91d79
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:347:31
  78:     0x563216cd8ff0 - tokio::task::task_local::LocalKey<T>::scope_inner::h463266c0d941491f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:217:19
  79:     0x563216cd8ff0 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::hf20715f9cbad758f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:343:19
  80:     0x563216d73914 - risingwave_common::util::epoch::task_local::scope::{{closure}}::h0a915a1535355750
                               at /risingwave/src/common/src/util/epoch.rs:192:47
  81:     0x563216d73914 - risingwave_stream::executor::wrapper::epoch_provide::epoch_provide::{{closure}}::h570e6c5e6fc9fdbc
                               at /risingwave/src/stream/src/executor/wrapper/epoch_provide.rs:31:59
  82:     0x563216d73914 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h6dfd56b5729ed9a3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  83:     0x563216d73914 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h1ae025d6451bfc56
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  84:     0x563216d73914 - futures_util::stream::stream::StreamExt::poll_next_unpin::h1a7e3d6e6a720de9
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  85:     0x563216d502ec - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::h63131217c5c2c864
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  86:     0x563216d502ec - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hb9d07a5e8fd4ff41
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  87:     0x563216d6babe - risingwave_stream::executor::wrapper::trace::trace::{{closure}}::he951a11c19010004
                               at /risingwave/src/stream/src/executor/wrapper/trace.rs:50:69
  88:     0x563216d6babe - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h71715fba4c625b9d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  89:     0x563216d91151 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h6027befd70b38012
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  90:     0x563211de5bc3 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h4f1da7344a5f08f1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  91:     0x563211de5bc3 - risingwave_stream::executor::mview::materialize::MaterializeExecutor<S,SD>::execute_inner::{{closure}}::h1294584bd36a94f6
                               at /risingwave/src/stream/src/executor/mview/materialize.rs:113:5
  92:     0x563211da397d - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::ha59dc77e35a59f62
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  93:     0x563216dab847 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h7668540d1b086471
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  94:     0x563216dab847 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::hcbdd9965839e0df1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  95:     0x563216dab847 - futures_util::stream::stream::StreamExt::poll_next_unpin::h62dcec6df6b6da71
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  96:     0x563216dab847 - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::he011d3f6e1b6ffc4
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  97:     0x563216dab847 - <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll::h258c499e854a6a1b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/future.rs:124:23
  98:     0x563216d745ab - risingwave_stream::executor::wrapper::trace::instrument_await_tree::{{closure}}::h79596790e365d0bb
                               at /risingwave/src/stream/src/executor/wrapper/trace.rs:124:10
  99:     0x563216d745ab - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h1bba7d0e38d6ec15
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
 100:     0x563216d745ab - risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}::h1a4b123fae93a2f1
                               at /risingwave/src/stream/src/executor/wrapper/schema_check.rs:24:1
 101:     0x563216d745ab - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h282145ac64771b96
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
 102:     0x563216d745ab - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::he9de0af48bb0da24
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
 103:     0x563216d745ab - futures_util::stream::stream::StreamExt::poll_next_unpin::hc7f5517d0354a134
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
 104:     0x563216d72d3e - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::h0cbdc5b6d51a6251
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:9
 105:     0x563216d72d3e - risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}::hbe4481905fd75e8d
                               at /risingwave/src/stream/src/executor/wrapper/epoch_check.rs:31:44
 106:     0x563216d72d3e - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::hcb1c9e54fb0fb94f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
 107:     0x563216d72d3e - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h5593441d692fb71a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
 108:     0x563216d72d3e - <S as futures_core::stream::TryStream>::try_poll_next::h27302e16c561f9fa
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9
 109:     0x563216d72d3e - futures_util::stream::try_stream::TryStreamExt::try_poll_next_unpin::h474e976cd75bf368
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/mod.rs:1068:9
 110:     0x563216cd8ff0 - <futures_util::stream::try_stream::try_next::TryNext<St> as core::future::future::Future>::poll::h4ae473f071f49c63
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/try_next.rs:32:9
 111:     0x563216cd8ff0 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}::h2deca66882e91d79
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:347:31
 112:     0x563216cd8ff0 - tokio::task::task_local::LocalKey<T>::scope_inner::h463266c0d941491f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:217:19
 113:     0x563216cd8ff0 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::hf20715f9cbad758f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:343:19
 114:     0x563216d73914 - risingwave_common::util::epoch::task_local::scope::{{closure}}::h0a915a1535355750
                               at /risingwave/src/common/src/util/epoch.rs:192:47
 115:     0x563216d73914 - risingwave_stream::executor::wrapper::epoch_provide::epoch_provide::{{closure}}::h570e6c5e6fc9fdbc
                               at /risingwave/src/stream/src/executor/wrapper/epoch_provide.rs:31:59
 116:     0x563216d73914 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h6dfd56b5729ed9a3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
 117:     0x563216d73914 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h1ae025d6451bfc56
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
 118:     0x563216d73914 - futures_util::stream::stream::StreamExt::poll_next_unpin::h1a7e3d6e6a720de9
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
 119:     0x563216d502ec - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::h63131217c5c2c864
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:9
 120:     0x563216d502ec - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hb9d07a5e8fd4ff41
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
 121:     0x563216d6babe - risingwave_stream::executor::wrapper::trace::trace::{{closure}}::he951a11c19010004
                               at /risingwave/src/stream/src/executor/wrapper/trace.rs:50:69
 122:     0x563216d6babe - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h71715fba4c625b9d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
 123:     0x563216d91151 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h6027befd70b38012
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
 124:     0x5632126046a6 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h4f1da7344a5f08f1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
 125:     0x5632126046a6 - <risingwave_stream::executor::dispatch::DispatchExecutor as risingwave_stream::executor::StreamConsumer>::execute::{{closure}}::hc137991b13210d9f
                               at /risingwave/src/stream/src/executor/dispatch.rs:312:9
 126:     0x5632126046a6 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h9db5e6303da5615b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
 127:     0x5632126046a6 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h42913077a6cdb32e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
 128:     0x5632126046a6 - <&mut S as futures_core::stream::Stream>::poll_next::h73ee74c3eb5ba669
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:104:9
 129:     0x56321224cd5a - <tokio_stream::stream_ext::next::Next<St> as core::future::future::Future>::poll::hae6a05ff55f0bc88
                               at /root/.cargo/git/checkouts/tokio-968c02b7a1a41bea/fe39bb8/tokio-stream/src/stream_ext/next.rs:42:9
 130:     0x56321224cd5a - <tokio_stream::stream_ext::try_next::TryNext<St> as core::future::future::Future>::poll::h437bd6bc1dcdae25
                               at /root/.cargo/git/checkouts/tokio-968c02b7a1a41bea/fe39bb8/tokio-stream/src/stream_ext/try_next.rs:43:9
 131:     0x56321224cd5a - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h7c2bdd857c68248a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
 132:     0x563211ee8e50 - <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll::hd71b852bb62322fc
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/future.rs:124:23
 133:     0x563211ee8e50 - risingwave_stream::executor::actor::Actor<C>::run_consumer::{{closure}}::hfaca0c9f75feb57b
                               at /risingwave/src/stream/src/executor/actor.rs:191:18
 134:     0x563211ee8e50 - <tokio::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll::he3cee74d8298e025
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/future/maybe_done.rs:68:48
 135:     0x56321282e77f - risingwave_stream::executor::actor::Actor<C>::run::{{closure}}::{{closure}}::ha275763cbaf51e4d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/macros/join.rs:126:24
 136:     0x56321282e77f - <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll::h165a85c3d86c734b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/future/poll_fn.rs:58:9
 137:     0x563211d9f5cb - risingwave_stream::executor::actor::Actor<C>::run::{{closure}}::h14fb0662f5b6fff9
                               at /risingwave/src/stream/src/executor/actor.rs:155:9
 138:     0x563211d9f5cb - risingwave_stream::task::stream_manager::LocalStreamManagerCore::build_actors::{{closure}}::{{closure}}::h5f1a15815e3892d3
                               at /risingwave/src/stream/src/task/stream_manager.rs:697:51
 139:     0x563211d9f5cb - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}::h5d45b7239cee6192
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:347:31
 140:     0x563211d9f5cb - tokio::task::task_local::LocalKey<T>::scope_inner::h70a7123c114e8b6a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:217:19
 141:     0x563211d9f5cb - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::hb10a0467c74c4ad1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:343:19
 142:     0x563212834f22 - await_tree::registry::TreeRoot::instrument::{{closure}}::h04190485f035eb77
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/registry.rs:51:45
 143:     0x563212834f22 - <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll::hf8265ac0b8893ba3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/future/either.rs:109:32
 144:     0x563211d96e03 - core::ops::function::FnOnce::call_once::h4a5a644004f22f7c
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/ops/function.rs:250:5
 145:     0x563211d96e03 - tokio_metrics::task::instrument_poll::hdab081f8f8b41aeb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-metrics-0.3.0/src/task.rs:2530:15
 146:     0x563211d96e03 - <tokio_metrics::task::Instrumented<T> as core::future::future::Future>::poll::h576747c5298ea1e0
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-metrics-0.3.0/src/task.rs:2430:9
 147:     0x56321226c46b - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hd3d5a5f0b84eb854
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
 148:     0x563211d64b0d - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h2e9eeac4f22557ea
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:334:17
 149:     0x563211d64b0d - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hb8bfe2086b03fe57
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/loom/std/unsafe_cell.rs:16:9
 150:     0x563211d64b0d - tokio::runtime::task::core::Core<T,S>::poll::h343cfa026e5a49d7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:323:13
 151:     0x5632127ccc07 - tokio::runtime::task::harness::poll_future::{{closure}}::h2a955f9911c4f2b6
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:485:19
 152:     0x5632127ccc07 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h33e0c7bdf5a53e24
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/panic/unwind_safe.rs:271:9
 153:     0x5632127ccc07 - std::panicking::try::do_call::hb3f8370823c46cff
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:504:40
 154:     0x5632127ccc07 - std::panicking::try::hf587eaccd6f43a34
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:468:19
 155:     0x5632127ccc07 - std::panic::catch_unwind::h7e04aa53be7a0bea
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panic.rs:142:14
 156:     0x5632127ccc07 - tokio::runtime::task::harness::poll_future::h52181cfb0360c191
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:473:18
 157:     0x5632127ccc07 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h72b37e83385f3493
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:208:27
 158:     0x5632127ccc07 - tokio::runtime::task::harness::Harness<T,S>::poll::hd5dfaec93c66c42d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:153:15
 159:     0x5632127ccc07 - tokio::runtime::task::raw::poll::h5f0e4e8ca03328ab
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/raw.rs:276:5
 160:     0x5632172797d8 - tokio::runtime::task::raw::RawTask::poll::h7ebd863582ea2ad6
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/raw.rs:200:18
 161:     0x5632172797d8 - tokio::runtime::task::LocalNotified<S>::run::h3b943f98ac88e2db
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/mod.rs:400:9
 162:     0x5632172797d8 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h62c1fe61004840f1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:576:18
 163:     0x5632172797d8 - tokio::runtime::coop::with_budget::h23f639a1a4aedc7d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/coop.rs:107:5
 164:     0x5632172797d8 - tokio::runtime::coop::budget::h01eeda705eb75561
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/coop.rs:73:5
 165:     0x5632172797d8 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::he79620c542bb42fe
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
 166:     0x5632172747b3 - tokio::runtime::scheduler::multi_thread::worker::Context::run::hbd22c315ff4cf11a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:538:24
 167:     0x5632172747b3 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::h15a0081203b5c601
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
 168:     0x5632172747b3 - tokio::runtime::context::scoped::Scoped<T>::set::hec82173cdfa8d173
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context/scoped.rs:40:9
 169:     0x5632172703cd - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h452c136d9bd5ab8a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
 170:     0x5632172703cd - tokio::runtime::context::runtime::enter_runtime::h53ff0ebc33332a6d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context/runtime.rs:65:16
 171:     0x5632172795cc - tokio::runtime::scheduler::multi_thread::worker::run::hd0e8f6ce922c9d1a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
 172:     0x563217270b43 - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h04fd26ff84544247
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
 173:     0x563217270b43 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h00556c95279a599a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/task.rs:42:21
 174:     0x56321728bd6b - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h214a1e751f9a4d8a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
 175:     0x5632172a54f1 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hab84489d8cca6c8e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:334:17
 176:     0x5632172a54f1 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h7603ecccc3a11fa0
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/loom/std/unsafe_cell.rs:16:9
 177:     0x5632172a54f1 - tokio::runtime::task::core::Core<T,S>::poll::ha384932544447281
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:323:13
 178:     0x56321728772a - tokio::runtime::task::harness::poll_future::{{closure}}::h798f13bf93bce16b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:485:19
 179:     0x56321728772a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h8c10762cea1f2dcc
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/panic/unwind_safe.rs:271:9
 180:     0x56321728772a - std::panicking::try::do_call::hd023da856054cada
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:504:40
 181:     0x56321728772a - std::panicking::try::h83bb2619fc473d67
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:468:19
 182:     0x56321728772a - std::panic::catch_unwind::h8f365cc2dc1a2a1c
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panic.rs:142:14
 183:     0x56321728772a - tokio::runtime::task::harness::poll_future::h4e9b72cf3eb94262
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:473:18
 184:     0x56321728772a - tokio::runtime::task::harness::Harness<T,S>::poll_inner::hb18195996bd0a551
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:208:27
 185:     0x56321728772a - tokio::runtime::task::harness::Harness<T,S>::poll::he1b11e9f43dc6e59
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:153:15
 186:     0x56321729a94e - tokio::runtime::task::raw::RawTask::poll::h7ebd863582ea2ad6
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/raw.rs:200:18
 187:     0x56321729a94e - tokio::runtime::task::UnownedTask<S>::run::hf58142dc7b390be9
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/mod.rs:437:9
 188:     0x56321729a94e - tokio::runtime::blocking::pool::Task::run::h5c9e517beb5fa38d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/pool.rs:159:9
 189:     0x56321729a94e - tokio::runtime::blocking::pool::Inner::run::he5bed20936091353
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/pool.rs:513:17
 190:     0x5632172a9ef0 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::hd9357aa96847ee5a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/pool.rs:471:13
 191:     0x5632172a9ef0 - std::sys_common::backtrace::__rust_begin_short_backtrace::hc8373561a17ffd12
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/sys_common/backtrace.rs:154:18
 192:     0x56321729c00d - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h510b0f71955e5baf
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/thread/mod.rs:529:17
 193:     0x56321729c00d - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h1ea1d852193adc5b
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/panic/unwind_safe.rs:271:9
 194:     0x56321729c00d - std::panicking::try::do_call::h5fd2d6cb3474bd61
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:504:40
 195:     0x56321729c00d - std::panicking::try::he99acd0f2a014b43
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:468:19
 196:     0x56321729c00d - std::panic::catch_unwind::h31595a6835076bf2
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panic.rs:142:14
 197:     0x56321729c00d - std::thread::Builder::spawn_unchecked_::{{closure}}::h176fef9e98084d6f
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/thread/mod.rs:528:30
 198:     0x56321729c00d - core::ops::function::FnOnce::call_once{{vtable.shim}}::hee67ae58f0fff7da
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/ops/function.rs:250:5
 199:     0x563217202245 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hac2e24a72c9237b1
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/alloc/src/boxed.rs:2007:9
 200:     0x563217202245 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h0a4ea247e7a2feab
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/alloc/src/boxed.rs:2007:9
 201:     0x563217202245 - std::sys::unix::thread::Thread::new::thread_start::ha157bf15852ae2a4
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/sys/unix/thread.rs:108:17
 202:     0x7f55d5cbcac3 - start_thread
                               at ./nptl/pthread_create.c:442:8
 203:     0x7f55d5d4dbf4 - clone
                               at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:100
 204:                0x0 - <unknown>
stack backtrace:
   0:     0x5632171fa697 - std::backtrace_rs::backtrace::libunwind::trace::he8dc9e634c3d5b69
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x5632171fa697 - std::backtrace_rs::backtrace::trace_unsynchronized::hcc1fc094cc2c1843
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5


*** await tree context of current task ***

Actor 3994: `dim_mk_user_dept_path` [11.861s]
  Epoch 5398601865560064 [180.000ms]
    Materialize F9A0000000A (actor 3994) [180.000ms]
      ProjectExecutor F9A00000009 (actor 3994) [180.000ms]
        HashAggExecutor F9A00000008 (actor 3994) [179.000ms]  <== current
[Detached 23]
  LocalInput (actor 4048) [179.000ms]



### To Reproduce

_No response_

### Expected behavior

_No response_

### How did you deploy RisingWave?

_No response_

### The version of RisingWave

_No response_

### Additional context

_No response_
@StrikeW StrikeW added the type/bug Something isn't working label Nov 13, 2023
@github-actions github-actions bot added this to the release-1.5 milestone Nov 13, 2023
@StrikeW
Copy link
Contributor Author

StrikeW commented Nov 13, 2023

cc @BugenZhao

@lmatz
Copy link
Contributor

lmatz commented Nov 13, 2023

After it is fixed, could you help construct a test case and add it to https://github.com/risingwavelabs/risingwave/tree/main/backwards-compat-tests

@lmatz lmatz mentioned this issue Nov 14, 2023
Closed
@stdrc
Copy link
Member

stdrc commented Nov 20, 2023

A late summary for this issue:

The query plan generated on v1.4 looks exactly the same as that on v1.3, table schema, executor schema and key indices are all the same. So I doubt whether it's really a backward-compatibility issue. The panic seems more like a random data error similar to #11170, and seems we don't have a good method to debug this kind of errors currently, except for manually careful review for all the code path.

@stdrc
Copy link
Member

stdrc commented Nov 22, 2023

After #13559 (comment) and #13465 I think I have more experience on operator compatibility issue, and I realized the incompatibility may happen in somewhere can't seen in query plan. Will have another try to check if there's any protobuf change.

@st1page
Copy link
Contributor

st1page commented Nov 28, 2023

can not reproduce and did not find any strange things

@st1page st1page closed this as not planned Won't fix, can't repro, duplicate, stale Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stdrc stdrc

Labels
type/bug Something isn't working
Projects
None yet
Milestone
release-1.5
Development

No branches or pull requests
4 participants
@stdrc @lmatz @StrikeW @st1page