Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: panic: already paused #18129

Closed
fuyufjh opened this issue Aug 20, 2024 · 2 comments
Closed

bug: panic: already paused #18129

fuyufjh opened this issue Aug 20, 2024 · 2 comments
Assignees
@zwang28
Labels
need-cherry-pick-release-2.0 priority/high type/bug Something isn't working
Milestone
release-2.0

Comments

@fuyufjh
Copy link
Member

fuyufjh commented Aug 20, 2024 •
edited
Loading

Describe the bug

Occurred twice recently.

Thanks for @zwang28 and @yezizp2012 for investigating.

@zwang28 : It is possible for pauses to occur repeatedly. For instance, a source executor may pause on its own due to barrier backpressure.

Error message/log

thread 'rw-streaming' panicked at src/stream/src/executor/stream_reader.rs:98:9:
already paused
stack backtrace:
   0:     0xaaaaf2071750 - std::backtrace_rs::backtrace::libunwind::trace::h64ef74c421ddf515
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:     0xaaaaf2071750 - std::backtrace_rs::backtrace::trace_unsynchronized::h0948cfffd26b9adf
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0xaaaaf2071750 - std::sys_common::backtrace::_print_fmt::hec7c7f44910dd893
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:68:5
   3:     0xaaaaf2071750 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h6ef892d3f076e581
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:44:22
   4:     0xaaaae82502d8 - core::fmt::rt::Argument::fmt::hc7ee4c2da8957d17
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/fmt/rt.rs:142:9
   5:     0xaaaae82502d8 - core::fmt::write::h0c4a627b12e3d78d
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/fmt/mod.rs:1120:17
   6:     0xaaaaf206a420 - std::io::Write::write_fmt::h318ea8fa9c57551f
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/io/mod.rs:1810:15
   7:     0xaaaaf207157c - std::sys_common::backtrace::_print::ha8e58b5a72e68af5
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:47:5
   8:     0xaaaaf207157c - std::sys_common::backtrace::print::hbce77ec9c0ee5615
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:34:9
   9:     0xaaaaf2072bf4 - std::panicking::default_hook::{{closure}}::h62de2f44289d99c3
  10:     0xaaaaf2072928 - std::panicking::default_hook::h6808acaf9c18352a
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:292:9
  11:     0xaaaaefd3aa00 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h85e60adb92856e67
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/alloc/src/boxed.rs:2029:9
  12:     0xaaaaefd3aa00 - risingwave_rt::panic_hook::set_panic_hook::{{closure}}::hb6c2d8651df127d7
                               at /risingwave/src/utils/runtime/src/panic_hook.rs:25:9
  13:     0xaaaaefd3aa00 - std::panicking::update_hook::{{closure}}::h9125a948bdea8585
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:234:47
  14:     0xaaaaf2073240 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h861e29f50d7dc506
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/alloc/src/boxed.rs:2029:9
  15:     0xaaaaf2073240 - std::panicking::rust_panic_with_hook::h7d9418a0e45d61fb
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:783:13
  16:     0xaaaaf2072fbc - std::panicking::begin_panic_handler::{{closure}}::he5c28c60ba487bed
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:649:13
  17:     0xaaaaf2071c24 - std::sys_common::backtrace::__rust_end_short_backtrace::hcdcd48a39dadcd83
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:171:18
  18:     0xaaaaf2072d90 - rust_begin_unwind
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:645:5
  19:     0xaaaae824cea4 - core::panicking::panic_fmt::had704c91ae6f0365
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/panicking.rs:72:14
  20:     0xaaaaf1476454 - risingwave_stream::executor::stream_reader::StreamReaderWithPause<_,M>::pause_stream::h51dd9caa462f1c65
                               at /risingwave/src/stream/src/executor/stream_reader.rs:98:9
  21:     0xaaaaf13a39d4 - risingwave_stream::executor::source::source_executor::SourceExecutor<S>::execute_with_stream_source::{{closure}}::h5fd8fada83199bf8
                               at /risingwave/src/stream/src/executor/source/source_executor.rs:506:48
  22:     0xaaaaf121c30c - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h23e179a34af0c5f5
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  23:     0xaaaaf0ee4f04 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h577cc1ff41586148
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:120:9
  24:     0xaaaaf0ee4f04 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h8c8a666ce7f09e03
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:120:9
  25:     0xaaaaf0ee4f04 - futures_util::stream::stream::StreamExt::poll_next_unpin::hb5ea4c39a14505de
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/mod.rs:1638:9
  26:     0xaaaaf0ee4f04 - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::hb19813d0b7c5275d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/next.rs:32:9
  27:     0xaaaaf0ee4f04 - <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll::hcbeb1f679dc4fbdb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/future.rs:119:15
  28:     0xaaaaf09bb1d0 - risingwave_stream::executor::wrapper::trace::instrument_await_tree::{{closure}}::h12c163a942bba76d
                               at /risingwave/src/stream/src/executor/wrapper/trace.rs:115:10
  29:     0xaaaaf09bb1d0 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h3b001e02507f456d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  30:     0xaaaaf09bb1d0 - risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}::h6121c4de8b068a82
                               at /risingwave/src/stream/src/executor/wrapper/schema_check.rs:24:1
  31:     0xaaaaf09bb1d0 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h7041204555f3e5ad
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  32:     0xaaaaf09bb1d0 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::hd21d225fd8c4e94c
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:120:9
  33:     0xaaaaf09bb1d0 - futures_util::stream::stream::StreamExt::poll_next_unpin::h6379c117d009370c
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/mod.rs:1638:9
  34:     0xaaaaf09b912c - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::hc50bd47d612517eb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/next.rs:32:9
  35:     0xaaaaf09b912c - risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}::ha12c747f6fb74cbe
                               at /risingwave/src/stream/src/executor/wrapper/epoch_check.rs:31:44
  36:     0xaaaaf09b912c - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h496b3eea58ba39ea
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  37:     0xaaaaf09b912c - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h222d1a0c6ddc9aab
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:120:9
  38:     0xaaaaf09b912c - <S as futures_core::stream::TryStream>::try_poll_next::ha3cc006fb0d0ad98
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:196:9
  39:     0xaaaaf09b912c - futures_util::stream::try_stream::TryStreamExt::try_poll_next_unpin::hefe0f2b248df04ef
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/try_stream/mod.rs:1131:9
  40:     0xaaaaf1755e08 - <futures_util::stream::try_stream::try_next::TryNext<St> as core::future::future::Future>::poll::he4dcea2912cfebeb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/try_stream/try_next.rs:32:9
  41:     0xaaaaf1755e08 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}::h98d98c72580c7a00
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:391:31
  42:     0xaaaaf1755e08 - tokio::task::task_local::LocalKey<T>::scope_inner::hac3a70bc6487f007
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:217:19
  43:     0xaaaaf1755e08 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::hfb34493fde5c500c
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:387:19
  44:     0xaaaaf09b9740 - risingwave_common::util::epoch::task_local::scope::{{closure}}::hcb25fa8fe2cd1fef
                               at /risingwave/src/common/src/util/epoch.rs:244:47
  45:     0xaaaaf09b9740 - risingwave_stream::executor::wrapper::epoch_provide::epoch_provide::{{closure}}::h8590d3417c5c69f0
                               at /risingwave/src/stream/src/executor/wrapper/epoch_provide.rs:31:59
  46:     0xaaaaf09b9740 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::hc7bce06645a83e8e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  47:     0xaaaaf09b9740 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h88ffd90ea14c0849
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:120:9
  48:     0xaaaaf09b9740 - futures_util::stream::stream::StreamExt::poll_next_unpin::h1896319324a1c825
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/mod.rs:1638:9
  49:     0xaaaaf0c27228 - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::h54ef41921b1bc09e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/next.rs:32:9
  50:     0xaaaaf0c27228 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hff3d97ec8f3c2f6a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  51:     0xaaaaf18c1a0c - risingwave_stream::executor::wrapper::trace::trace::{{closure}}::h7e21621af30d43b0
                               at /risingwave/src/stream/src/executor/wrapper/trace.rs:48:69
  52:     0xaaaaf18c1a0c - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h3253885993835022
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  53:     0xaaaaf0a0f74c - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h577cc1ff41586148
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:120:9
  54:     0xaaaaf0a0f74c - <risingwave_stream::executor::dispatch::DispatchExecutor as risingwave_stream::executor::StreamConsumer>::execute::{{closure}}::h3aa58d3a8efcfac1
                               at /risingwave/src/stream/src/executor/dispatch.rs:386:9
  55:     0xaaaaf0a0f74c - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h92dbae1e3404dc6e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  56:     0xaaaaf0a0f74c - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h717fa52506694db8
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:120:9
  57:     0xaaaaf0a0f74c - <&mut S as futures_core::stream::Stream>::poll_next::h0e9284f6bb8dbb62
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:104:9
  58:     0xaaaaf0c113b4 - <tokio_stream::stream_ext::next::Next<St> as core::future::future::Future>::poll::h09b4d642f79014ee
                               at /root/.cargo/git/checkouts/tokio-968c02b7a1a41bea/fe39bb8/tokio-stream/src/stream_ext/next.rs:42:9
  59:     0xaaaaf0c113b4 - <tokio_stream::stream_ext::try_next::TryNext<St> as core::future::future::Future>::poll::h3ab1ed035dc314d3
                               at /root/.cargo/git/checkouts/tokio-968c02b7a1a41bea/fe39bb8/tokio-stream/src/stream_ext/try_next.rs:43:9
  60:     0xaaaaf0c113b4 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h6569f3370ae50205
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  61:     0xaaaaf0ebe2f4 - <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll::h44ef3780344a651b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/future.rs:119:15
  62:     0xaaaaf1754530 - risingwave_stream::executor::actor::Actor<C>::run_consumer::{{closure}}::hce919554efe65aac
                               at /risingwave/src/stream/src/executor/actor.rs:214:18
  63:     0xaaaaf1754530 - <tokio::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll::h76b68ec0d8efb7bf
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/future/maybe_done.rs:62:56
  64:     0xaaaaf1754530 - risingwave_stream::executor::actor::Actor<C>::run::{{closure}}::{{closure}}::{{closure}}::h8ccf375db0bdc602
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/macros/join.rs:126:24
  65:     0xaaaaf1754530 - <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll::h92c016ea9e417fb3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/future/poll_fn.rs:58:9
  66:     0xaaaaf1754530 - risingwave_stream::executor::actor::Actor<C>::run::{{closure}}::{{closure}}::hf70eed1bd7d4b1c9
                               at /risingwave/src/stream/src/executor/actor.rs:170:17
  67:     0xaaaaf1754530 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}::h016f1b04289bd294
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:391:31
  68:     0xaaaaf1754530 - tokio::task::task_local::LocalKey<T>::scope_inner::hcaf93a963b944912
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:217:19
  69:     0xaaaaf1754530 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::h91b8fcb4635fdaeb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:387:19
  70:     0xaaaaf1754530 - risingwave_expr::expr_context::expr_context_scope::{{closure}}::h670e273ac1836958
                               at /risingwave/src/expr/core/src/expr_context.rs:35:65
  71:     0xaaaaf1754530 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}::he5f812274479dfbd
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:391:31
  72:     0xaaaaf1754530 - tokio::task::task_local::LocalKey<T>::scope_inner::hdfde48866e49c512
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:217:19
  73:     0xaaaaf1754530 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::hd28a87df04004b63
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:387:19
  74:     0xaaaaf0c591b8 - risingwave_stream::executor::actor::Actor<C>::run::{{closure}}::h808a90381bdfcceb
                               at /risingwave/src/stream/src/executor/actor.rs:178:10
  75:     0xaaaaf0c591b8 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::hd82bfb6d88abee1b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:55:37
  76:     0xaaaaf17558a0 - <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll::h87191a308062371e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
  77:     0xaaaaf17558a0 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}::h3489220495a61dc6
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:391:31
  78:     0xaaaaf17558a0 - tokio::task::task_local::LocalKey<T>::scope_inner::ha381e93eb8e83bf2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:217:19
  79:     0xaaaaf17558a0 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::hf923be965e99d1b6
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:387:19
  80:     0xaaaaf17558a0 - await_tree::root::TreeRoot::instrument::{{closure}}::hcf59ea178d3c8c1f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/root.rs:43:34
  81:     0xaaaaf17558a0 - <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll::h1be1bcf62383b5b7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/either.rs:109:32
  82:     0xaaaaf17558a0 - core::ops::function::FnOnce::call_once::hda5e441e6cdea480
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/ops/function.rs:250:5
  83:     0xaaaaf17558a0 - tokio_metrics::task::instrument_poll::hc2eb384fd857ddf9
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-metrics-0.3.1/src/task.rs:2530:15
  84:     0xaaaaf17558a0 - <tokio_metrics::task::Instrumented<T> as core::future::future::Future>::poll::h966d89e3c414a8a8
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-metrics-0.3.1/src/task.rs:2430:9
  85:     0xaaaaf17558a0 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}::hff9e9a099b2feb4e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:391:31
  86:     0xaaaaf17558a0 - tokio::task::task_local::LocalKey<T>::scope_inner::h1ba5cb2cb382bb93
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:217:19
  87:     0xaaaaf17558a0 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::hf7708de63246f9dc
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:387:19
  88:     0xaaaaf0c20434 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hd1ef3d4eb1b1e06a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  89:     0xaaaaf11a8994 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h3761dca74a0e4f95
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  90:     0xaaaaf11a8994 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hf2d5a17dc447893f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  91:     0xaaaaf11a8994 - tokio::runtime::task::core::Core<T,S>::poll::h1062221d38492caf
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  92:     0xaaaaf17057c4 - tokio::runtime::task::harness::poll_future::{{closure}}::h6de82830bdf55a40
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  93:     0xaaaaf17057c4 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h4bfbd801a86b2d36
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/panic/unwind_safe.rs:272:9
  94:     0xaaaaf17057c4 - std::panicking::try::do_call::h317752bdb1926f57
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:552:40
  95:     0xaaaaf17057c4 - std::panicking::try::h4a834347e6e3dde0
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:516:19
  96:     0xaaaaf17057c4 - std::panic::catch_unwind::h2d7cf1c8c3953e59
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panic.rs:142:14
  97:     0xaaaaf17057c4 - tokio::runtime::task::harness::poll_future::h615ffc40cab110b0
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  98:     0xaaaaf17057c4 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::hbb8b54bff87ac749
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  99:     0xaaaaf17057c4 - tokio::runtime::task::harness::Harness<T,S>::poll::hb220452048d33a66
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
 100:     0xaaaaf2118294 - tokio::runtime::task::raw::RawTask::poll::h9d96154b4838b0b6
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
 101:     0xaaaaf2118294 - tokio::runtime::task::LocalNotified<S>::run::hda0b6cef14fbb6dc
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
 102:     0xaaaaf2118294 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h27fe118842a96d9b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:576:18
 103:     0xaaaaf2118294 - tokio::runtime::coop::with_budget::h3b7152f6002e0e4b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
 104:     0xaaaaf2118294 - tokio::runtime::coop::budget::hddde173f4ee324f5
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
 105:     0xaaaaf2118294 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h1a3e884f378d0b30
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
 106:     0xaaaaf21134e0 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h8c6174e500958761
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
 107:     0xaaaaf21134e0 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::hacd0570de53d8c91
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
 108:     0xaaaaf21134e0 - tokio::runtime::context::scoped::Scoped<T>::set::h2dd93cdcc8b8fb1d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
 109:     0xaaaaf210fc40 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h16d62cbd64d3767e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
 110:     0xaaaaf210fc40 - tokio::runtime::context::runtime::enter_runtime::hed4fc3c9e4867994
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
 111:     0xaaaaf2118024 - tokio::runtime::scheduler::multi_thread::worker::run::h0452635f67c85d24
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
 112:     0xaaaaf211db3c - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h8050cc54016eb23c
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
 113:     0xaaaaf211db3c - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::hb978e2f2144ea4eb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
 114:     0xaaaaf20fd424 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h33f42398eaf48b89
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
 115:     0xaaaaf2106cbc - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h071a829e5d913542
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
 116:     0xaaaaf2106cbc - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hfbf2e0674ae40403
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
 117:     0xaaaaf2106cbc - tokio::runtime::task::core::Core<T,S>::poll::h0ffae1266b03a385
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
 118:     0xaaaaf2120ba4 - tokio::runtime::task::harness::poll_future::{{closure}}::h53845b13076012c2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
 119:     0xaaaaf2120ba4 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h6f71580609e36c5f
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/panic/unwind_safe.rs:272:9
 120:     0xaaaaf2120ba4 - std::panicking::try::do_call::hed1c69bedaa6a98b
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:552:40
 121:     0xaaaaf2120ba4 - std::panicking::try::h81af1d7ae6256c72
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:516:19
 122:     0xaaaaf2120ba4 - std::panic::catch_unwind::hf7d2c435a0ae2731
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panic.rs:142:14
 123:     0xaaaaf2120ba4 - tokio::runtime::task::harness::poll_future::h22daf6ef0fa590d7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
 124:     0xaaaaf2120ba4 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h2161eb1dfd479001
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
 125:     0xaaaaf2120ba4 - tokio::runtime::task::harness::Harness<T,S>::poll::ha11170cb5b5e3edb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
 126:     0xaaaaf20eb634 - tokio::runtime::task::raw::RawTask::poll::h9d96154b4838b0b6
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
 127:     0xaaaaf20eb634 - tokio::runtime::task::UnownedTask<S>::run::h998663e060e749ba
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
 128:     0xaaaaf20eb634 - tokio::runtime::blocking::pool::Task::run::h1d25812c8a719a3a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
 129:     0xaaaaf20eb634 - tokio::runtime::blocking::pool::Inner::run::h0b3aab04e9e4f3fb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
 130:     0xaaaaf2119508 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::hf2b2e2d4d6d2320f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13
 131:     0xaaaaf2119508 - std::sys_common::backtrace::__rust_begin_short_backtrace::h70c6b23d47d4e5ea
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:155:18
 132:     0xaaaaf20dcf4c - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hf4c5f1df0db9bc8d
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/thread/mod.rs:529:17
 133:     0xaaaaf20dcf4c - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h78637008a5cab536
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/panic/unwind_safe.rs:272:9
 134:     0xaaaaf20dcf4c - std::panicking::try::do_call::h4ff8bb2e2cdf93ec
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:552:40
 135:     0xaaaaf20dcf4c - std::panicking::try::hf52459e5d967e87e
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:516:19
 136:     0xaaaaf20dcf4c - std::panic::catch_unwind::h44ee22e1819e3219
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panic.rs:142:14
 137:     0xaaaaf20dcf4c - std::thread::Builder::spawn_unchecked_::{{closure}}::hbabc84131c9a8a94
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/thread/mod.rs:528:30
 138:     0xaaaaf20dcf4c - core::ops::function::FnOnce::call_once{{vtable.shim}}::h6703ee6b4c669d55
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/ops/function.rs:250:5
 139:     0xaaaaf2078f84 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h7bd40c020b8a94f5
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/alloc/src/boxed.rs:2015:9
 140:     0xaaaaf2078f84 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h458d66792ce1a28c
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/alloc/src/boxed.rs:2015:9
 141:     0xaaaaf2078f84 - std::sys::unix::thread::Thread::new::thread_start::h979e8ba27e8109aa
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys/unix/thread.rs:108:17
 142:     0xffffa70dd5c8 - start_thread
                               at ./nptl/pthread_create.c:442:8
 143:     0xffffa7145edc - thread_start
                               at ./misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79
 144:                0x0 - <unknown>


*** await tree context of current task ***

Actor 771877: `**redacted***` [273.148s]
  Epoch 7005952601948160 [1.070s]
    Source BC72500002712 [1.070s]  <== current

How did you deploy RisingWave?

cloud

The version of RisingWave

main

Additional context

More context in Slack: https://risingwave-labs.slack.com/archives/C07ES5Q3TSA/p1722477934479369?thread_ts=1722476840.640829&cid=C07ES5Q3TSA
@fuyufjh fuyufjh added type/bug Something isn't working priority/high labels Aug 20, 2024
@github-actions github-actions bot added this to the release-2.1 milestone Aug 20, 2024
@fuyufjh fuyufjh modified the milestones: release-2.1, release-2.0 Aug 20, 2024
@fuyufjh fuyufjh added need-cherry-pick-release-1.10 Open a cherry-pick PR to branch release-1.10 after the current PR is merged need-cherry-pick-release-2.0 and removed need-cherry-pick-release-1.10 Open a cherry-pick PR to branch release-1.10 after the current PR is merged labels Aug 20, 2024
@yezizp2012
Copy link
Member

The root cause has been identified, I reproduced it locally. It will only happen in a cluster where there is a shared source and no mv dependencies have been created on it. The CN will panic if we send a pause command to it:

risingwave/src/stream/src/executor/source/source_executor.rs

Lines 470 to 476 in c4adffc

if (self.is_shared && is_uninitialized) || barrier.is_pause_on_startup() {
tracing::info!(
is_shared = self.is_shared,
is_uninitialized = is_uninitialized,
"source paused on startup"
);
stream.pause_stream();

Also found the related logs in the problematic cluster:

{"timestamp":"2024-08-01T01:56:48.965078236Z","level":"INFO","fields":{"message":"source paused on startup","is_shared":true,"is_uninitialized":true},"target":"risingwave_stream::executor::source::source_executor","spans":[{"actor_id":771877,"otel.name":"Actor 771877","name":"actor"},{"otel.name":"Source BC72500002712","name":"executor"}]}

@yezizp2012
Copy link
Member

It can be resolved by #18131 .

@yezizp2012 yezizp2012 closed this as not planned Won't fix, can't repro, duplicate, stale Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zwang28 zwang28

Labels
need-cherry-pick-release-2.0 priority/high type/bug Something isn't working
Projects
None yet
Milestone
release-2.0
Development

No branches or pull requests
3 participants
@fuyufjh @yezizp2012 @zwang28