Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: sink: vnode should not be accessed by this table #14390

Closed
fuyufjh opened this issue Jan 5, 2024 · 4 comments
Closed

bug: sink: vnode should not be accessed by this table #14390

fuyufjh opened this issue Jan 5, 2024 · 4 comments
Assignees
@st1page @wenym1
Labels
no-issue-activity priority/high type/bug Something isn't working
Milestone
release-1.7

Comments

@fuyufjh
Copy link
Member

fuyufjh commented Jan 5, 2024 •
edited
Loading

Describe the bug

Compute node panicked when creating a sink:

thread 'risingwave-streaming-actor' panicked at src/storage/src/table/mod.rs:202:5:
vnode 71 should not be accessed by this table
...
  20:     0x56034b253b9e - risingwave_storage::table::check_vnode_is_set::h79c95ccbe513e56c
                               at /risingwave/src/storage/src/table/mod.rs:202:5
  21:     0x56034b253b9e - risingwave_storage::table::compute_vnode::hea68d8a62725e05b
                               at /risingwave/src/storage/src/table/mod.rs:161:9
  22:     0x56034b8d7649 - risingwave_stream::common::log_store_impl::kv_log_store::serde::LogStoreRowSerde::serialize_data_row::hfd6b42fe725362c7
                               at /risingwave/src/stream/src/common/log_store_impl/kv_log_store/serde.rs:221:21
  23:     0x56034beb1417 - <risingwave_stream::common::log_store_impl::kv_log_store::writer::KvLogStoreWriter<LS> as risingwave_connector::sink::log_store::LogWriter>::flush_current_epoch::{{closure}}::{{closure}}::h9940b8d797d41160
                               at /risingwave/src/stream/src/common/log_store_impl/kv_log_store/writer.rs:146:43
  24:     0x56034beb1417 - risingwave_stream::common::log_store_impl::kv_log_store::buffer::LogStoreBufferSender::flush_all_unflushed::h5ce2c4108928d11c
                               at /risingwave/src/stream/src/common/log_store_impl/kv_log_store/buffer.rs:285:17
  25:     0x56034c114e27 - <risingwave_stream::common::log_store_impl::kv_log_store::writer::KvLogStoreWriter<LS> as risingwave_connector::sink::log_store::LogWriter>::flush_current_epoch::{{closure}}::hbb68cf4a43601133
                               at /risingwave/src/stream/src/common/log_store_impl/kv_log_store/writer.rs:141:9
  26:     0x56034c114e27 - <risingwave_connector::sink::log_store::MonitoredLogWriter<W> as risingwave_connector::sink::log_store::LogWriter>::flush_current_epoch::{{closure}}::h65839010b697bfcf
                               at /risingwave/src/connector/src/sink/log_store.rs:294:14
  27:     0x56034c1212dc - risingwave_stream::executor::sink::SinkExecutor<F>::execute_write_log::{{closure}}::h8e5d0f524a51b5c8
                               at /risingwave/src/stream/src/executor/sink.rs:302:30

The distributed_key of log store table might incorrect. Still asking the definition from users, and I'll post it here then.

There are lots of vnode should not be accessed by this table logs printed, so it must be related to the plan instead of specific data.

Error message/log

full stack trace 
thread 'risingwave-streaming-actor' panicked at src/storage/src/table/mod.rs:202:5:
vnode 71 should not be accessed by this table
   0:     0x5603510e8da7 - std::backtrace_rs::backtrace::libunwind::trace::he8dc9e634c3d5b69
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x5603510e8da7 - std::backtrace_rs::backtrace::trace_unsynchronized::hcc1fc094cc2c1843
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x5603510e8da7 - std::sys_common::backtrace::_print_fmt::he6c8e9059e21630d
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x5603510e8da7 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h145361bce15f4a6a
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x5603486c8d00 - core::fmt::rt::Argument::fmt::hb890552c7d6e8c2b
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/fmt/rt.rs:142:9
   5:     0x5603486c8d00 - core::fmt::write::h97d71643815ece8a
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/fmt/mod.rs:1117:17
   6:     0x5603510e194f - std::io::Write::write_fmt::h9145d12d2fb312aa
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/io/mod.rs:1762:15
   7:     0x5603510e8b84 - std::sys_common::backtrace::_print::h34d8c7ca46068513
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x5603510e8b84 - std::sys_common::backtrace::print::h69de4357a88b4f9d
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x5603510ea247 - std::panicking::default_hook::{{closure}}::hbf7d8b588847f0d6
  10:     0x5603510e9fa5 - std::panicking::default_hook::h797e21049b5f2e1d
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:292:9
  11:     0x5603505553c7 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hce30f1612a4090cb
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/alloc/src/boxed.rs:2021:9
  12:     0x5603505553c7 - risingwave_rt::panic_hook::set_panic_hook::{{closure}}::h15beeff4c9efc3da
                               at /risingwave/src/utils/runtime/src/panic_hook.rs:25:9
  13:     0x5603505553c7 - std::panicking::update_hook::{{closure}}::h7ef834bab1b3b689
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:234:47
  14:     0x5603510ea89a - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h49df07f7cb25728d
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/alloc/src/boxed.rs:2021:9
  15:     0x5603510ea89a - std::panicking::rust_panic_with_hook::h4b1939882c095f9c
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:735:13
  16:     0x5603510ea61e - std::panicking::begin_panic_handler::{{closure}}::ha8983f1edca62350
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:609:13
  17:     0x5603510e9266 - std::sys_common::backtrace::__rust_end_short_backtrace::h14cba478a9ee1fbd
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/sys_common/backtrace.rs:170:18
  18:     0x5603510ea392 - rust_begin_unwind
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:597:5
  19:     0x5603486c5c65 - core::panicking::panic_fmt::h6433adef86579fb6
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/panicking.rs:72:14
  20:     0x56034b253b9e - risingwave_storage::table::check_vnode_is_set::h79c95ccbe513e56c
                               at /risingwave/src/storage/src/table/mod.rs:202:5
  21:     0x56034b253b9e - risingwave_storage::table::compute_vnode::hea68d8a62725e05b
                               at /risingwave/src/storage/src/table/mod.rs:161:9
  22:     0x56034b8d7649 - risingwave_stream::common::log_store_impl::kv_log_store::serde::LogStoreRowSerde::serialize_data_row::hfd6b42fe725362c7
                               at /risingwave/src/stream/src/common/log_store_impl/kv_log_store/serde.rs:221:21
  23:     0x56034beb1417 - <risingwave_stream::common::log_store_impl::kv_log_store::writer::KvLogStoreWriter<LS> as risingwave_connector::sink::log_store::LogWriter>::flush_current_epoch::{{closure}}::{{closure}}::h9940b8d797d41160
                               at /risingwave/src/stream/src/common/log_store_impl/kv_log_store/writer.rs:146:43
  24:     0x56034beb1417 - risingwave_stream::common::log_store_impl::kv_log_store::buffer::LogStoreBufferSender::flush_all_unflushed::h5ce2c4108928d11c
                               at /risingwave/src/stream/src/common/log_store_impl/kv_log_store/buffer.rs:285:17
  25:     0x56034c114e27 - <risingwave_stream::common::log_store_impl::kv_log_store::writer::KvLogStoreWriter<LS> as risingwave_connector::sink::log_store::LogWriter>::flush_current_epoch::{{closure}}::hbb68cf4a43601133
                               at /risingwave/src/stream/src/common/log_store_impl/kv_log_store/writer.rs:141:9
  26:     0x56034c114e27 - <risingwave_connector::sink::log_store::MonitoredLogWriter<W> as risingwave_connector::sink::log_store::LogWriter>::flush_current_epoch::{{closure}}::h65839010b697bfcf
                               at /risingwave/src/connector/src/sink/log_store.rs:294:14
  27:     0x56034c1212dc - risingwave_stream::executor::sink::SinkExecutor<F>::execute_write_log::{{closure}}::h8e5d0f524a51b5c8
                               at /risingwave/src/stream/src/executor/sink.rs:302:30
  28:     0x56034c0be3c2 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h5996442261db3779
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  29:     0x56034c0be3c2 - futures_util::stream::select_with_strategy::poll_side::h7c7e8adfb01d83dc
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/select_with_strategy.rs:220:28
  30:     0x56034c0be3c2 - futures_util::stream::select_with_strategy::poll_inner::h1d42f05d91cc820a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/select_with_strategy.rs:234:28
  31:     0x56034c0be3c2 - <futures_util::stream::select_with_strategy::SelectWithStrategy<St1,St2,Clos,State> as futures_core::stream::Stream>::poll_next::h038afc82b488c107
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/select_with_strategy.rs:270:17
  32:     0x56034c0be3c2 - <futures_util::stream::select::Select<St1,St2> as futures_core::stream::Stream>::poll_next::h72c5230cd76e89fa
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/select.rs:115:9
  33:     0x560350cc8271 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h6da07c5f3e6424ae
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  34:     0x560350cc8271 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::hdd913dc92c490252
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  35:     0x560350cc8271 - futures_util::stream::stream::StreamExt::poll_next_unpin::hf84fa2ae6a302303
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  36:     0x560350cc8271 - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::hf5d5bd9b152f3396
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  37:     0x560350cc8271 - <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll::heecf62acbef78ea8
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/future.rs:124:23
  38:     0x560350c032e9 - risingwave_stream::executor::wrapper::trace::instrument_await_tree::{{closure}}::h8a8a8fa15f7112f8
                               at /risingwave/src/stream/src/executor/wrapper/trace.rs:124:10
  39:     0x560350c032e9 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::ha548aae77dc11786
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  40:     0x560350c032e9 - risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}::hfe06fb8d023b656b
                               at /risingwave/src/stream/src/executor/wrapper/schema_check.rs:24:1
  41:     0x560350c032e9 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::hf8ef11231ec9d008
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  42:     0x560350c032e9 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h69db276e30dcbc9a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  43:     0x560350c032e9 - futures_util::stream::stream::StreamExt::poll_next_unpin::hf0f32221dde91796
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  44:     0x560350c01c1e - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::hb23179cb3509c3ba
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  45:     0x560350c01c1e - risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}::haec30b281c47c65e
                               at /risingwave/src/stream/src/executor/wrapper/epoch_check.rs:31:44
  46:     0x560350c01c1e - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h354bf978c16b143e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  47:     0x560350c01c1e - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h29d1e8012f9e9752
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  48:     0x560350c01c1e - <S as futures_core::stream::TryStream>::try_poll_next::he40106a544888c62
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9
  49:     0x560350c01c1e - futures_util::stream::try_stream::TryStreamExt::try_poll_next_unpin::hd437187b71d371ee
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/mod.rs:1068:9
  50:     0x560350c02869 - <futures_util::stream::try_stream::try_next::TryNext<St> as core::future::future::Future>::poll::ha31460fa31462cc3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/try_next.rs:32:9
  51:     0x560350c02869 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}::h7a16c0448bb4c016
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:347:31
  52:     0x560350c02869 - tokio::task::task_local::LocalKey<T>::scope_inner::h0e7f4dee7faf8de0
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:217:19
  53:     0x560350c02869 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::ha6c6ebb763b5726d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:343:19
  54:     0x560350c02869 - risingwave_common::util::epoch::task_local::scope::{{closure}}::h4f0a7e0217008098
                               at /risingwave/src/common/src/util/epoch.rs:210:47
  55:     0x560350c02869 - risingwave_stream::executor::wrapper::epoch_provide::epoch_provide::{{closure}}::h8a202b33c9287e0b
                               at /risingwave/src/stream/src/executor/wrapper/epoch_provide.rs:31:59
  56:     0x560350c02869 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::hff5e59fe9177048a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  57:     0x560350c02869 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h3513f8968eade55f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  58:     0x560350c02869 - futures_util::stream::stream::StreamExt::poll_next_unpin::hcf05222418f9b457
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  59:     0x560350ba6e1c - <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::h1bda0e9f70688ef7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  60:     0x560350ba6e1c - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h826c964dbf3f39ed
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  61:     0x560350c552be - risingwave_stream::executor::wrapper::trace::trace::{{closure}}::he718aad107a74196
                               at /risingwave/src/stream/src/executor/wrapper/trace.rs:50:69
  62:     0x560350c552be - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h17d04226b241e2ce
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  63:     0x560350c0cb81 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h6da07c5f3e6424ae
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  64:     0x56034bdebfb3 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h42c9ba99634e61ea
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  65:     0x56034bdebfb3 - <risingwave_stream::executor::dispatch::DispatchExecutor as risingwave_stream::executor::StreamConsumer>::execute::{{closure}}::hdf4db51106a85e7d
                               at /risingwave/src/stream/src/executor/dispatch.rs:353:9
  66:     0x56034bdebfb3 - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h70d6ab6e5e64d8cc
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  67:     0x56034bdebfb3 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h989959c9cc88f5b1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  68:     0x56034bdebfb3 - <&mut S as futures_core::stream::Stream>::poll_next::hb425411486fff037
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:104:9
  69:     0x56034bf971aa - <tokio_stream::stream_ext::next::Next<St> as core::future::future::Future>::poll::hef5ae8a4be1a6627
                               at /root/.cargo/git/checkouts/tokio-968c02b7a1a41bea/fe39bb8/tokio-stream/src/stream_ext/next.rs:42:9
  70:     0x56034bf971aa - <tokio_stream::stream_ext::try_next::TryNext<St> as core::future::future::Future>::poll::h6485c29d7fd2e448
                               at /root/.cargo/git/checkouts/tokio-968c02b7a1a41bea/fe39bb8/tokio-stream/src/stream_ext/try_next.rs:43:9
  71:     0x56034bf971aa - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h5385881ea326fc6b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  72:     0x56034be36bc2 - <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll::ha8e6fd2ea12a017d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/future.rs:124:23
  73:     0x56034b4a4a4b - risingwave_stream::executor::actor::Actor<C>::run_consumer::{{closure}}::h865cc72845d5264c
                               at /risingwave/src/stream/src/executor/actor.rs:196:18
  74:     0x56034b4a4a4b - <tokio::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll::h235a78ac3786c9d3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/future/maybe_done.rs:68:48
  75:     0x56034b77bf7d - risingwave_stream::executor::actor::Actor<C>::run::{{closure}}::{{closure}}::h424b2cb59008a393
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/macros/join.rs:126:24
  76:     0x56034b77bf7d - <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll::h01618562ea321a80
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/future/poll_fn.rs:58:9
  77:     0x56034b395af6 - risingwave_stream::executor::actor::Actor<C>::run::{{closure}}::he46ddfd1f94853df
                               at /risingwave/src/stream/src/executor/actor.rs:155:9
  78:     0x56034b395af6 - risingwave_stream::task::stream_manager::LocalStreamManagerCore::build_actors::{{closure}}::{{closure}}::he06559e597ea6cba
                               at /risingwave/src/stream/src/task/stream_manager.rs:698:51
  79:     0x56034b395af6 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}::hab4e83e376733a39
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:347:31
  80:     0x56034b395af6 - tokio::task::task_local::LocalKey<T>::scope_inner::hb8006ee80eafc9e1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:217:19
  81:     0x56034b395af6 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::h76a4098b154b37a5
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/task/task_local.rs:343:19
  82:     0x56034c0a58c2 - await_tree::registry::TreeRoot::instrument::{{closure}}::h2e06930ce6190bdb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.1.1/src/registry.rs:51:45
  83:     0x56034c0a58c2 - <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll::h4a5a46b1ce581170
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/future/either.rs:109:32
  84:     0x56034c0a58c2 - core::ops::function::FnOnce::call_once::hc7d591263801a760
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/ops/function.rs:250:5
  85:     0x56034c0a58c2 - tokio_metrics::task::instrument_poll::h93a13bdbced8a792
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-metrics-0.3.0/src/task.rs:2530:15
  86:     0x56034c0a58c2 - <tokio_metrics::task::Instrumented<T> as core::future::future::Future>::poll::h456c7e93b1825937
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-metrics-0.3.0/src/task.rs:2430:9
  87:     0x56034bfb90cb - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hc48d23e19ea1cd98
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  88:     0x56034bb2f38d - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h84b4952878639d05
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:334:17
  89:     0x56034bb2f38d - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h7cea07a5a407c4f1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/loom/std/unsafe_cell.rs:16:9
  90:     0x56034bb2f38d - tokio::runtime::task::core::Core<T,S>::poll::h23ff61dc96984c52
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:323:13
  91:     0x56034baeef8a - tokio::runtime::task::harness::poll_future::{{closure}}::h752204cb27e04ac1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:485:19
  92:     0x56034baeef8a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h9c2a186006b4d3a2
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/panic/unwind_safe.rs:271:9
  93:     0x56034baeef8a - std::panicking::try::do_call::hce260b8e6411ce44
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:504:40
  94:     0x56034baeef8a - std::panicking::try::h5b7081a222aa0d2a
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:468:19
  95:     0x56034baeef8a - std::panic::catch_unwind::h4a22f9e6327331e2
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panic.rs:142:14
  96:     0x56034baeef8a - tokio::runtime::task::harness::poll_future::hfea95c4386a2b829
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:473:18
  97:     0x56034baeef8a - tokio::runtime::task::harness::Harness<T,S>::poll_inner::hd302f36f2f994868
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:208:27
  98:     0x56034baeef8a - tokio::runtime::task::harness::Harness<T,S>::poll::hbaf30996769f43d7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:153:15
  99:     0x56034baeef8a - tokio::runtime::task::raw::poll::h2528f6d60d0449e5
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/raw.rs:276:5
 100:     0x5603511697eb - tokio::runtime::task::raw::RawTask::poll::h3411e2d2094a8398
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/raw.rs:200:18
 101:     0x5603511697eb - tokio::runtime::task::LocalNotified<S>::run::h1b234fe523436f65
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/mod.rs:400:9
 102:     0x5603511697eb - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h43544b88b7324f96
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:639:22
 103:     0x5603511697eb - tokio::runtime::coop::with_budget::hc2e1ba91dc4343ba
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/coop.rs:107:5
 104:     0x5603511697eb - tokio::runtime::coop::budget::h6914ce9705ab5185
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/coop.rs:73:5
 105:     0x5603511697eb - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::hf2150a2ed144755e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
 106:     0x560351163f24 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h5a0f74052acdf7aa
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
 107:     0x560351163f24 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::h8e32ba4f9377cbeb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
 108:     0x560351163f24 - tokio::runtime::context::scoped::Scoped<T>::set::h50d293442ec3c30f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context/scoped.rs:40:9
 109:     0x5603511605dd - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h9ec63a91dd7464d8
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
 110:     0x5603511605dd - tokio::runtime::context::runtime::enter_runtime::h79f151d8e5b7dcfe
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/context/runtime.rs:65:16
 111:     0x5603511694cc - tokio::runtime::scheduler::multi_thread::worker::run::h94f458d6e31ca12f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
 112:     0x560351160a43 - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h8ad055d17f7134e5
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
 113:     0x560351160a43 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h32b7aafb106ee064
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/task.rs:42:21
 114:     0x56035117bc6b - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h3e192d8e607298d2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
 115:     0x560351195831 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h5010035d46013761
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:334:17
 116:     0x560351195831 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h607b1db4fa129b9f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/loom/std/unsafe_cell.rs:16:9
 117:     0x560351195831 - tokio::runtime::task::core::Core<T,S>::poll::ha745bcf27d80ea23
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/core.rs:323:13
 118:     0x56035117666a - tokio::runtime::task::harness::poll_future::{{closure}}::h63a6ace95d72ebe3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:485:19
 119:     0x56035117666a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h3b36e779ade21bc0
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/panic/unwind_safe.rs:271:9
 120:     0x56035117666a - std::panicking::try::do_call::hfc1abc43f6d847ad
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:504:40
 121:     0x56035117666a - std::panicking::try::hb1c8d83ed9254c57
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:468:19
 122:     0x56035117666a - std::panic::catch_unwind::h7f826221717df4ee
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panic.rs:142:14
 123:     0x56035117666a - tokio::runtime::task::harness::poll_future::hb9e09854f0339be2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:473:18
 124:     0x56035117666a - tokio::runtime::task::harness::Harness<T,S>::poll_inner::ha3a343d2b9313af0
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:208:27
 125:     0x56035117666a - tokio::runtime::task::harness::Harness<T,S>::poll::h2e140b9d87d8ac8e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/harness.rs:153:15
 126:     0x56035118a89e - tokio::runtime::task::raw::RawTask::poll::h3411e2d2094a8398
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/raw.rs:200:18
 127:     0x56035118a89e - tokio::runtime::task::UnownedTask<S>::run::h572704cc4b2af822
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/task/mod.rs:437:9
 128:     0x56035118a89e - tokio::runtime::blocking::pool::Task::run::h0685e9615187ae8e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/pool.rs:159:9
 129:     0x56035118a89e - tokio::runtime::blocking::pool::Inner::run::h602aedddc3052c3c
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/pool.rs:513:17
 130:     0x56035119a060 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::hc8d0833541aabfeb
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.32.0/src/runtime/blocking/pool.rs:471:13
 131:     0x56035119a060 - std::sys_common::backtrace::__rust_begin_short_backtrace::h5de4298adbaf4bd4
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/sys_common/backtrace.rs:154:18
 132:     0x56035118bf5d - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h8faefb8db5ab8109
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/thread/mod.rs:529:17
 133:     0x56035118bf5d - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h6f640689fdeb9a50
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/panic/unwind_safe.rs:271:9
 134:     0x56035118bf5d - std::panicking::try::do_call::h1cdc9b3cc1caa67b
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:504:40
 135:     0x56035118bf5d - std::panicking::try::h03c21a59e4dbd614
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:468:19
 136:     0x56035118bf5d - std::panic::catch_unwind::h41252732620c3f23
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panic.rs:142:14
 137:     0x56035118bf5d - std::thread::Builder::spawn_unchecked_::{{closure}}::hc4c4b0474cd10277
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/thread/mod.rs:528:30
 138:     0x56035118bf5d - core::ops::function::FnOnce::call_once{{vtable.shim}}::h2fb3b88c7929fd95
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/ops/function.rs:250:5
 139:     0x5603510f0a25 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hac2e24a72c9237b1
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/alloc/src/boxed.rs:2007:9
 140:     0x5603510f0a25 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h0a4ea247e7a2feab
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/alloc/src/boxed.rs:2007:9
 141:     0x5603510f0a25 - std::sys::unix::thread::Thread::new::thread_start::ha157bf15852ae2a4
                               at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/sys/unix/thread.rs:108:17

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

1.5.3

Additional context

No response
@fuyufjh fuyufjh added the type/bug Something isn't working label Jan 5, 2024
@github-actions github-actions bot added this to the release-1.7 milestone Jan 5, 2024
@fuyufjh fuyufjh assigned st1page and wenym1 and unassigned st1page Jan 5, 2024
@fuyufjh
Copy link
Member Author

fuyufjh commented Jan 12, 2024

Update (2024-Jan-10): We failed to reproduce it.

However, with #14395, we can obtain more information at next time. Hopefully it can help with the debug.

@github-actions GitHub Actions
Copy link
Contributor

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.

@hzxa21
Copy link
Collaborator

hzxa21 commented Oct 8, 2024

@wenym1 Is the issue fixed?

@wenym1
Copy link
Contributor

wenym1 commented Oct 8, 2024

The issue cannot be reproduced and haven't happened since long time ago. May reopen the issue when it happens again.

@wenym1 wenym1 closed this as not planned Won't fix, can't repro, duplicate, stale Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@st1page st1page

@wenym1 wenym1

Labels
no-issue-activity priority/high type/bug Something isn't working
Projects
None yet
Milestone
release-1.7
Development

No branches or pull requests
4 participants
@hzxa21 @fuyufjh @st1page @wenym1