Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

random ci failure: meta node panic at failed to execute barrier #17331

Closed
xxchan opened this issue Jun 19, 2024 · 2 comments
Closed

random ci failure: meta node panic at failed to execute barrier #17331

xxchan opened this issue Jun 19, 2024 · 2 comments
Labels
type/bug Something isn't working
Milestone
release-1.10

Comments

@xxchan
Copy link
Member

xxchan commented Jun 19, 2024

Describe the bug

https://buildkite.com/risingwavelabs/pull-request/builds/52235#01902e10-968f-4e9f-b681-cfc34c7cd9e9

Error message/log

2024-06-19T01:33:25.348024788Z  WARN risingwave_meta::barrier::rpc: get error from response stream node=WorkerNode { id: 1, r#type: ComputeNode, host: Some(HostAddress { host: "127.0.0.1", port: 5687 }), state: Running, parallel_units: [ParallelUnit { id: 0, worker_node_id: 1 }, ParallelUnit { id: 1, worker_node_id: 1 }, ParallelUnit { id: 2, worker_node_id: 1 }, ParallelUnit { id: 3, worker_node_id: 1 }], property: Some(Property { is_streaming: true, is_serving: false, is_unschedulable: false }), transactional_id: Some(0), resource: None, started_at: None } err=gRPC request to stream service failed: Internal error: failed to collect barrier for epoch [6653572543021056, 6653572755357696, 6653572860215296, 6653572876664832, 6653572893048832, 6653572909432832, 6653572925816832, 6653572942200832, 6653572958584832, 6653572975165440]: Actor 15538 exited unexpectedly: Executor error: Storage error: Hummock error: Foyer error: ObjectStore failed with IO error: Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=16000) and read_retry_attempts (current=6) under [storage.object_store.retry] in the config accordingly if needed.
2024-06-19T01:33:25.499532264Z  INFO risingwave_meta::hummock::manager::compaction: CancelTask operation for task_id [] has been sent to node with context_id 9
2024-06-19T01:33:26.499643274Z  INFO risingwave_meta::hummock::manager::compaction: CancelTask operation for task_id [] has been sent to node with context_id 9
2024-06-19T01:33:27.499507868Z  INFO risingwave_meta::hummock::manager::compaction: CancelTask operation for task_id [] has been sent to node with context_id 9
thread 'rw-main' panicked at src/meta/src/barrier/mod.rs:813:13:
failed to execute barrier: get error from control stream: worker node 1, gRPC request to stream service failed: Internal error: failed to collect barrier for epoch [6653572543021056, 6653572755357696, 6653572860215296, 6653572876664832, 6653572893048832, 6653572909432832, 6653572925816832, 6653572942200832, 6653572958584832, 6653572975165440]: Actor 15538 exited unexpectedly: Executor error: Storage error: Hummock error: Foyer error: ObjectStore failed with IO error: Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=16000) and read_retry_attempts (current=6) under [storage.object_store.retry] in the config accordingly if needed.; worker node 2, gRPC request to stream service failed: Internal error: failed to collect barrier for epoch [6653572543021056, 6653572755357696, 6653572860215296, 6653572876664832, 6653572893048832, 6653572909432832, 6653572925816832, 6653572942200832, 6653572958584832, 6653572975165440]: Actor 14803 exited unexpectedly: Executor error: Storage error: Hummock error: Foyer error: ObjectStore failed with IO error: Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=16000) and read_retry_attempts (current=6) under [storage.object_store.retry] in the config accordingly if needed.;
stack backtrace:
2024-06-19T01:33:28.499045286Z  INFO risingwave_meta::hummock::manager::compaction: CancelTask operation for task_id [] has been sent to node with context_id 9
2024-06-19T01:33:29.499119163Z  INFO risingwave_meta::hummock::manager::compaction: CancelTask operation for task_id [] has been sent to node with context_id 9
   0: rust_begin_unwind
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panicking.rs:72:14
   2: {async_fn#0}
             at ./src/meta/src/barrier/mod.rs:813:13
   3: {async_fn#0}
             at ./src/meta/src/barrier/mod.rs:683:54
   4: {async_block#0}
             at ./src/meta/src/barrier/mod.rs:497:46
   5: poll<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/future/future.rs:123:9
   6: poll<core::pin::Pin<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
   7: {closure#0}<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
   8: with_mut<tokio::runtime::task::core::Stage<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
   9: poll<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30
  10: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  11: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panic/unwind_safe.rs:272:9
  12: do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:552:40
  13: __rust_try
  14: try<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>>
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:516:19
  15: catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panic.rs:146:14
  16: poll_future<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  17: poll_inner<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  18: poll<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  19: poll<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<risingwave_meta::barrier::{impl#5}::start::{async_block_env#0}, alloc::alloc::Global>>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
  20: run<alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
  21: {closure#0}
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:576:18
  22: with_budget<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
  23: budget<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
  24: run_task
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
  25: run
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
  26: {closure#0}
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  27: set<tokio::runtime::scheduler::Context, tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}, ()>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
  28: {closure#0}<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26
  29: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}>, ()>
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/thread/local.rs:284:16
  30: with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}>, ()>
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/thread/local.rs:260:9
  31: set_scheduler<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:17
  32: {closure#0}
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  33: enter_runtime<tokio::runtime::scheduler::multi_thread::worker::run::{closure_env#0}, ()>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
  34: run
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  35: {closure#0}
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  36: poll<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}, ()>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
  37: poll<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  38: {closure#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  39: with_mut<tokio::runtime::task::core::Stage<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  40: poll<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30
  41: {closure#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  42: call_once<core::task::poll::Poll<()>, tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panic/unwind_safe.rs:272:9
  43: do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>>
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:552:40
  44: try<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>>
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:516:19
  45: catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>>
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panic.rs:146:14
  46: poll_future<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  47: poll_inner<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  48: poll<tracing::instrument::Instrumented<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, tokio::runtime::blocking::schedule::BlockingSchedule>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  49: run<tokio::runtime::blocking::schedule::BlockingSchedule>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
  50: run
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
  51: run
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
  52: {closure#0}
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response
@xxchan xxchan added the type/bug Something isn't working label Jun 19, 2024
@github-actions github-actions bot added this to the release-1.10 milestone Jun 19, 2024
@xxchan
Copy link
Member Author

xxchan commented Jun 19, 2024

risedev-logs/compute-node-5687.log has many errors

2024-06-19T01:33:22.331983491Z ERROR risingwave_stream::task::stream_manager: actor exit with error actor_id=15538 error=Executor error: Storage error: Hummock error: Foyer error: ObjectStore failed with IO error: Timeout error: Retry attempts exhausted for read. Please modify read_attempt_timeout_ms (current=16000) and read_retry_attempts (current=6) under [storage.object_store.retry] in the config accordingly if needed.

Backtrace:
   0: capture
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/thiserror-ext-0.1.2/src/backtrace.rs:30:18
   1: thiserror_ext::ptr::ErrorBox<T,B>::new
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/thiserror-ext-0.1.2/src/ptr.rs:40:33
   2: <risingwave_storage::hummock::error::HummockError as core::convert::From<E>>::from
             at ./src/storage/src/hummock/error.rs:22:45
   3: <T as core::convert::Into<U>>::into
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/convert/mod.rs:759:9
   4: risingwave_storage::hummock::error::HummockError::foyer_error
             at ./src/storage/src/hummock/error.rs:162:9
   5: core::ops::function::FnOnce::call_once
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/ops/function.rs:250:5
   6: core::result::Result<T,E>::map_err
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/result.rs:829:27
   7: {async_fn#0}
             at ./src/storage/src/hummock/block_cache.rs:103:33
   8: {async_fn#0}
             at ./src/storage/src/hummock/sstable_store.rs:518:57
   9: {async_fn#0}
             at ./src/storage/src/hummock/sstable/forward_sstable_iterator.rs:221:18
  10: {async_fn#0}
             at ./src/storage/src/hummock/sstable/forward_sstable_iterator.rs:287:48
  11: {async_fn#0}
             at ./src/storage/src/hummock/mod.rs:112:25
  12: {async_fn#0}
             at ./src/storage/src/hummock/store/version.rs:623:22
  13: {async_fn#0}
             at ./src/storage/src/hummock/store/local_hummock_storage.rs:133:14
  14: {async_fn#0}<risingwave_storage::hummock::store::local_hummock_storage::LocalHummockStorage>
             at ./src/storage/src/hummock/utils.rs:434:68
  15: {async_fn#0}
             at ./src/storage/src/hummock/store/local_hummock_storage.rs:422:26
  16: {async_fn#0}<risingwave_storage::hummock::store::local_hummock_storage::LocalHummockStorage, risingwave_storage::mem_table::MemtableLocalStateStore<risingwave_storage::memory::RangeKvStateStore<risingwave_storage::memory::sled::SledRangeKv>>>
             at ./src/storage/src/store_impl.rs:501:33
  17: {async_block#0}<risingwave_storage::store_impl::verify::VerifyStateStore<risingwave_storage::hummock::store::local_hummock_storage::LocalHummockStorage, risingwave_storage::mem_table::MemtableLocalStateStore<risingwave_storage::memory::RangeKvStateStore<risingwave_storage::memory::sled::SledRangeKv>>, ()>>
             at ./src/storage/src/store_impl.rs:1040:26
  18: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<usize, risingwave_storage::error::StorageError>> + core::marker::Send), alloc::alloc::Global>>
             at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/future/future.rs:123:9
  19: poll<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<usize, risingwave_storage::error::StorageError>> + core::marker::Send), alloc::alloc::Global>>, true>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/future.rs:113:39
  20: {async_fn#0}<risingwave_storage::monitor::monitored_store::MonitoredStateStore<alloc::boxed::Box<dyn risingwave_storage::store_impl::boxed_state_store::DynamicDispatchedStateStore, alloc::alloc::Global>>, risingwave_common::util::value_encoding::BasicSerde, false, risingwave_stream::common::table::watermark::WatermarkBufferByEpoch<300>, false>
             at ./src/stream/src/common/table/state_table.rs:1342:34
  21: poll<risingwave_stream::common::table::state_table::{impl#5}::seal_current_epoch::{async_fn_env#0}<risingwave_storage::monitor::monitored_store::MonitoredStateStore<alloc::boxed::Box<dyn risingwave_storage::store_impl::boxed_state_store::DynamicDispatchedStateStore, alloc::alloc::Global>>, risingwave_common::util::value_encoding::BasicSerde, false, risingwave_stream::common::table::watermark::WatermarkBufferByEpoch<300>, false>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  22: {async_fn#0}<risingwave_storage::monitor::monitored_store::MonitoredStateStore<alloc::boxed::Box<dyn risingwave_storage::store_impl::boxed_state_store::DynamicDispatchedStateStore, alloc::alloc::Global>>, risingwave_common::util::value_encoding::BasicSerde, false, risingwave_stream::common::table::watermark::WatermarkBufferByEpoch<300>, false>
             at ./src/stream/src/common/table/state_table.rs:1197:18
  23: {async_fn#0}<risingwave_storage::monitor::monitored_store::MonitoredStateStore<alloc::boxed::Box<dyn risingwave_storage::store_impl::boxed_state_store::DynamicDispatchedStateStore, alloc::alloc::Global>>, risingwave_common::util::value_encoding::BasicSerde, false, risingwave_stream::common::table::watermark::WatermarkBufferByEpoch<300>, false>
             at ./src/stream/src/common/table/state_table.rs:1141:44
  24: {async_fn#0}<risingwave_common::hash::key_v2::HashKeyImpl<risingwave_common::hash::key_v2::StackStorage<4>, risingwave_common::hash::key::StackNullBitmap>, risingwave_storage::monitor::monitored_store::MonitoredStateStore<alloc::boxed::Box<dyn risingwave_storage::store_impl::boxed_state_store::DynamicDispatchedStateStore, alloc::alloc::Global>>>
             at ./src/stream/src/executor/join/hash_join.rs:500:40
  25: {async_fn#0}<risingwave_common::hash::key_v2::HashKeyImpl<risingwave_common::hash::key_v2::StackStorage<4>, risingwave_common::hash::key::StackNullBitmap>, risingwave_storage::monitor::monitored_store::MonitoredStateStore<alloc::boxed::Box<dyn risingwave_storage::store_impl::boxed_state_store::DynamicDispatchedStateStore, alloc::alloc::Global>>, 0>
             at ./src/stream/src/executor/hash_join.rs:619:37
  26: {coroutine#0}<risingwave_common::hash::key_v2::HashKeyImpl<risingwave_common::hash::key_v2::StackStorage<4>, risingwave_common::hash::key::StackNullBitmap>, risingwave_storage::monitor::monitored_store::MonitoredStateStore<alloc::boxed::Box<dyn risingwave_storage::store_impl::boxed_state_store::DynamicDispatchedStateStore, alloc::alloc::Global>>, 0>
             at ./src/stream/src/executor/hash_join.rs:585:52
  27: poll_next<risingwave_stream::executor::hash_join::{impl#5}::into_stream::{coroutine_env#0}<risingwave_common::hash::key_v2::HashKeyImpl<risingwave_common::hash::key_v2::StackStorage<4>, risingwave_common::hash::key::StackNullBitmap>, risingwave_storage::monitor::monitored_store::MonitoredStateStore<alloc::boxed::Box<dyn risingwave_storage::store_impl::boxed_state_store::DynamicDispatchedStateStore, alloc::alloc::Global>>, 0>, risingwave_stream::executor::Message, risingwave_stream::executor::error::StreamExecutorError>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.9/src/lib.rs:506:33
  28: poll_next<alloc::boxed::Box<(dyn futures_core::stream::Stream<Item=core::result::Result<risingwave_stream::executor::Message, risingwave_stream::executor::error::StreamExecutorError>> + core::marker::Send), alloc::alloc::Global>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:120:9
  29: poll_next<&mut core::pin::Pin<alloc::boxed::Box<(dyn futures_core::stream::Stream<Item=core::result::Result<risingwave_stream::executor::Message, risingwave_stream::executor::error::StreamExecutorError>> + core::marker::Send), alloc::alloc::Global>>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:120:9
  30: poll_next_unpin<core::pin::Pin<&mut core::pin::Pin<alloc::boxed::Box<(dyn futures_core::stream::Stream<Item=core::result::Result<risingwave_stream::executor::Message, risingwave_stream::executor::error::StreamExecutorError>> + core::marker::Send), alloc::alloc::Global>>>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/mod.rs:1638:9
  31: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/next.rs:32:9
  32: poll<futures_util::stream::stream::next::Next<core::pin::Pin<&mut core::pin::Pin<alloc::boxed::Box<(dyn futures_core::stream::Stream<Item=core::result::Result<risingwave_stream::executor::Message, risingwave_stream::executor::error::StreamExecutorError>> + core::marker::Send), alloc::alloc::Global>>>>, false>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/future.rs:119:15
  33: {coroutine#0}<core::pin::Pin<alloc::boxed::Box<(dyn futures_core::stream::Stream<Item=core::result::Result<risingwave_stream::executor::Message, risingwave_stream::executor::error::StreamExecutorError>> + core::marker::Send), alloc::alloc::Global>>>
             at ./src/stream/src/executor/wrapper/trace.rs:116:10

@xxchan
Copy link
Member Author

xxchan commented Jul 2, 2024 •
edited
Loading

Meet this again
https://buildkite.com/risingwavelabs/pull-request/builds/53092#01907214-cdc8-42e3-99d2-3e6d07047704

Update: Oh actually compute node panicked by a bug!

So I guess the OP is also a compute note panic? But I roughly remember meta node won't panic directly 🤔

@xxchan xxchan closed this as not planned Won't fix, can't repro, duplicate, stale Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug Something isn't working
Projects
None yet
Milestone
release-1.10
Development

No branches or pull requests
1 participant
@xxchan