meta node panicked at src/common/src/hash/consistent_hash/mapping.rs:264:53: no entry found for key

[lmatz]

Describe the bug

Execution:
https://buildkite.com/risingwave-test/chaos-mesh/builds/880#019012fa-8ff7-42d4-a187-1dc2422113e6

Log:
https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?orgId=1&from=1718294400000&to=1718467199000&var-data_source=PE59595AED52CF917&var-namespace=longcmkf-20240613-190314&var-pod=benchmark-risingwave-meta-m-0&var-search=

CrashLoopBackOff

Error message/log

thread 'rw-main' panicked at src/common/src/hash/consistent_hash/mapping.rs:264:53:
no entry found for key
stack backtrace:
   0:     0xaaaae380958c - std::backtrace_rs::backtrace::libunwind::trace::h60f81c370ee2d74a
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/../../backtrace/src/backtrace/libunwind.rs:105:5
   1:     0xaaaae380958c - std::backtrace_rs::backtrace::trace_unsynchronized::hc781a479ffed43a0
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0xaaaae380958c - std::sys_common::backtrace::_print_fmt::hc2ba1099d7059606
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/sys_common/backtrace.rs:68:5
   3:     0xaaaae380958c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h5da766a0cb3364ef
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/sys_common/backtrace.rs:44:22
   4:     0xaaaad9b88a2c - core::fmt::rt::Argument::fmt::ha9a422623c322253
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/fmt/rt.rs:142:9
   5:     0xaaaad9b88a2c - core::fmt::write::h3b9f40ac86be61a9
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/fmt/mod.rs:1153:17
   6:     0xaaaae3802e20 - std::io::Write::write_fmt::hddd130a5e6d78272
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/io/mod.rs:1843:15
   7:     0xaaaae38093d8 - std::sys_common::backtrace::_print::h2960d4a17f76801e
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/sys_common/backtrace.rs:47:5
   8:     0xaaaae38093d8 - std::sys_common::backtrace::print::h1b460b0b5f16d80a
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/sys_common/backtrace.rs:34:9
   9:     0xaaaae380a9fc - std::panicking::default_hook::{{closure}}::h3a3c1f6291670090
  10:     0xaaaae380a6bc - std::panicking::default_hook::h0d2c66366a9ceb34
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:292:9
  11:     0xaaaae1604c74 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hbafce31672a75aa8
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/alloc/src/boxed.rs:2034:9
  12:     0xaaaae1604c74 - risingwave_rt::panic_hook::set_panic_hook::{{closure}}::h17ed77c377735ec3
                               at /risingwave/src/utils/runtime/src/panic_hook.rs:25:9
  13:     0xaaaae1604c74 - std::panicking::update_hook::{{closure}}::h714d37e4212867bd
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:234:47
  14:     0xaaaae380afb8 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h864b08d2d7a96ad7
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/alloc/src/boxed.rs:2034:9
  15:     0xaaaae380afb8 - std::panicking::rust_panic_with_hook::haa099cfae0290c03
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:783:13
  16:     0xaaaae380ad64 - std::panicking::begin_panic_handler::{{closure}}::h46a1eaac51f90de0
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:657:13
  17:     0xaaaae3809a50 - std::sys_common::backtrace::__rust_end_short_backtrace::hd376660cb47e77ef
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/sys_common/backtrace.rs:171:18
  18:     0xaaaae380ab04 - rust_begin_unwind
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:645:5
  19:     0xaaaad9b854e8 - core::panicking::panic_fmt::h88d9e2b8e8a7f41c
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panicking.rs:72:14
  20:     0xaaaad9b8562c - core::panicking::panic_display::hc46b2b656467eab9
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panicking.rs:197:5
  21:     0xaaaad9b85344 - core::panicking::panic_str::he5ae61780b42858a
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panicking.rs:172:5
  22:     0xaaaad9b85344 - core::option::expect_failed::ha9217015ccda0384
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/option.rs:1995:5
  23:     0xaaaadc78ddec - core::option::Option<T>::expect::h1f7f9ccccfcf7f92
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/option.rs:896:21
  24:     0xaaaadc78ddec - <std::collections::hash::map::HashMap<K,V,S> as core::ops::index::Index<&Q>>::index::h73d1dcf3112568b4
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/collections/hash/map.rs:1343:23
  25:     0xaaaadc78ddec - risingwave_common::hash::consistent_hash::mapping::VnodeMapping<T>::transform::{{closure}}::hc5b8cd26554f9f69
                               at /risingwave/src/common/src/hash/consistent_hash/mapping.rs:264:53
  26:     0xaaaadc78ddec - core::iter::adapters::map::map_fold::{{closure}}::h06fd3742ce9073da
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/iter/adapters/map.rs:89:28
  27:     0xaaaadc78ddec - <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::fold::he9153edc332769a1
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/slice/iter/macros.rs:232:27
  28:     0xaaaadc78ddec - <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold::hb2f2cdec058aebb7
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/iter/adapters/map.rs:129:9
  29:     0xaaaadc788260 - core::iter::traits::iterator::Iterator::for_each::he8a89eaca9b3931c
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/iter/traits/iterator.rs:817:9
  30:     0xaaaadc788260 - alloc::vec::Vec<T,A>::extend_trusted::hdc239ee4833a65d9
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/alloc/src/vec/mod.rs:3020:17
  31:     0xaaaadc788260 - <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend::h913b61a5a2a7bb33
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/alloc/src/vec/spec_extend.rs:26:9
  32:     0xaaaadc788260 - <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter::hbe18659f4c68153e
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/alloc/src/vec/spec_from_iter_nested.rs:62:9
  33:     0xaaaadc788260 - <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter::h86e14039bf6e667d
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/alloc/src/vec/spec_from_iter.rs:33:9
  34:     0xaaaadc7d5d38 - <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter::h3f9a71707558928a
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/alloc/src/vec/mod.rs:2894:9
  35:     0xaaaadc7d5d38 - core::iter::traits::iterator::Iterator::collect::h193d3412a607ce20
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/iter/traits/iterator.rs:2003:9
  36:     0xaaaadc7d5d38 - risingwave_common::hash::consistent_hash::mapping::VnodeMapping<T>::transform::he804f6bf1049dd33
                               at /risingwave/src/common/src/hash/consistent_hash/mapping.rs:264:61
  37:     0xaaaadc7d5d38 - risingwave_common::hash::consistent_hash::mapping::VnodeMapping<risingwave_common::hash::consistent_hash::mapping::marker::ParallelUnit>::to_worker_slot::h61a8f5a311d04d19
                               at /risingwave/src/common/src/hash/consistent_hash/mapping.rs:404:9
  38:     0xaaaae108bdf8 - risingwave_meta::controller::fragment::<impl risingwave_meta::controller::catalog::CatalogControllerInner>::all_running_fragment_mappings::{{closure}}::{{closure}}::hfa102c89b3dc0544
                               at /risingwave/src/meta/src/controller/fragment.rs:85:21
  39:     0xaaaae106ef60 - core::iter::adapters::map::map_fold::{{closure}}::h0b50ee8ae08eeb8d
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/iter/adapters/map.rs:89:28
  40:     0xaaaae106ef60 - core::iter::traits::iterator::Iterator::fold::h4062a2a36e7091c9
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/iter/traits/iterator.rs:2587:21
  41:     0xaaaae106ef60 - <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold::h2b022d90dcab19ed
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/iter/adapters/map.rs:129:9
  42:     0xaaaae0998f74 - core::iter::traits::iterator::Iterator::for_each::h1fd82d9fce418b59
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/iter/traits/iterator.rs:817:9
  43:     0xaaaae0998f74 - alloc::vec::Vec<T,A>::extend_trusted::hd1407fddea00698b
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/alloc/src/vec/mod.rs:3020:17
  44:     0xaaaae0998f74 - <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend::h8ae0804891fcbb3c
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/alloc/src/vec/spec_extend.rs:26:9
  45:     0xaaaae0998f74 - <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter::he077a48d14e477f9
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/alloc/src/vec/spec_from_iter_nested.rs:62:9
  46:     0xaaaae0998f74 - alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter::hc7cb9cdc5f0d6764
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/alloc/src/vec/in_place_collect.rs:237:20
  47:     0xaaaae0e38858 - <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter::h2bd573ca6036d7cb
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/alloc/src/vec/mod.rs:2894:9
  48:     0xaaaae0e38858 - core::iter::traits::iterator::Iterator::collect::h60acef546ac08d9c
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/iter/traits/iterator.rs:2003:9
  49:     0xaaaae0e38858 - itertools::Itertools::collect_vec::h6a47be11ed39904b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/itertools-0.12.1/src/lib.rs:2200:9
  50:     0xaaaae0e38858 - risingwave_meta_service::notification_service::NotificationServiceImpl::get_worker_slot_mapping_snapshot::{{closure}}::hc4a48426a5fb198f
                               at /risingwave/src/meta/service/src/notification_service.rs:158:46
  51:     0xaaaae0e38858 - risingwave_meta_service::notification_service::NotificationServiceImpl::frontend_subscribe::{{closure}}::h7f9bed2ae93715ce
                               at /risingwave/src/meta/service/src/notification_service.rs:251:53
  52:     0xaaaae0e61698 - <risingwave_meta_service::notification_service::NotificationServiceImpl as risingwave_pb::meta::notification_service_server::NotificationService>::subscribe::{{closure}}::hdf6e13fbb9ea4ad1
                               at /risingwave/src/meta/service/src/notification_service.rs:350:43
  53:     0xaaaae02178fc - <core::pin::Pin<P> as core::future::future::Future>::poll::hbe8d56020ac0cb84
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/future/future.rs:123:9
  54:     0xaaaae02178fc - <<risingwave_pb::meta::notification_service_server::NotificationServiceServer<T> as tower_service::Service<http::request::Request<B>>>::call::SubscribeSvc<T> as tonic::server::service::ServerStreamingService<risingwave_pb::meta::SubscribeRequest>>::call::{{closure}}::h7e68ecb8178a9f85
                               at /risingwave/src/prost/src/meta.rs:4862:88
  55:     0xaaaae01b73d0 - <core::pin::Pin<P> as core::future::future::Future>::poll::hbe8d56020ac0cb84
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/future/future.rs:123:9
  56:     0xaaaae01b73d0 - tonic::server::grpc::Grpc<T>::server_streaming::{{closure}}::hb60bf503d6afc4db
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tonic-0.10.2/src/server/grpc.rs:292:46
  57:     0xaaaae01b73d0 - <risingwave_pb::meta::notification_service_server::NotificationServiceServer<T> as tower_service::Service<http::request::Request<B>>>::call::{{closure}}::h827a1595205f5629
                               at /risingwave/src/prost/src/meta.rs:4885:70
  58:     0xaaaae03f1bf4 - <core::pin::Pin<P> as core::future::future::Future>::poll::h302bd73d8c0a2536
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/future/future.rs:123:9
  59:     0xaaaae03f1bf4 - <F as futures_core::future::TryFuture>::try_poll::h1fcfa6cac1448d75
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  60:     0xaaaae03f1bf4 - <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll::hfaf3635cbe1cbbd4
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34:9
  61:     0xaaaae03f1bf4 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h6b35dce4979b1794
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:55:37
  62:     0xaaaae03efc88 - <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll::h3e9f011c15fee5bd
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
  63:     0xaaaae03efc88 - <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll::h5002bc21bd0945df
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
  64:     0xaaaae03efc88 - <tower::util::map_response::MapResponseFuture<F,N> as core::future::future::Future>::poll::hf55d55105b1185b6
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-0.4.13/src/macros.rs:38:17
  65:     0xaaaae03efc88 - <F as futures_core::future::TryFuture>::try_poll::h56935d88bb7377a7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  66:     0xaaaae03efc88 - <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll::h98c5da6c2c40ba8a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34:9
  67:     0xaaaae03efc88 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h0462cd0de519f435
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:55:37
  68:     0xaaaae392ea98 - <core::pin::Pin<P> as core::future::future::Future>::poll::h1c2e341b5d848696
                               at /rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/future/future.rs:123:9
  69:     0xaaaae392ea98 - <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll::h9bbd9f3618c26ca9
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-0.4.13/src/util/oneshot.rs:97:38
  70:     0xaaaae39273c4 - <axum::routing::route::RouteFuture<B,E> as core::future::future::Future>::poll::h62d6e35e2cf58cb7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/axum-0.6.20/src/routing/route.rs:161:61
  71:     0xaaaae39273c4 - <tonic::transport::service::router::RoutesFuture as core::future::future::Future>::poll::hca328ee9426235f7
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tonic-0.10.2/src/transport/service/router.rs:134:22
  72:     0xaaaadff74d60 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hbb4e266de186a6d3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  73:     0xaaaae004d2a4 - <risingwave_common_service::tracing::TracingExtract<S> as tower_service::Service<http::request::Request<hyper::body::body::Body>>>::call::{{closure}}::h708519081f802f1c
                               at /risingwave/src/common/common_service/src/tracing.rs:86:46
  74:     0xaaaae004d2a4 - <risingwave_meta::rpc::intercept::MetricsMiddleware<S> as tower_service::Service<http::request::Request<hyper::body::body::Body>>>::call::{{closure}}::h0c6d6648069283c1
                               at /risingwave/src/meta/src/rpc/intercept.rs:82:44
  75:     0xaaaae004d2a4 - <tonic::transport::service::grpc_timeout::ResponseFuture<F> as core::future::future::Future>::poll::h7af9b82b3195567f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tonic-0.10.2/src/transport/service/grpc_timeout.rs:87:38

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

[lmatz]

let's add a dedicated test case for it together with the fix

[fuyufjh]

Panicked here: no entry found for key

risingwave/src/common/src/hash/consistent_hash/mapping.rs

Line 264 in 04cd1b5

data: self.data.iter().map(|item| to_map[item]).collect(),

Called by

risingwave/src/meta/src/controller/fragment.rs

Line 86 in 04cd1b5

.to_worker_slot(&parallel_unit_to_worker)

cc. @shanicky Can you please help to take an initial look?

[lmatz]

I label the issue as block-release-v1.10 because it happened on 6/14, before the time of the code freeze.

[huangjw806]

In the v1.10.0-rc.1 release test, the same issue occurred after migrating metastore from etcd to pg.

================================================================================
longevity-test Result
================================================================================
Result               FAIL                
Pipeline Message     [release test: v1.10.0-rc.1] [workflow triggered] metastore migrate test
Namespace            usrlngvty-20240618-031504
TestBed              medium-arm-3cn-all-affinity
RW Version           v1.10.0-rc.1        
Test Start time      2024-06-18 03:24:40 
Test End time        2024-06-18 05:26:28 
Test Queries         nexmark_q0,nexmark_q1,nexmark_q2,nexmark_q3,nexmark_q4,nexmark_q5,nexmark_q6_group_top1,nexmark_q7,nexmark_q8,nexmark_q9,nexmark_q10,nexmark_q12,nexmark_q14,nexmark_q15,nexmark_q16,nexmark_q17,nexmark_q18,nexmark_q19,nexmark_q20,nexmark_q21,nexmark_q22,nexmark_q101,nexmark_q102,nexmark_q103,nexmark_q104,nexmark_q105
Grafana Metric       https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=usrlngvty-20240618-031504&from=1718681080000&to=1718688388000
Grafana Logs         https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?orgId=1&var-data_source=Logging:%20test-useast1-eks-a&var-namespace=usrlngvty-20240618-031504&from=1718681080000&to=1718688388000
Memory Dumps         https://s3.console.aws.amazon.com/s3/buckets/test-useast1-mgmt-bucket-archiver?region=us-east-1&bucketType=general&prefix=k8s/usrlngvty-20240618-031504/&showversions=false
Buildkite Job        https://buildkite.com/risingwave-test/longevity-test/builds/1490

================================================================================
Restarted/Crashed Pods Details
 ================================================================================
Pod crashed/Restarted: benchmark-risingwave-compute-c-1 restart_count:2  phase:Running status:True
Pod crashed/Restarted: benchmark-risingwave-frontend-f-69575c5fbf-qz77x restart_count:1  phase:Running status:True
Pod crashed/Restarted: benchmark-risingwave-meta-m-0 restart_count:3  phase:Running status:True

meta log : https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?from=1718682033595&orgId=1&to=1718682090330&var-data_source=PE59595AED52CF917&var-namespace=usrlngvty-20240618-031504&var-pod=benchmark-risingwave-meta-m-0&var-search=