panic: pop_lru_by_epoch

[fuyufjh]

Describe the bug

https://buildkite.com/risingwave-test/longevity-kubebench/builds/655#018ad53b-c557-4092-b562-82c3d99b48c6

Error message/log

Actor 21: `nexmark_q20_1` [9098.259s]
  Epoch 5149930943873024 [1.030s]
    MaterializeExecutor 150000000D (actor 21, operator 13) [440.002ms]
      HashJoinExecutor 150000000C (actor 21, operator 12) [440.002ms]  <== current
[Detached 4]
  LocalInput (actor 27) [450.002ms]
[Detached 13]
  RemoteInput (actor 25) [450.002ms]



  21:     0x5588c8e04841 - core::option::Option<T>::unwrap::hf62df86808561e00
                               at /rustc/62ebe3a2b177d50ec664798d731b8a8d1a9120d1/library/core/src/option.rs:935:21
  22:     0x5588c8e04841 - lru::LruCache<K,V,S,A>::pop_lru_by_epoch::hbbddf220e5bf9371
                               at /root/.cargo/git/checkouts/lru-rs-b0d90f175c30fd68/cb2d7c7/src/lib.rs:836:62
  23:     0x5588c885f7ea - risingwave_stream::cache::managed_lru::ManagedLruCache<K,V,S,A>::evict_by_epoch::h147b718f9bc8d6ce
                               at /risingwave/src/stream/src/cache/managed_lru.rs:130:40
  24:     0x5588c812d577 - risingwave_stream::cache::managed_lru::ManagedLruCache<K,V,S,A>::evict::h4650285ef1577863
                               at /risingwave/src/stream/src/cache/managed_lru.rs:119:9
  25:     0x5588c812d577 - risingwave_stream::executor::managed_state::join::JoinHashMap<K,S>::evict::h6680a7212438cacf
                               at /risingwave/src/stream/src/executor/managed_state/join/mod.rs:605:20
  26:     0x5588c812d577 - risingwave_stream::executor::hash_join::HashJoinExecutor<K,S,_>::evict_cache::h17f22679767d18e3
                               at /risingwave/src/stream/src/executor/hash_join.rs:859:13
  27:     0x5588c812d577 - risingwave_stream::executor::hash_join::HashJoinExecutor<K,S,_>::eq_join_oneside::{{closure}}::h8f9c3b523a3900ab
                               at /risingwave/src/stream/src/executor/hash_join.rs:1044:13
  28:     0x5588c7d3078d - <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next::h15d56e57618c4128
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-async-stream-0.2.7/src/lib.rs:493:33
  29:     0x5588c7d3078d - risingwave_stream::executor::hash_join::HashJoinExecutor<K,S,_>::into_stream::{{closure}}::h774cb08588392509
                               at /risingwave/src/stream/src/executor/hash_join.rs:696:5

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

Today's date:2023-09-27
Result               FAIL                
Pipeline Message     manual trigger nexmark q17 q18 q19 q20 with 200000 throughput
TestBed              kubebench/3264g-medium-all-affinity
RW Version           nightly-20230926    
Test Start time      2023-09-27 06:07:46 
Test End time        2023-09-27 18:10:00 
Namespace            usrlngvty-20230927-060308
Queries              nexmark_q17,nexmark_q18,nexmark_q19,nexmark_q20
Grafana Metric       https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=usrlngvty-20230927-060308&from=1695794866000&to=1695838200000
Grafana Logs         https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?orgId=1&var-data_source=Logging:%20test-useast1-eks-a&var-namespace=usrlngvty-20230927-060308&from=1695794866000&to=1695838200000
Buildkite Job        https://buildkite.com/risingwave-test/longevity-kubebench/builds/655
Crash Container logs https://rw-qa-artifacts-public.s3.us-east-1.amazonaws.com/longevity/655_logs.txt
Report               https://rw-qa-artifacts-public.s3.us-east-1.amazonaws.com/longevity/655_report.txt