perf(cache): reduce exclusive lock usage while dropping in-memory cache entry

[MrCroxx]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

• Update foyer
  • Allow "get"/"entry drop" to acquire read lock or lock-free if the algorithm allows.
  • For LRU, optimize entry drop. Only the last reference requires the write lock of the shard.

FYI: foyer-rs/foyer#785

TODO:

• Switch to a released version of foyer.

Benchmark (nexmark):

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

[hzxa21]

The changes in RW are trivial. LGTM. Shall we wait for the foyer release before merging?

[MrCroxx]

The changes in RW are trivial. LGTM. Shall we wait for the foyer release before merging?

I agree that changes in this PR are trivial. However, there are several future changes that do nothing with this PR but rely on the changes this PR does.

I'd like to merge this PR first, then continue the others. And guarantee that the next RW release will rely on a released foyer version. Does it work for you? cc @hzxa21

[MrCroxx]

An assertion failed. Let me check.

thread '<unnamed>' panicked at /risingwave/.cargo/git/checkouts/foyer-e7ae575671bf01ff-shallow/430045c/foyer-memory/src/eviction/lru.rs:248:9:
assertion `left == right` failed
  left: 112
 right: 0
stack backtrace:
   0: rust_begin_unwind
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/panicking.rs:74:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/panicking.rs:367:5
   4: clear<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>
             at ./.cargo/git/checkouts/foyer-e7ae575671bf01ff-shallow/430045c/foyer-memory/src/eviction/lru.rs:248:9
   5: clear<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>
             at ./.cargo/git/checkouts/foyer-e7ae575671bf01ff-shallow/430045c/foyer-memory/src/raw.rs:239:9
   6: {closure#1}<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>
             at ./.cargo/git/checkouts/foyer-e7ae575671bf01ff-shallow/430045c/foyer-memory/src/raw.rs:372:35
   7: {closure#0}<lock_api::rwlock::RwLockWriteGuard<parking_lot::raw_rwlock::RawRwLock, foyer_memory::raw::RawCacheShard<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>, foyer_memory::raw::{impl#2}::clear::{closure_env#1}<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/iter/traits/iterator.rs:810:29
   8: {closure#0}<&lock_api::rwlock::RwLock<parking_lot::raw_rwlock::RawRwLock, foyer_memory::raw::RawCacheShard<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>, lock_api::rwlock::RwLockWriteGuard<parking_lot::raw_rwlock::RawRwLock, foyer_memory::raw::RawCacheShard<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>, (), foyer_memory::raw::{impl#2}::clear::{closure_env#0}<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>, core::iter::traits::iterator::Iterator::for_each::call::{closure_env#0}<lock_api::rwlock::RwLockWriteGuard<parking_lot::raw_rwlock::RawRwLock, foyer_memory::raw::RawCacheShard<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>, foyer_memory::raw::{impl#2}::clear::{closure_env#1}<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/iter/adapters/map.rs:88:21
   9: fold<lock_api::rwlock::RwLock<parking_lot::raw_rwlock::RawRwLock, foyer_memory::raw::RawCacheShard<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>, (), core::iter::adapters::map::map_fold::{closure_env#0}<&lock_api::rwlock::RwLock<parking_lot::raw_rwlock::RawRwLock, foyer_memory::raw::RawCacheShard<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>, lock_api::rwlock::RwLockWriteGuard<parking_lot::raw_rwlock::RawRwLock, foyer_memory::raw::RawCacheShard<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>, (), foyer_memory::raw::{impl#2}::clear::{closure_env#0}<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>, core::iter::traits::iterator::Iterator::for_each::call::{closure_env#0}<lock_api::rwlock::RwLockWriteGuard<parking_lot::raw_rwlock::RawRwLock, foyer_memory::raw::RawCacheShard<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>, foyer_memory::raw::{impl#2}::clear::{closure_env#1}<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/slice/iter/macros.rs:232:27
  10: fold<lock_api::rwlock::RwLockWriteGuard<parking_lot::raw_rwlock::RawRwLock, foyer_memory::raw::RawCacheShard<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>, core::slice::iter::Iter<lock_api::rwlock::RwLock<parking_lot::raw_rwlock::RawRwLock, foyer_memory::raw::RawCacheShard<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>>, foyer_memory::raw::{impl#2}::clear::{closure_env#0}<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>, (), core::iter::traits::iterator::Iterator::for_each::call::{closure_env#0}<lock_api::rwlock::RwLockWriteGuard<parking_lot::raw_rwlock::RawRwLock, foyer_memory::raw::RawCacheShard<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>, foyer_memory::raw::{impl#2}::clear::{closure_env#1}<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/iter/adapters/map.rs:128:9
  11: for_each<core::iter::adapters::map::Map<core::slice::iter::Iter<lock_api::rwlock::RwLock<parking_lot::raw_rwlock::RawRwLock, foyer_memory::raw::RawCacheShard<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>>, foyer_memory::raw::{impl#2}::clear::{closure_env#0}<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>, foyer_memory::raw::{impl#2}::clear::{closure_env#1}<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/iter/traits/iterator.rs:813:14
  12: clear<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>
             at ./.cargo/git/checkouts/foyer-e7ae575671bf01ff-shallow/430045c/foyer-memory/src/raw.rs:372:14
  13: drop<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>
             at ./.cargo/git/checkouts/foyer-e7ae575671bf01ff-shallow/430045c/foyer-memory/src/raw.rs:399:9
  14: drop_in_place<foyer_memory::raw::RawCacheInner<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  15: drop_slow<foyer_memory::raw::RawCacheInner<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>, alloc::alloc::Global>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/alloc/src/sync.rs:1877:18
  16: drop<foyer_memory::raw::RawCacheInner<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>, alloc::alloc::Global>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/alloc/src/sync.rs:2570:13
  17: drop_in_place<alloc::sync::Arc<foyer_memory::raw::RawCacheInner<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>, alloc::alloc::Global>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  18: drop_in_place<foyer_memory::raw::RawCache<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  19: drop_slow<foyer_memory::raw::RawCache<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>, ahash::random_state::RandomState, foyer_memory::indexer::hash_table::HashTableIndexer<foyer_memory::eviction::lru::Lru<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>>>>, alloc::alloc::Global>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/alloc/src/sync.rs:1877:18
  20: drop_in_place<foyer_storage::store::StoreInner<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, ahash::random_state::RandomState>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  21: drop_slow<foyer_storage::store::StoreInner<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, ahash::random_state::RandomState>, alloc::alloc::Global>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/alloc/src/sync.rs:1877:18
  22: drop<foyer_storage::store::StoreInner<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, ahash::random_state::RandomState>, alloc::alloc::Global>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/alloc/src/sync.rs:2570:13
  23: drop_in_place<alloc::sync::Arc<foyer_storage::store::StoreInner<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, ahash::random_state::RandomState>, alloc::alloc::Global>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  24: drop_in_place<foyer_storage::store::Store<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, ahash::random_state::RandomState>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  25: drop_in_place<foyer::hybrid::cache::HybridCache<risingwave_storage::hummock::sstable_store::SstableBlockIndex, alloc::boxed::Box<risingwave_storage::hummock::sstable::block::Block, alloc::alloc::Global>, ahash::random_state::RandomState>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  26: drop_in_place<risingwave_storage::hummock::sstable_store::SstableStore>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  27: drop_slow<risingwave_storage::hummock::sstable_store::SstableStore, alloc::alloc::Global>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/alloc/src/sync.rs:1877:18
  28: drop<risingwave_storage::hummock::sstable_store::SstableStore, alloc::alloc::Global>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/alloc/src/sync.rs:2570:13
  29: drop_in_place<alloc::sync::Arc<risingwave_storage::hummock::sstable_store::SstableStore, alloc::alloc::Global>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  30: drop_in_place<risingwave_storage::hummock::store::version::HummockVersionReader>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  31: drop_in_place<risingwave_storage::hummock::store::hummock_storage::HummockStorage>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  32: drop_in_place<risingwave_storage::store_impl::verify::VerifyStateStore<risingwave_storage::hummock::store::hummock_storage::HummockStorage, risingwave_storage::memory::RangeKvStateStore<risingwave_storage::memory::sled::SledRangeKv>, ()>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  33: drop_in_place<alloc::boxed::Box<dyn risingwave_storage::store_impl::boxed_state_store::DynamicDispatchedStateStore, alloc::alloc::Global>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  34: drop_in_place<alloc::boxed::Box<alloc::boxed::Box<dyn risingwave_storage::store_impl::boxed_state_store::DynamicDispatchedStateStore, alloc::alloc::Global>, alloc::alloc::Global>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  35: drop_in_place<risingwave_storage::monitor::monitored_store::MonitoredStateStore<alloc::boxed::Box<dyn risingwave_storage::store_impl::boxed_state_store::DynamicDispatchedStateStore, alloc::alloc::Global>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  36: drop_in_place<risingwave_storage::store_impl::StateStoreImpl>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  37: drop_in_place<risingwave_stream::task::env::StreamEnvironment>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  38: drop_in_place<risingwave_stream::task::barrier_manager::StreamActorManager>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  39: drop_slow<risingwave_stream::task::barrier_manager::StreamActorManager, alloc::alloc::Global>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/alloc/src/sync.rs:1877:18
  40: drop<risingwave_stream::task::barrier_manager::StreamActorManager, alloc::alloc::Global>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/alloc/src/sync.rs:2570:13
  41: drop_in_place<alloc::sync::Arc<risingwave_stream::task::barrier_manager::StreamActorManager, alloc::alloc::Global>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  42: drop_in_place<risingwave_stream::task::barrier_manager::LocalBarrierWorker>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  43: drop_in_place<madsim::sim::task::{impl#10}::spawn_inner::{closure#0}::{async_block_env#0}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/madsim-0.2.31/src/sim/task/mod.rs:645:32
  44: drop<madsim::sim::task::{impl#10}::spawn_inner::{closure#0}::{async_block_env#0}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/mem/manually_drop.rs:256:18
  45: drop<madsim::sim::task::{impl#10}::spawn_inner::{closure#0}::{async_block_env#0}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-task-4.4.0/src/runnable.rs:442:21
  46: drop_in_place<async_task::runnable::{impl#6}::spawn_local::Checked<madsim::sim::task::{impl#10}::spawn_inner::{closure#0}::{async_block_env#0}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  47: drop_in_place<alloc::boxed::Box<async_task::runnable::{impl#6}::spawn_local::Checked<madsim::sim::task::{impl#10}::spawn_inner::{closure#0}::{async_block_env#0}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>>, alloc::alloc::Global>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  48: drop_in_place<core::pin::Pin<alloc::boxed::Box<async_task::runnable::{impl#6}::spawn_local::Checked<madsim::sim::task::{impl#10}::spawn_inner::{closure#0}::{async_block_env#0}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>>, alloc::alloc::Global>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  49: drop_in_place<core::pin::Pin<alloc::boxed::Box<async_task::runnable::{impl#6}::spawn_local::Checked<madsim::sim::task::{impl#10}::spawn_inner::{closure#0}::{async_block_env#0}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>>, alloc::alloc::Global>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mut_ptr.rs:1442:18
  50: {closure#0}<core::pin::Pin<alloc::boxed::Box<async_task::runnable::{impl#6}::spawn_local::Checked<madsim::sim::task::{impl#10}::spawn_inner::{closure#0}::{async_block_env#0}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>>, alloc::alloc::Global>>, (), madsim::sim::task::{impl#10}::spawn_inner::{closure_env#1}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>, alloc::sync::Weak<madsim::sim::task::TaskInfo, alloc::alloc::Global>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-task-4.4.0/src/raw.rs:454:24
  51: abort_on_panic<(), async_task::raw::{impl#3}::drop_future::{closure_env#0}<core::pin::Pin<alloc::boxed::Box<async_task::runnable::{impl#6}::spawn_local::Checked<madsim::sim::task::{impl#10}::spawn_inner::{closure#0}::{async_block_env#0}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>>, alloc::alloc::Global>>, (), madsim::sim::task::{impl#10}::spawn_inner::{closure_env#1}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>, alloc::sync::Weak<madsim::sim::task::TaskInfo, alloc::alloc::Global>>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-task-4.4.0/src/utils.rs:34:13
  52: drop_future<core::pin::Pin<alloc::boxed::Box<async_task::runnable::{impl#6}::spawn_local::Checked<madsim::sim::task::{impl#10}::spawn_inner::{closure#0}::{async_block_env#0}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>>, alloc::alloc::Global>>, (), madsim::sim::task::{impl#10}::spawn_inner::{closure_env#1}<risingwave_stream::task::barrier_manager::{impl#2}::run::{async_fn_env#0}>, alloc::sync::Weak<madsim::sim::task::TaskInfo, alloc::alloc::Global>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-task-4.4.0/src/raw.rs:453:9
  53: drop<alloc::sync::Weak<madsim::sim::task::TaskInfo, alloc::alloc::Global>>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-task-4.4.0/src/runnable.rs:850:13
  54: drop_in_place<async_task::runnable::Runnable<alloc::sync::Weak<madsim::sim::task::TaskInfo, alloc::alloc::Global>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/ptr/mod.rs:574:1
  55: drop<async_task::runnable::Runnable<alloc::sync::Weak<madsim::sim::task::TaskInfo, alloc::alloc::Global>>>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/core/src/mem/mod.rs:942:24
  56: {closure#0}
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/madsim-0.2.31/src/sim/task/mod.rs:287:50
  57: do_call<madsim::sim::task::{impl#3}::run_all_ready::{closure_env#0}, ()>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/std/src/panicking.rs:557:40
  58: try<(), madsim::sim::task::{impl#3}::run_all_ready::{closure_env#0}>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/std/src/panicking.rs:520:19
  59: catch_unwind<madsim::sim::task::{impl#3}::run_all_ready::{closure_env#0}, ()>
             at /rustc/52fd9983996d9fcfb719749838336be66dee68f9/library/std/src/panic.rs:358:14
  60: run_all_ready
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/madsim-0.2.31/src/sim/task/mod.rs:287:17
  61: block_on<risingwave_simulation::main::{closure#0}::{async_block_env#0}>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/madsim-0.2.31/src/sim/task/mod.rs:240:13
  62: block_on<risingwave_simulation::main::{closure#0}::{async_block_env#0}>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/madsim-0.2.31/src/sim/runtime/mod.rs:129:9
  63: {closure#0}<risingwave_simulation::main::{closure#0}::{async_block_env#0}>
             at ./.cargo/registry/src/index.crates.io-6f17d22bba15001f/madsim-0.2.31/src/sim/runtime/builder.rs:141:35
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.