Mito engine panics while pruning row groups (index out of bounds) #2524
Assignees
Labels
C-bug
Category Bugs
Comments
|
We should rewrite greptimedb/src/table/src/predicate/stats.rs Lines 30 to 36 in 201acd1 I can open a PR to fix this first. @v0y4g3r is working on a new predicate pruner for our new SST format since tags filters actually don't work now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What type of bug is this?
Crash
What subsystems are affected?
Standalone mode, Datanode
What happened?
Now the region server can push down filters to the mito engine. However, when the engine applies the predicate to prune row groups, it panics due to using an incorrect index to access the parquet row group metadata.
The root cause is that we use the schema of the region to build the predicate, instead of the actual schema of the parquet file we prepare to scan.
greptimedb/src/mito2/src/read/scan_region.rs
Lines 175 to 179 in 201acd1
What operating system did you use?
Unrelated
Relevant log output and stack trace
2023-10-01T09:53:40.889814Z ERROR common_telemetry::panic_hook: panicked at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parquet-43.0.0/src/file/metadata.rs:295:22: index out of bounds: the len is 14 but the index is 20 backtrace= 0: common_telemetry::panic_hook::set_panic_hook::{{closure}} at /home/root/greptimedb/src/common/telemetry/src/panic_hook.rs:31:25 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/alloc/src/boxed.rs:2021:9 std::panicking::rust_panic_with_hook at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/std/src/panicking.rs:733:13 2: std::panicking::begin_panic_handler::{{closure}} at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/std/src/panicking.rs:621:13 3: std::sys_common::backtrace::__rust_end_short_backtrace at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/std/src/sys_common/backtrace.rs:170:18 4: rust_begin_unwind at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/std/src/panicking.rs:617:5 5: core::panicking::panic_fmt at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/panicking.rs:67:14 6: core::panicking::panic_bounds_check at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/panicking.rs:162:5 7: <usize as core::slice::index::SliceIndex<[T]>>::index at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/slice/index.rs:261:10 core::slice::index::<impl core::ops::index::Index<I> for [T]>::index at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/slice/index.rs:18:9 <alloc::vec::Vec<T,A> as core::ops::index::Index<I>>::index at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/alloc/src/vec/mod.rs:2675:9 parquet::file::metadata::RowGroupMetaData::column at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parquet-43.0.0/src/file/metadata.rs:295:22 8: <table::predicate::stats::RowGroupPruningStatistics as datafusion::physical_optimizer::pruning::PruningStatistics>::max_values::{{closure}} at /home/root/greptimedb/src/table/src/predicate/stats.rs:65:29 core::iter::adapters::map::map_fold::{{closure}} at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/iter/adapters/map.rs:84:28 <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::fold at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/slice/iter/macros.rs:232:27 <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/iter/adapters/map.rs:124:9 <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/iter/adapters/map.rs:124:9 9: core::iter::traits::iterator::Iterator::for_each at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/iter/traits/iterator.rs:856:9 alloc::vec::Vec<T,A>::extend_trusted at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/alloc/src/vec/mod.rs:2828:17 <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/alloc/src/vec/spec_extend.rs:26:9 <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/alloc/src/vec/spec_from_iter_nested.rs:62:9 <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/alloc/src/vec/spec_from_iter.rs:33:9 10: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/alloc/src/vec/mod.rs:2696:9 core::iter::traits::iterator::Iterator::collect at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/iter/traits/iterator.rs:1895:9 <table::predicate::stats::RowGroupPruningStatistics as datafusion::physical_optimizer::pruning::PruningStatistics>::max_values at /home/root/greptimedb/src/table/src/predicate/stats.rs:88:14 11: datafusion::physical_optimizer::pruning::build_statistics_record_batch at /home/root/.cargo/git/checkouts/arrow-datafusion-8f64698f94ffb3e6/c0b0fca/datafusion/core/src/physical_optimizer/pruning.rs:393:36 datafusion::physical_optimizer::pruning::PruningPredicate::prune at /home/root/.cargo/git/checkouts/arrow-datafusion-8f64698f94ffb3e6/c0b0fca/datafusion/core/src/physical_optimizer/pruning.rs:161:13 12: table::predicate::Predicate::prune_row_groups at /home/root/greptimedb/src/table/src/predicate.rs:103:27 13: mito2::sst::parquet::reader::ParquetReaderBuilder::init_stream::{{closure}} at /home/root/greptimedb/src/mito2/src/sst/parquet/reader.rs:165:37 mito2::sst::parquet::reader::ParquetReaderBuilder::build::{{closure}} at /home/root/greptimedb/src/mito2/src/sst/parquet/reader.rs:118:66 mito2::read::seq_scan::SeqScan::build_reader::{{closure}} at /home/root/greptimedb/src/mito2/src/read/seq_scan.rs:162:18 14: mito2::read::seq_scan::SeqScan::build_stream::{{closure}} at /home/root/greptimedb/src/mito2/src/read/seq_scan.rs:111:46 mito2::read::scan_region::Scanner::scan::{{closure}} at /home/root/greptimedb/src/mito2/src/read/scan_region.rs:43:63 <mito2::engine::MitoEngine as store_api::region_engine::RegionEngine>::handle_query::{{closure}} at /home/root/greptimedb/src/mito2/src/engine.rs:197:14 15: <core::pin::Pin<P> as core::future::future::Future>::poll at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/future/future.rs:125:9 <datanode::region_server::DummyTableProvider as datafusion::datasource::provider::TableProvider>::scan::{{closure}} at /home/root/greptimedb/src/datanode/src/region_server.rs:495:14 16: <core::pin::Pin<P> as core::future::future::Future>::poll at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/future/future.rs:125:9 datafusion::physical_planner::DefaultPhysicalPlanner::create_initial_plan::{{closure}} at /home/root/.cargo/git/checkouts/arrow-datafusion-8f64698f94ffb3e6/c0b0fca/datafusion/core/src/physical_planner.rs:531:89 17: <core::pin::Pin<P> as core::future::future::Future>::poll at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/future/future.rs:125:9 datafusion::physical_planner::DefaultPhysicalPlanner::create_initial_plan::{{closure}} at /home/root/.cargo/git/checkouts/arrow-datafusion-8f64698f94ffb3e6/c0b0fca/datafusion/core/src/physical_planner.rs:805:97 18: <core::pin::Pin<P> as core::future::future::Future>::poll at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/future/future.rs:125:9 datafusion::physical_planner::DefaultPhysicalPlanner::create_initial_plan::{{closure}} at /home/root/.cargo/git/checkouts/arrow-datafusion-8f64698f94ffb3e6/c0b0fca/datafusion/core/src/physical_planner.rs:753:85 19: <core::pin::Pin<P> as core::future::future::Future>::poll at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/future/future.rs:125:9 datafusion::physical_planner::DefaultPhysicalPlanner::create_initial_plan::{{closure}} at /home/root/.cargo/git/checkouts/arrow-datafusion-8f64698f94ffb3e6/c0b0fca/datafusion/core/src/physical_planner.rs:681:85 20: <core::pin::Pin<P> as core::future::future::Future>::poll at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/future/future.rs:125:9 <datafusion::physical_planner::DefaultPhysicalPlanner as datafusion::physical_planner::PhysicalPlanner>::create_physical_plan::{{closure}} at /home/root/.cargo/git/checkouts/arrow-datafusion-8f64698f94ffb3e6/c0b0fca/datafusion/core/src/physical_planner.rs:424:22 21: <core::pin::Pin<P> as core::future::future::Future>::poll at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/future/future.rs:125:9 <query::query_engine::state::DfQueryPlanner as datafusion::execution::context::QueryPlanner>::create_physical_plan::{{closure}} at /home/root/greptimedb/src/query/src/query_engine/state.rs:226:14 22: <core::pin::Pin<P> as core::future::future::Future>::poll at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/future/future.rs:125:9 datafusion::execution::context::SessionState::create_physical_plan::{{closure}} at /home/root/.cargo/git/checkouts/arrow-datafusion-8f64698f94ffb3e6/c0b0fca/datafusion/core/src/execution/context.rs:1870:14 <query::datafusion::DatafusionQueryEngine as query::physical_planner::PhysicalPlanner>::create_physical_plan::{{closure}} at /home/root/greptimedb/src/query/src/datafusion.rs:319:22 23: <core::pin::Pin<P> as core::future::future::Future>::poll at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/future/future.rs:125:9 query::datafusion::DatafusionQueryEngine::exec_query_plan::{{closure}} at /home/root/greptimedb/src/query/src/datafusion.rs:88:72 24: <query::datafusion::DatafusionQueryEngine as query::query_engine::QueryEngine>::execute::{{closure}} at /home/root/greptimedb/src/query/src/datafusion.rs:253:56 25: <core::pin::Pin<P> as core::future::future::Future>::poll at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/future/future.rs:125:9 datanode::region_server::RegionServerInner::handle_read::{{closure}} at /home/root/greptimedb/src/datanode/src/region_server.rs:310:14 datanode::region_server::RegionServer::handle_read::{{closure}} at /home/root/greptimedb/src/datanode/src/region_server.rs:97:41 <frontend::instance::standalone::RegionInvoker as common_meta::datanode_manager::Datanode>::handle_query::{{closure}} at /home/root/greptimedb/src/frontend/src/instance/standalone.rs:88:14 26: <core::pin::Pin<P> as core::future::future::Future>::poll at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/future/future.rs:125:9 frontend::instance::region_query::FrontendRegionQueryHandler::do_get_inner::{{closure}} at /home/root/greptimedb/src/frontend/src/instance/region_query.rs:78:14 <frontend::instance::region_query::FrontendRegionQueryHandler as query::region_query::RegionQueryHandler>::do_get::{{closure}} at /home/root/greptimedb/src/frontend/src/instance/region_query.rs:51:14 27: <core::pin::Pin<P> as core::future::future::Future>::poll at /rustc/f3623871cfa0763c95ebd6ceafaa6dc2e44ca68f/library/core/src/future/future.rs:125:9 query::dist_plan::merge_scan::MergeScanExec::to_stream::{{closure}} at /home/root/greptimedb/src/query/src/dist_plan/merge_scan.rs:172:22 <async_stream::async_stream::AsyncStream<T,U> as futures_core::stream::Stream>::poll_next at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-stream-0.3.5/src/async_stream.rs:56:13 28: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 <common_recordbatch::RecordBatchStreamAdaptor<S> as futures_core::stream::Stream>::poll_next at /home/root/greptimedb/src/common/recordbatch/src/lib.rs:238:9 29: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 <common_recordbatch::adapter::DfRecordBatchStreamAdapter as futures_core::stream::Stream>::poll_next at /home/root/greptimedb/src/common/recordbatch/src/adapter.rs:135:15 30: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 futures_util::stream::stream::StreamExt::poll_next_unpin at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9 <datafusion::physical_plan::projection::ProjectionStream as futures_core::stream::Stream>::poll_next at /home/root/.cargo/git/checkouts/arrow-datafusion-8f64698f94ffb3e6/c0b0fca/datafusion/core/src/physical_plan/projection.rs:404:31 31: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 <common_recordbatch::adapter::RecordBatchStreamAdapter as futures_core::stream::Stream>::poll_next at /home/root/greptimedb/src/common/recordbatch/src/adapter.rs:200:15 32: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9 <S as futures_core::stream::TryStream>::try_poll_next at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:196:9 <futures_util::stream::try_stream::try_collect::TryCollect<St,C> as core::future::future::Future>::poll at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/try_collect.rs:46:26 33: common_recordbatch::util::collect::{{closure}} at /home/root/greptimedb/src/common/recordbatch/src/util.rs:22:36 servers::http::JsonResponse::from_output::{{closure}} at /home/root/greptimedb/src/servers/src/http.rs:290:49 34: servers::http::handler::sql::{{closure}} at /home/root/greptimedb/src/servers/src/http/handler.rs:63:79 <F as axum::handler::Handler<(M,T1,T2,T3,T4),S,B>>::call::{{closure}} at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/axum-0.6.20/src/handler/mod.rs:248:53 panic.file="/home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parquet-43.0.0/src/file/metadata.rs" panic.line=295 panic.column=22 thread 'tokio-runtime-worker' panicked at /home/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parquet-43.0.0/src/file/metadata.rs:295:22: index out of bounds: the len is 14 but the index is 20 note: run with `RUST_BACKTRACE=1` environment variable to display a backtraceHow can we reproduce the bug?
Creates a table with several tags and selects it with a
WHEREclause.The text was updated successfully, but these errors were encountered: