-
Notifications
You must be signed in to change notification settings - Fork 332
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(parquet): introduce inverted index applier to reader #3130
feat(parquet): introduce inverted index applier to reader #3130
Conversation
Signed-off-by: Zhenchi <[email protected]>
Signed-off-by: Zhenchi <[email protected]>
Signed-off-by: Zhenchi <[email protected]>
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #3130 +/- ##
==========================================
- Coverage 85.48% 85.04% -0.45%
==========================================
Files 822 822
Lines 134403 134560 +157
==========================================
- Hits 114899 114431 -468
- Misses 19504 20129 +625 |
Signed-off-by: Zhenchi <[email protected]>
Signed-off-by: Zhenchi <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Co-authored-by: dennis zhuang <[email protected]>
Co-authored-by: dennis zhuang <[email protected]>
Signed-off-by: Zhenchi <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 🚀
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Signed-off-by: Zhenchi <[email protected]>
Signed-off-by: Zhenchi <[email protected]>
…x-sst-reader-intro
Signed-off-by: Zhenchi <[email protected]>
Signed-off-by: Zhenchi <[email protected]>
Signed-off-by: Zhenchi <[email protected]>
Signed-off-by: Zhenchi <[email protected]>
Signed-off-by: Zhenchi <[email protected]>
I hereby agree to the terms of the GreptimeDB CLA
What's changed and what's your intention?
Add the index applier in the Parquet reader to filter row groups:
inverted_index_available
property toSstInfo
andFileMeta
row_groups_to_read
method forParquetReaderBuilder
, which returns row groups that still need to be read after being filtered through the inverted index and min-max indexMoreover, once
inverted_index_available
becomes a property ofFileMeta
, it not only represents a single SST File but also includes the associated index files. Therefore, when handling deletions, they should be deleted together.Checklist
Refer to a related PR or issue link (optional)
#2705