Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Range query optim #3194

Open
fulmicoton opened this issue Apr 19, 2023 · 3 comments
Open

Range query optim #3194

fulmicoton opened this issue Apr 19, 2023 · 3 comments
Assignees
Labels
enhancement New feature or request

Comments

@fulmicoton
Copy link
Contributor

We have two ways to deal with range queries...

  • a DocSet that uses the scans the columnar and creates a buffer of the next valid docs. In its current version it is already quite sophisticated... The buffering window change size to try to adapt to different match ratio dynamically, and the buffering itself relies on SIMD. The multilinear codec probably hinders perf a lot however.
  • a filter at the collector level.

Experiment with the two solutions, and see if the filter solution outperforms the docset solution for most of the queries.
If it is the case, we can then work on the QueryAST (once #3148 has landed) to bubble up range queries and extract range queries as filters.

@trinity-1686a
Copy link
Contributor

trinity-1686a commented May 22, 2023

#3329 will add support for warming up a range of inverted index. This means there is a 3rd option, using classic tantivy RangeQuery over the inverted index. This is likely slower on large ranges, but likely faster on smaller ranges.

@PSeitz
Copy link
Contributor

PSeitz commented Nov 22, 2023

Related issue: quickwit-oss/tantivy#2266

@PSeitz
Copy link
Contributor

PSeitz commented Oct 28, 2024

Related issue: quickwit-oss/tantivy#2531

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants