Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: optimize compaction strategy #15628

Open
4 tasks
Little-Wallace opened this issue Mar 12, 2024 · 1 comment
Open
4 tasks

feat: optimize compaction strategy #15628

Little-Wallace opened this issue Mar 12, 2024 · 1 comment
Assignees
Milestone

Comments

@Little-Wallace
Copy link
Contributor

Is your feature request related to a problem? Please describe.

When user try to create a big MView, it will keep generate a lot of data to RisingWave.
If our system does not compact these data to bottom level, it may cause a back-pressure and let create job stop.

Describe the solution you'd like

There are many problems waiting us to solve:

  • Meta node does not split state-table which belongs to a creating MV and it means that meta-node would not partition this state-table and it makes our compaction efficiency inefficient.
  • High throughput write will also cause a large overlapping level and compactor must read a lot of files concurrently for a large overlapping level.
  • A large task to base level may write too many small files and some object storage may not supply a high performance for small object writing.
  • According to write-amplification calculate algorithm, we shall pick a compact task to base level as large as possible to reduce write-amplification when there are so much data waiting in level 0 to be compacted.

Describe alternatives you've considered

No response

Additional context

No response

@github-actions github-actions bot added this to the release-1.8 milestone Mar 12, 2024
@Little-Wallace Little-Wallace changed the title feat: optimize compaction to speed up creating materialize view feat: optimize compaction strategy Mar 15, 2024
@Little-Wallace
Copy link
Contributor Author

optimize each state-table in one group as separate LSM tree so that we can calculate compaction score for each of them.
: #16919

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants