Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(mito): Implements compaction scheduler #2413

Merged
merged 14 commits into from
Sep 17, 2023

Conversation

evenyag
Copy link
Contributor

@evenyag evenyag commented Sep 16, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

This PR implements the compaction scheduler for mito2. The CompactionScheduler

  • tracks region that is compacting and its pending compaction requests
  • ensures only one compaction task is running for a region, and merge compaction request waiters
  • cancels compaction tasks and notifies waiters if compaction is failed or the region is dropped/closed/truncated

It adds some tests for compaction and flush schedulers. I'll add more tests in later PRs.

Other changes

  • schedules compaction after a flush job is finished
  • fixes an incorrect debug assertion in FlushScheduler

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

@evenyag evenyag force-pushed the feat/mito2-compact-schedule branch from 76e6b77 to b734dc8 Compare September 16, 2023 08:42
@evenyag evenyag marked this pull request as ready for review September 16, 2023 08:48
@codecov
Copy link

codecov bot commented Sep 16, 2023

Codecov Report

Merging #2413 (cb56ea8) into develop (627c5b7) will increase coverage by 0.08%.
Report is 3 commits behind head on develop.
The diff coverage is 82.54%.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2413      +/-   ##
===========================================
+ Coverage    77.60%   77.69%   +0.08%     
===========================================
  Files          718      721       +3     
  Lines       113634   114015     +381     
===========================================
+ Hits         88191    88586     +395     
+ Misses       25443    25429      -14     

src/mito2/src/compaction.rs Outdated Show resolved Hide resolved
src/mito2/src/compaction.rs Outdated Show resolved Hide resolved
src/mito2/src/compaction.rs Show resolved Hide resolved
@killme2008 killme2008 added this pull request to the merge queue Sep 17, 2023
Merged via the queue into GreptimeTeam:develop with commit 55ae5e5 Sep 17, 2023
11 checks passed
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
* feat: allow multiple waiters in compaction request

* feat: compaction status wip

* feat: track region status in compaction scheduler

* feat: impl compaction scheduler

* feat: call compaction scheduler

* feat: remove status if nothing to compact

* feat: schedule compaction after flush

* feat: set compacting to false after compaction finished

* refactor: flush status only needs region id and version control

* refactor: schedule_compaction don't need region as argument

* test: test flush/scheduler for empty requests

* test: trigger compaction in test

* feat: notify scheduler on truncated

* chore: Apply suggestions from code review

Co-authored-by: JeremyHi <[email protected]>

---------

Co-authored-by: JeremyHi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants