Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(mito): Implement skeleton for alteration #2343

Merged
merged 28 commits into from
Sep 11, 2023
Merged

feat(mito): Implement skeleton for alteration #2343

merged 28 commits into from
Sep 11, 2023

Conversation

evenyag
Copy link
Contributor

@evenyag evenyag commented Sep 6, 2023 •
edited
Loading

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

This PR implements a skeleton to alter a region. Steps to alter a region's schema are:

  1. Flushes all memtables. It ensures no data with old schema in WAL and memtables so we don't need to deal with these data during replay
  2. Writes the alteration to the manifest and applies it to the Version after the flush job is finished

Minor fixes

  • Removes flushed memtables after a flush job is done.
  • Compaction picker doesn't notify waiters if it returns None.

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

@evenyag evenyag changed the title feat: Implement skeleton for alteration feat(mito): Implement skeleton for alteration Sep 7, 2023
@evenyag evenyag mentioned this pull request Sep 8, 2023
Closed
40 tasks
@evenyag evenyag marked this pull request as ready for review September 8, 2023 06:40
@codecov
Copy link

codecov bot commented Sep 8, 2023 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (migrate-region-server@475d083). Click here to learn what that means.
The diff coverage is n/a.

Additional details and impacted files 
@@                   Coverage Diff                    @@
##             migrate-region-server    #2343   +/-   ##
========================================================
  Coverage                         ?   77.81%           
========================================================
  Files                            ?      724           
  Lines                            ?   115671           
  Branches                         ?        0           
========================================================
  Hits                             ?    90006           
  Misses                           ?    25665           
  Partials                         ?        0

@evenyag evenyag requested review from v0y4g3r and waynexia September 8, 2023 07:27
@evenyag evenyag marked this pull request as draft September 8, 2023 10:13
@evenyag evenyag marked this pull request as ready for review September 11, 2023 07:35
@evenyag evenyag requested a review from fengjiachun September 11, 2023 07:36
fengjiachun
fengjiachun approved these changes Sep 11, 2023
View reviewed changes
src/mito2/src/worker/handle_alter.rs Outdated Show resolved Hide resolved
fengjiachun
fengjiachun approved these changes Sep 11, 2023
View reviewed changes
Copy link
Collaborator

@fengjiachun fengjiachun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

killme2008
killme2008 reviewed Sep 11, 2023
View reviewed changes
src/mito2/src/compaction/twcs.rs Show resolved Hide resolved
src/mito2/src/worker/handle_alter.rs Show resolved Hide resolved
src/mito2/src/worker/handle_alter.rs Outdated Show resolved Hide resolved
src/store-api/src/region_request.rs Outdated Show resolved Hide resolved
src/store-api/src/region_request.rs Show resolved Hide resolved
killme2008
killme2008 approved these changes Sep 11, 2023
View reviewed changes
Copy link
Contributor

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@killme2008 killme2008 added this pull request to the merge queue Sep 11, 2023
Merged via the queue into GreptimeTeam:migrate-region-server with commit 7bb6ac3 Sep 11, 2023
waynexia pushed a commit that referenced this pull request Sep 12, 2023
@evenyag @waynexia
feat(mito): Implement skeleton for alteration (#2343)
6968fc0 
* feat: impl handle_alter wip

* refactor: move send_result to worker.rs

* feat: skeleton for handle_alter_request

* feat: write requests should wait for alteration

* feat: define alter request

* chore: no warnings

* fix: remove memtables after flush

* chore: update comments and impl add_write_request_to_pending

* feat: add schema version to RegionMetadata

* feat: impl alter_schema/can_alter_directly

* chore: use send_result

* test: pull next_batch again

* feat: convert pb AlterRequest to RegionAlterRequest

* feat: validate alter request

* feat: validate request and alter metadata

* feat: allow none location

* test: test alter

* fix: recover files and flushed entry id from manifest

* test: test alter

* chore: change comments and variables

* chore: fix compiler errors

* feat: add is_empty() to MemtableVersion

* test: fix metadata alter test

* fix: Compaction picker doesn't notify waiters if it returns None

* chore: address CR comments

* test: add tests for alter request

* refactor: use send_result
waynexia pushed a commit that referenced this pull request Sep 12, 2023
@evenyag @waynexia
feat(mito): Implement skeleton for alteration (#2343)
91c7f1f 
* feat: impl handle_alter wip

* refactor: move send_result to worker.rs

* feat: skeleton for handle_alter_request

* feat: write requests should wait for alteration

* feat: define alter request

* chore: no warnings

* fix: remove memtables after flush

* chore: update comments and impl add_write_request_to_pending

* feat: add schema version to RegionMetadata

* feat: impl alter_schema/can_alter_directly

* chore: use send_result

* test: pull next_batch again

* feat: convert pb AlterRequest to RegionAlterRequest

* feat: validate alter request

* feat: validate request and alter metadata

* feat: allow none location

* test: test alter

* fix: recover files and flushed entry id from manifest

* test: test alter

* chore: change comments and variables

* chore: fix compiler errors

* feat: add is_empty() to MemtableVersion

* test: fix metadata alter test

* fix: Compaction picker doesn't notify waiters if it returns None

* chore: address CR comments

* test: add tests for alter request

* refactor: use send_result
waynexia pushed a commit that referenced this pull request Sep 12, 2023
@evenyag @waynexia
feat(mito): Implement skeleton for alteration (#2343)
606ee43 
* feat: impl handle_alter wip

* refactor: move send_result to worker.rs

* feat: skeleton for handle_alter_request

* feat: write requests should wait for alteration

* feat: define alter request

* chore: no warnings

* fix: remove memtables after flush

* chore: update comments and impl add_write_request_to_pending

* feat: add schema version to RegionMetadata

* feat: impl alter_schema/can_alter_directly

* chore: use send_result

* test: pull next_batch again

* feat: convert pb AlterRequest to RegionAlterRequest

* feat: validate alter request

* feat: validate request and alter metadata

* feat: allow none location

* test: test alter

* fix: recover files and flushed entry id from manifest

* test: test alter

* chore: change comments and variables

* chore: fix compiler errors

* feat: add is_empty() to MemtableVersion

* test: fix metadata alter test

* fix: Compaction picker doesn't notify waiters if it returns None

* chore: address CR comments

* test: add tests for alter request

* refactor: use send_result
WenyXu pushed a commit to WenyXu/greptimedb that referenced this pull request Sep 13, 2023
@evenyag @WenyXu
feat(mito): Implement skeleton for alteration (GreptimeTeam#2343)
64c17d0 
* feat: impl handle_alter wip

* refactor: move send_result to worker.rs

* feat: skeleton for handle_alter_request

* feat: write requests should wait for alteration

* feat: define alter request

* chore: no warnings

* fix: remove memtables after flush

* chore: update comments and impl add_write_request_to_pending

* feat: add schema version to RegionMetadata

* feat: impl alter_schema/can_alter_directly

* chore: use send_result

* test: pull next_batch again

* feat: convert pb AlterRequest to RegionAlterRequest

* feat: validate alter request

* feat: validate request and alter metadata

* feat: allow none location

* test: test alter

* fix: recover files and flushed entry id from manifest

* test: test alter

* chore: change comments and variables

* chore: fix compiler errors

* feat: add is_empty() to MemtableVersion

* test: fix metadata alter test

* fix: Compaction picker doesn't notify waiters if it returns None

* chore: address CR comments

* test: add tests for alter request

* refactor: use send_result
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
@evenyag @paomian
feat(mito): Implement skeleton for alteration (GreptimeTeam#2343)
5bfc56a 
* feat: impl handle_alter wip

* refactor: move send_result to worker.rs

* feat: skeleton for handle_alter_request

* feat: write requests should wait for alteration

* feat: define alter request

* chore: no warnings

* fix: remove memtables after flush

* chore: update comments and impl add_write_request_to_pending

* feat: add schema version to RegionMetadata

* feat: impl alter_schema/can_alter_directly

* chore: use send_result

* test: pull next_batch again

* feat: convert pb AlterRequest to RegionAlterRequest

* feat: validate alter request

* feat: validate request and alter metadata

* feat: allow none location

* test: test alter

* fix: recover files and flushed entry id from manifest

* test: test alter

* chore: change comments and variables

* chore: fix compiler errors

* feat: add is_empty() to MemtableVersion

* test: fix metadata alter test

* fix: Compaction picker doesn't notify waiters if it returns None

* chore: address CR comments

* test: add tests for alter request

* refactor: use send_result
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fengjiachun fengjiachun fengjiachun approved these changes

@killme2008 killme2008 killme2008 approved these changes

@v0y4g3r v0y4g3r Awaiting requested review from v0y4g3r

@waynexia waynexia Awaiting requested review from waynexia

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@evenyag @killme2008 @fengjiachun