feat: add sql migration script and related ci tests #16000
Conversation
yezizp2012
added
ci/skip-ci
ci/run-e2e-sql-migration-tests
and removed
type/feature
labels
| println!("system parameters migrated"); | ||
|
|
||
| // workers. | ||
| let workers = model::Worker::list(&meta_store).await?; |
There was a problem hiding this comment.
It's fine not to migrate Worker to sql store. Because when workers join again, the info is fulfilled correspondingly.
Same for HummockPinnedSnapshot, HummockPinnedVersion.
There was a problem hiding this comment.
Migrating workers is to avoid unnecessary migration or scaling. Especially when cn continues to join, it may trigger multiple scalings.
| } | ||
| println!("compaction task migrated"); | ||
|
|
||
| // hummock sequence |
There was a problem hiding this comment.
For ease of future maintenance, may make the init value global constant. Because the init value must be equal to that from src/meta/src/hummock/manager/sequence.rs.
e2e_test/sql_migration/prepare.slt
Outdated
There was a problem hiding this comment.
It appears to be hard to cover all metadata here. 😕 For example, the system parameters.
There was a problem hiding this comment.
Indeed 😕 . I haven't thought of a better way to cover all the metadata yet, so I can only add them on a case-by-case basis for now. I will add the system parameters as well.
src/ctl/Cargo.toml
Outdated
| risingwave_object_store = { workspace = true } | ||
| risingwave_pb = { workspace = true } | ||
| risingwave_rpc_client = { workspace = true } | ||
| risingwave_storage = { workspace = true } | ||
| risingwave_stream = { workspace = true } | ||
| sea-orm = { version = "0.12.14", features = [ |
There was a problem hiding this comment.
Can we make it a workspace dependency?
There was a problem hiding this comment.
May I ask how long we plan to keep both meta backends? Can we somehow enforce developers to update this file when introducing new meta data during this period (otherwise it can be really dangerous for upgraders)?
There was a problem hiding this comment.
May I ask how long we plan to keep both meta backends?
I think we gonna keep both of them for two major versions, until the cloud fully integrate sql backend.
🤔 If the developer introduce new meta backend without migration, the cluster will fail to start due to inconsistent cluster id.
There was a problem hiding this comment.
until the cloud fully integrate sql backend.
May I kindly ask that if there's an exact scheduling for integrating SQL meta backend into cloud in approximately 2 months? cc @arkbriar @fuyufjh Maintaining both backends will be a great pain for developers.
without migration, the cluster will fail to start due to inconsistent cluster id
This is true. I was referring to the case where some kind of newly introduced metadata is accidentally uncovered in the migration script.
There was a problem hiding this comment.
This is true. I was referring to the case where some kind of newly introduced metadata is accidentally uncovered in the migration script.
Indeed, 🥵 . Or for new requirements after 1.8, the etcd backend does not support and returns an error suggesting users to upgrade to the sql backend first. This way, there is no need to synchronize and maintain migration scripts.
There was a problem hiding this comment.
Or we can rely on backup and restore to cover this situation, do a backup from the etcd cluster, and then restore it to the SQL backend. This part of the functionality is being worked on by @zwang28. Before that, we can help users upgrade through this migration script. I think this's the best way to handle it.
There was a problem hiding this comment.
May I kindly ask that if there's an exact scheduling for integrating SQL meta backend into cloud in approximately 2 months? cc @arkbriar @fuyufjh Maintaining both backends will be a great pain for developers.
For now, no. However, it will be considered in the next quarter. I think first of all we need to support SQL backends. Migration is what should be considered afterwards.
Co-authored-by: August <[email protected]> Co-authored-by: Huangjw <[email protected]>
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Example:
Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.