feat(meta): initialization and modeling for sql-backend based meta store #12454
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
yezizp2012
force-pushed
the
feat/sql-backend-meta
branch
2 times, most recently
from
66a9b0d to
db605b8
Compare
| @@ -0,0 +1,51 @@ | |||
| // Copyright 2023 RisingWave Labs | |||
There was a problem hiding this comment.
Defined all models under model_v2 directory.
| pub mod worker_property; | ||
|
|
||
| #[derive(Clone, Debug, PartialEq, FromJsonQueryResult, Eq, Serialize, Deserialize, Default)] | ||
| pub struct I32Array(pub Vec<i32>); |
There was a problem hiding this comment.
Define I32Array struct which can be used in other models. Since only PostgreSQL supports array type, we will define the field as Json and use I32Array instead in the model.
| use sea_orm::entity::prelude::*; | ||
| #[derive(Clone, Debug, PartialEq, Eq, EnumIter, DeriveActiveEnum)] | ||
| #[sea_orm(rs_type = "String", db_type = "String(None)")] | ||
| pub enum WorkerType { |
There was a problem hiding this comment.
Define enum for model field and store as string in table.
|
|
||
| #[async_trait::async_trait] | ||
| impl MigrationTrait for Migration { | ||
| async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { |
There was a problem hiding this comment.
Initial version of sql backend, defined tables, foreign keys, indexes and some initial data.
There was a problem hiding this comment.
May I ask if this file was generated by sea-orm, or was hand-written?
There was a problem hiding this comment.
The migration file itself is generated by sea-orm, but the detailed migration logic is hand-written.
That sounds okay since we don't have to maintain this migration file anymore. 😄
There was a problem hiding this comment.
The migration file itself is generated by sea-orm, but the detailed migration logic is hand-written.
That sounds okay since we don't have to maintain this migration file anymore. 😄
Yes, we only need to add some new migration file to include any data correctness, table schema change etc in it, then the system will upgrade automatically. The file won't be too complicate for each version I believe.
| .boolean() | ||
| .not_null(), | ||
| ) | ||
| .foreign_key( |
There was a problem hiding this comment.
For SQLite, foreign key can only be defined along with the table.
| ) | ||
| .col(ColumnDef::new(Fragment::StreamNode).json().not_null()) | ||
| .col(ColumnDef::new(Fragment::VnodeMapping).json()) | ||
| .col(ColumnDef::new(Fragment::StateTableIds).json()) |
There was a problem hiding this comment.
For example, defines json type here and derived to array in the model of table fragment.
| } | ||
|
|
||
| #[derive(DeriveIden)] | ||
| enum Cluster { |
There was a problem hiding this comment.
Define each table as enum and use it in migration.
| @@ -352,12 +378,31 @@ pub async fn start_service_as_election_leader( | |||
| mut svc_shutdown_rx: WatchReceiver<()>, | |||
| ) -> MetaResult<()> { | |||
| tracing::info!("Defining leader services"); | |||
| if let Some(sql_store) = &meta_store_sql { | |||
There was a problem hiding this comment.
Check and do auto upgrade when meta starts. It will run any migration plans if found in this version. It will ignore if all plans are applied already.
Codecov Report
@@ Coverage Diff @@
## main #12454 +/- ##
==========================================
+ Coverage 69.37% 69.41% +0.03%
==========================================
Files 1440 1469 +29
Lines 238947 240650 +1703
==========================================
+ Hits 165762 167038 +1276
- Misses 73185 73612 +427
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 14 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
Will review later. 🥵 |
|
|
||
| #[async_trait::async_trait] | ||
| impl MigrationTrait for Migration { | ||
| async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { |
There was a problem hiding this comment.
May I ask if this file was generated by sea-orm, or was hand-written?
|
BTW, have we investigated |
diesel is non-async right now. Using |
I see.
|
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Related #12449 , this is the very first step of sql-backend based meta store. In this PR, we have modeled and initialized the existing entities except those related to hummock, and implemented some controller logic based on these models, such as
SystemParamsController,CatalogController, etc. For other models, you can refer to their implementation to gradually replace other managers.I added a README.md doc under
model_v2to introduce the steps of defining relations and changes of new model and generate of related files. Cc @zwang28, you can refer to this to model hummock metadata.Close #12475.
Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.