Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(connector): add proper prefix for aws related properties for connector #16671

Merged
merged 2 commits into from
May 14, 2024
Merged

refactor(connector): add proper prefix for aws related properties for connector #16671

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5d7d4a1
rename
yuhao-su May 9, 2024
6c3008f
fix
yuhao-su May 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion src/connector/src/connector_common/common.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,16 +61,28 @@ use aws_types::SdkConfig;
/// A flatten config map for aws auth.
#[derive(Deserialize, Debug, Clone, WithOptions)]
pub struct AwsAuthProps {
#[serde(rename = "aws.region", alias = "region")]
pub region: Option<String>,
#[serde(alias = "endpoint_url")]

#[serde(
rename = "aws.endpoint_url",
alias = "endpoint_url",
alias = "endpoint"
)]
pub endpoint: Option<String>,
#[serde(rename = "aws.credentials.access_key_id", alias = "access_key")]
pub access_key: Option<String>,
#[serde(rename = "aws.credentials.secret_access_key", alias = "secret_key")]
pub secret_key: Option<String>,
#[serde(rename = "aws.credentials.session_token", alias = "session_token")]
pub session_token: Option<String>,
/// IAM role
#[serde(rename = "aws.credentials.role.arn", alias = "arn")]
pub arn: Option<String>,
/// external ID in IAM role trust policy
#[serde(rename = "aws.credentials.role.external_id", alias = "external_id")]
pub external_id: Option<String>,
#[serde(rename = "aws.region.profile", alias = "profile")]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why credentials.role before arn, external_id, and why region before profile? any reference?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The arn this for the arn of the role. And the role is used for authentication. I also considered aws.authentication.role.xxx but Kinesis already used aws.credentials.role.xxx.

I'll fix the region.

pub profile: Option<String>,
}

Expand Down
8 changes: 4 additions & 4 deletions src/connector/src/with_options_test.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,15 +191,15 @@ struct FieldInfo {
#[serde(skip_serializing_if = "Option::is_none")]
default: Option<String>,

#[serde(skip_serializing_if = "Option::is_none")]
alias: Option<String>,
#[serde(skip_serializing_if = "Vec::is_empty")]
alias: Vec<String>,
}

#[derive(Default)]
struct SerdeProperties {
default_func: Option<String>,
rename: Option<String>,
alias: Option<String>,
alias: Vec<String>,
}

#[derive(Debug, Serialize, Default)]
Expand Down Expand Up @@ -270,7 +270,7 @@ fn extract_serde_properties(field: &Field) -> SerdeProperties {
}
} else if path.is_ident("alias") {
if let Lit::Str(lit_str) = lit {
serde_props.alias = Some(lit_str.value());
serde_props.alias.push(lit_str.value());
}
} else if path.is_ident("default") {
if let Lit::Str(lit_str) = lit {
Expand Down
144 changes: 103 additions & 41 deletions src/connector/with_options_sink.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,33 +21,49 @@ BigQueryConfig:
field_type: usize
required: false
default: '1024'
- name: region
- name: aws.region
field_type: String
required: false
- name: endpoint
alias:
- region
- name: aws.endpoint_url
field_type: String
required: false
alias: endpoint_url
- name: access_key
alias:
- endpoint_url
- endpoint
- name: aws.credentials.access_key_id
field_type: String
required: false
- name: secret_key
alias:
- access_key
- name: aws.credentials.secret_access_key
field_type: String
required: false
- name: session_token
alias:
- secret_key
- name: aws.credentials.session_token
field_type: String
required: false
- name: arn
alias:
- session_token
- name: aws.credentials.role.arn
field_type: String
comments: IAM role
required: false
- name: external_id
alias:
- arn
- name: aws.credentials.role.external_id
field_type: String
comments: external ID in IAM role trust policy
required: false
- name: profile
alias:
- external_id
- name: aws.region.profile
field_type: String
required: false
alias:
- profile
- name: r#type
field_type: String
required: true
Expand Down Expand Up @@ -197,11 +213,13 @@ KafkaConfig:
- name: properties.bootstrap.server
field_type: String
required: true
alias: kafka.brokers
alias:
- kafka.brokers
- name: topic
field_type: String
required: true
alias: kafka.topic
alias:
- kafka.topic
- name: properties.sync.call.timeout
field_type: Duration
required: false
Expand Down Expand Up @@ -352,67 +370,91 @@ KafkaConfig:
field_type: HashMap<String,String>
comments: This is generated from `private_link_targets` and `private_link_endpoint` in frontend, instead of given by users.
required: false
- name: region
- name: aws.region
field_type: String
required: false
- name: endpoint
alias:
- region
- name: aws.endpoint_url
field_type: String
required: false
alias: endpoint_url
- name: access_key
alias:
- endpoint_url
- endpoint
- name: aws.credentials.access_key_id
field_type: String
required: false
- name: secret_key
alias:
- access_key
- name: aws.credentials.secret_access_key
field_type: String
required: false
- name: session_token
alias:
- secret_key
- name: aws.credentials.session_token
field_type: String
required: false
- name: arn
alias:
- session_token
- name: aws.credentials.role.arn
field_type: String
comments: IAM role
required: false
- name: external_id
alias:
- arn
- name: aws.credentials.role.external_id
field_type: String
comments: external ID in IAM role trust policy
required: false
- name: profile
alias:
- external_id
- name: aws.region.profile
field_type: String
required: false
alias:
- profile
KinesisSinkConfig:
fields:
- name: stream
field_type: String
required: true
alias: kinesis.stream.name
alias:
- kinesis.stream.name
- name: aws.region
field_type: String
required: true
alias: kinesis.stream.region
alias:
- kinesis.stream.region
- name: endpoint
field_type: String
required: false
alias: kinesis.endpoint
alias:
- kinesis.endpoint
- name: aws.credentials.access_key_id
field_type: String
required: false
alias: kinesis.credentials.access
alias:
- kinesis.credentials.access
- name: aws.credentials.secret_access_key
field_type: String
required: false
alias: kinesis.credentials.secret
alias:
- kinesis.credentials.secret
- name: aws.credentials.session_token
field_type: String
required: false
alias: kinesis.credentials.session_token
alias:
- kinesis.credentials.session_token
- name: aws.credentials.role.arn
field_type: String
required: false
alias: kinesis.assumerole.arn
alias:
- kinesis.assumerole.arn
- name: aws.credentials.role.external_id
field_type: String
required: false
alias: kinesis.assumerole.external_id
alias:
- kinesis.assumerole.external_id
MqttConfig:
fields:
- name: url
Expand Down Expand Up @@ -525,11 +567,13 @@ PulsarConfig:
- name: topic
field_type: String
required: true
alias: pulsar.topic
alias:
- pulsar.topic
- name: service.url
field_type: String
required: true
alias: pulsar.service.url
alias:
- pulsar.service.url
- name: auth.token
field_type: String
required: false
Expand All @@ -545,33 +589,49 @@ PulsarConfig:
- name: oauth.scope
field_type: String
required: false
- name: region
- name: aws.region
field_type: String
required: false
- name: endpoint
alias:
- region
- name: aws.endpoint_url
field_type: String
required: false
alias: endpoint_url
- name: access_key
alias:
- endpoint_url
- endpoint
- name: aws.credentials.access_key_id
field_type: String
required: false
- name: secret_key
alias:
- access_key
- name: aws.credentials.secret_access_key
field_type: String
required: false
- name: session_token
alias:
- secret_key
- name: aws.credentials.session_token
field_type: String
required: false
- name: arn
alias:
- session_token
- name: aws.credentials.role.arn
field_type: String
comments: IAM role
required: false
- name: external_id
alias:
- arn
- name: aws.credentials.role.external_id
field_type: String
comments: external ID in IAM role trust policy
required: false
- name: profile
alias:
- external_id
- name: aws.region.profile
field_type: String
required: false
alias:
- profile
- name: properties.batch.size
field_type: u32
required: false
Expand Down Expand Up @@ -645,12 +705,14 @@ StarrocksConfig:
field_type: String
comments: The port to the MySQL server of `StarRocks` FE.
required: true
alias: starrocks.query_port
alias:
- starrocks.query_port
- name: starrocks.httpport
field_type: String
comments: The port to the HTTP server of `StarRocks` FE.
required: true
alias: starrocks.http_port
alias:
- starrocks.http_port
- name: starrocks.user
field_type: String
comments: The user name used to access the `StarRocks` database.
Expand Down
Loading
Loading