Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use AWS_MSK_IAM as properties.sasl.mechanism for MSK source #103

Merged
merged 1 commit into from
Dec 2, 2024
Merged

fix: use AWS_MSK_IAM as properties.sasl.mechanism for MSK source #103

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion integrations/sources/amazon-msk.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ WITH (
connector = 'kafka', topic = '<topic-name>',
properties.bootstrap.server = '<broker-url>',
scan.startup.mode = 'earliest',
properties.sasl.mechanism = 'SCRAM-SHA-512',
properties.sasl.mechanism = 'AWS_MSK_IAM',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems problematic to me. The whole tutorial is to configure MSK to use SASL/SCRAM, so we can't simply change it here.

image

Besides, to use IAM, we have to give RisingWave the IAM role with permission to access the MSK, and there's no username and password.

This also means that IAM can only be used for on-prem/BYOC, but not cloud. So perhaps we should keep SCRAM in the doc. Maybe add another section for IAM.

cc @yuhao-su to confirm my understanding.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, it is a glitch indeed.
For IAM, RW loads credentials from EC2 and does not require explicit AK/SK, and yes, for on-perm only.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, for SASL/SCRAM, it also seems not possible for cloud, because we need to turn on public access for MSK, which will then require ACL, and we can't add RW cloud to the ACL.

Is it even possible to connect to MSK on cloud? 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the brokers are still available when with the help of private link.

public access for MSK

why we need a public access? Or, why assume role cannot work.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, indeed private link may help.

This comment was marked as resolved.

Sign in to view

Copy link
Member

@xxchan xxchan Jan 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a service account that assumes role, and we use that account's AKSK to connect

(We also didn't mention that detail in the doc btw

properties.security.protocol = 'sasl_ssl',
properties.sasl.username = '<your-username>',
properties.sasl.password = '<your-password>'
Expand Down
Loading