Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(sink): Avro with AWS Glue Schema Registry #20181

Merged
merged 8 commits into from
Jan 17, 2025

Conversation

xiangjinwu
Copy link
Contributor

@xiangjinwu xiangjinwu commented Jan 16, 2025

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Supports loading schema from and encoding metadata for AWS Glue Schema Registry, in addition to Confluent Schema Registry, when using encode avro.

Notes:

  • Glue Schema Registry has no references (eg import in protobuf).
  • Glue Schema Registry has no name strategy. So it would require an additional aws.glue.key_schema_arn. This is left unimplemented in favor of real use case KEY ENCODE [TEXT | BYTES].

Related: #17605 for source

Refactor to come next: {Confluent, Glue} x {Source, Sink} x {Avro, Protobuf}

Checklist

  • I have written necessary rustdoc comments.
  • I have added necessary unit tests and integration tests.
  • I have added test labels as necessary.
  • I have added fuzzing tests or opened an issue to track them.
  • My PR contains breaking changes.
  • My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
  • My PR contains critical fixes that are necessary to be merged into the latest release.

Documentation

  • My PR needs documentation updates.

Added support for AWS Glue Schema Registry when using encode avro for sinks:

encode avro (
  aws.glue.schema_arn = 'arn:aws:glue:ap-southeast-1:123456123456:schema/default-registry/MyEvent',
  // other aws auth options same as s3 / kinesis
  aws.region = '...',
  aws.credentials.access_key_id = '...',
  ...
) [KEY ENCODE [TEXT | BYTES]] -- when specifying `primary_key` as kafka key

When using IAM permission policies, the following actions shall be allowed:

  • glue:GetSchemaVersion

Copy link
Member

@xxchan xxchan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants