Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(storage): update multipart operations to support multiple buckets/regions #2899

Conversation

phantumcode
Copy link
Contributor

@phantumcode phantumcode commented Aug 20, 2024

Multi-bucket support part 3

This the third part of a series of changes/pull request to support multi-bucket in the S3 Storage Plugin. The series of changes will be reviewed and merged into a feature branch and combined into a single pull request to be reviewed and merged into main branch.

part 1: #2895
part 2: #2896

  • PR title and description conform to Pull Request guidelines.

Issue #, if available:

Description of changes:

  • Updated transfer table to upgrade to new version with region column added.
  • Refactored transfer workers to get new s3 client from a client provider that gets the S3 client based on the transfer record's region and bucket name
  • Updated transfer operations to get AWSS3Service based on bucket name and region

Integration tests will be added in a separate followup PR (#2900)

How did you test these changes?

  • Manually tested storage apis and verified uploads/downloads works as expected.
  • Verified migrating from old transfer db table to new transfer db table works as expecting while in the middle of an large file upload. Large file upload was able to continue and complete successfully after table migration/upgrade and region column was default to null.

Documentation update required?

  • No
  • Yes (Please include a PR link for the documentation update)

General Checklist

  • Added Unit Tests
  • Added Integration Tests
  • Security oriented best practices and standards are followed (e.g. using input sanitization, principle of least privilege, etc)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@phantumcode phantumcode marked this pull request as ready for review August 20, 2024 15:19
@phantumcode phantumcode requested a review from a team as a code owner August 20, 2024 15:19
Base automatically changed from phantum/storage_options_bucket to feature/storage_multi-bucket_support August 22, 2024 15:22
@phantumcode phantumcode merged commit 5125555 into feature/storage_multi-bucket_support Aug 27, 2024
@phantumcode phantumcode deleted the phantum/storage-update-transfer-db branch August 27, 2024 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants