Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(services/s3): add append support #5428

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

feat(services/s3): add append support #5428

wants to merge 3 commits into from
+135 −41

Conversation

Frank-III
Copy link
Contributor

Which issue does this PR close?

Close: #5351

Are there any user-facing changes?

when writing to S3 Express One Zone, user can:

let writer = op.writer_with(path).append(true).await?;
writer.write(bs).await?;
writer.close().await?;

@Frank-III Frank-III requested a review from Xuanwo as a code owner December 18, 2024 22:57
Xuanwo
Xuanwo reviewed Dec 23, 2024
View reviewed changes
core/src/services/s3/backend.rs Outdated
// Only S3 Express One Zone storage class supports append.
write_can_append: self
.core
.default_storage_class
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, checking with default_storage_class doesn't seem like a good idea to me. How about adding a new flag called enable_write_with_append? Users who confirm that they support append can enable this.

core/src/services/s3/core.rs
req = req.header(format!("{X_AMZ_META_PREFIX}{key}"), value)
}
}
req = self.insert_metadata_headers(req, Some(size), args);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I assume that only the first append call supports setting up metadata. We don't need to repeat the metadata for subsequent append requests.

@Frank-III Frank-III marked this pull request as draft December 24, 2024 00:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Xuanwo Xuanwo Xuanwo left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
releases-note/feat
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

new feature: Implement append object for S3
2 participants
@Frank-III @Xuanwo