Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(middleware-flexible-checksums): delay checksum validation until stream read #6629

Merged
merged 4 commits into from
Nov 6, 2024

Conversation

trivikr
Copy link
Member

@trivikr trivikr commented Nov 6, 2024

Issue

Internal JS-5537

Description

Delays checksum validation until stream read

Testing

import assert from "assert";
import { S3 } from "@aws-sdk/client-s3";
// import { S3 } from "../aws-sdk-js-v3/clients/client-s3/dist-cjs/index.js";

const s3 = new S3({ region: "us-west-2" });
const Bucket = "test-flexible-checksums"; // Change to your bucket name
const Key = "test.txt";
const Body = "helloworld".repeat(10_000_000); // A string of size ~100MB

await s3.putObject({ Bucket, Key, Body, ChecksumAlgorithm: "CRC32" });

const apiCallStart = performance.now();
const response = await s3.getObject({ Bucket, Key, ChecksumMode: "ENABLED" });
console.log("time taken for request:", (performance.now() - apiCallStart).toFixed(2), "ms");

const transformStart = performance.now();
const body = await response.Body.transformToString();
console.log("time taken for transform:", (performance.now() - transformStart).toFixed(2), "ms");

assert(body === Body);

Before

Response checksum is validated when API call is made.

time taken for request: 3141.02 ms
time taken for transform: 221.46 ms

After

Response checksum is validation when stream is consumed (i.e. when transform call is made)

time taken for request: 1217.04 ms
time taken for transform: 2141.58 ms

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@trivikr trivikr marked this pull request as ready for review November 6, 2024 05:09
@trivikr trivikr requested a review from a team as a code owner November 6, 2024 05:09
@trivikr trivikr marked this pull request as draft November 6, 2024 17:02
@trivikr trivikr marked this pull request as ready for review November 6, 2024 17:03
@trivikr
Copy link
Member Author

trivikr commented Nov 6, 2024

E2E test was added in d9d7534 (#6629)

@trivikr trivikr merged commit 43ee3c1 into aws:main Nov 6, 2024
4 checks passed
@trivikr trivikr deleted the middleware-flexible-checksums-validation branch November 6, 2024 23:04
Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants