Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verify put object options #4274

Merged
merged 38 commits into from
Dec 31, 2024
Merged
Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
ad446f9
cli test script
drernie Dec 27, 2024
639bc67
use kms policy
drernie Dec 27, 2024
054541e
class TestPutOptions
drernie Dec 27, 2024
b66c754
bucket.put_file fails
drernie Dec 27, 2024
a7768a7
pass put_file
drernie Dec 27, 2024
2d94bdb
Merge branch 'put-object-options' into verify-put-object-options
drernie Dec 27, 2024
a4fcefa
Delete README.md
drernie Dec 27, 2024
f02a676
lint TEST.md
drernie Dec 27, 2024
e4f6d96
use boto3 to verify/cleanup
drernie Dec 27, 2024
bb636cb
test_bucket_put_dir
drernie Dec 27, 2024
64068ca
fail test_package_push
drernie Dec 27, 2024
71c7e11
isort
drernie Dec 27, 2024
36ebd47
Merge branch 'put-object-options' into verify-put-object-options
drernie Dec 27, 2024
6ca9501
verify package created
drernie Dec 27, 2024
454698f
pkg_entry.fetch
drernie Dec 27, 2024
f5fd64d
fetch entry
drernie Dec 28, 2024
1ec19fd
delete_key
drernie Dec 28, 2024
9dc1160
pkg.build
drernie Dec 28, 2024
4d67b74
mock using QuiltTestCase
drernie Dec 28, 2024
4400a11
mock list-url
drernie Dec 30, 2024
5070a43
pass fetch test (!mpu)
drernie Dec 30, 2024
08153e5
split out test_package_entry_fetch
drernie Dec 31, 2024
e4b6e84
manual PackageEntry
drernie Dec 31, 2024
5919d17
mock_copy_object_side_effect
drernie Dec 31, 2024
05317d8
refactor MockEntry
drernie Dec 31, 2024
ddb23e1
test_package_entry_fetch_mpu
drernie Dec 31, 2024
0ee231c
pass mock_create_mpu
drernie Dec 31, 2024
3d743a2
mock all calls
drernie Dec 31, 2024
6cd6dda
lint
drernie Dec 31, 2024
2f9179c
don't add to parts
drernie Dec 31, 2024
877d981
stub test_package_push_mpu
drernie Dec 31, 2024
f576b65
pass test_package_push_mpu
drernie Dec 31, 2024
c342f25
revert s3_upload_params
drernie Dec 31, 2024
9adf81d
lint
drernie Dec 31, 2024
2218daa
Merge branch 'put-object-options' into verify-put-object-options
drernie Dec 31, 2024
71f9b48
remove manual tests
drernie Dec 31, 2024
285d098
Merge branch 'verify-put-object-options' of https://github.com/quiltd…
drernie Dec 31, 2024
2ae1311
remove dest_key
drernie Dec 31, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 4 additions & 16 deletions api/python/quilt3/data_transfer.py
Original file line number Diff line number Diff line change
Expand Up @@ -352,16 +352,14 @@ def upload_part(i, start, end):
nonlocal remaining
part_id = i + 1
with ReadFileChunk.from_filename(src_path, start, end-start, [ctx.progress]) as fd:
s3_upload_params = dict(
part = s3_client.upload_part(
Body=fd,
Bucket=dest_bucket,
Key=dest_key,
UploadId=upload_id,
PartNumber=part_id,
ChecksumAlgorithm='SHA256',
ChecksumAlgorithm="SHA256",
)
add_put_options_safely(s3_upload_params, put_options)
part = s3_client.upload_part(**s3_upload_params)
with lock:
parts[i] = dict(
PartNumber=part_id,
Expand All @@ -372,13 +370,6 @@ def upload_part(i, start, end):
done = remaining == 0

if done:
s3_complete_params = dict(
Bucket=dest_bucket,
Key=dest_key,
UploadId=upload_id,
MultipartUpload={'Parts': parts},
)
add_put_options_safely(s3_complete_params, put_options)
resp = s3_client.complete_multipart_upload(
Bucket=dest_bucket,
Key=dest_key,
Expand Down Expand Up @@ -520,16 +511,14 @@ def _copy_remote_file(ctx: WorkerContext, size: int, src_bucket: str, src_key: s
def upload_part(i, start, end):
nonlocal remaining
part_id = i + 1
s3_upload_params = dict(
part = s3_client.upload_part_copy(
CopySource=src_params,
CopySourceRange=f'bytes={start}-{end-1}',
CopySourceRange=f"bytes={start}-{end-1}",
Bucket=dest_bucket,
Key=dest_key,
UploadId=upload_id,
PartNumber=part_id,
)
add_put_options_safely(s3_upload_params, extra_args)
part = s3_client.upload_part_copy(**s3_upload_params)
with lock:
parts[i] = dict(
PartNumber=part_id,
Expand All @@ -548,7 +537,6 @@ def upload_part(i, start, end):
UploadId=upload_id,
MultipartUpload={'Parts': parts},
)
add_put_options_safely(s3_complete_params, extra_args)
resp = s3_client.complete_multipart_upload(**s3_complete_params)
version_id = resp.get('VersionId') # Absent in unversioned buckets.
checksum, _ = resp['ChecksumSHA256'].split('-', 1)
Expand Down
Loading
Loading