Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use randomized content ID for Azure multipart uploads #6869

Merged
merged 10 commits into from
Dec 12, 2024
1 change: 1 addition & 0 deletions object_store/src/aws/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -570,6 +570,7 @@ mod tests {
rename_and_copy(&integration).await;
stream_get(&integration).await;
multipart(&integration, &integration).await;
multipart_race_condition(&integration, true).await;
signing(&integration).await;
s3_encryption(&integration).await;
put_get_attributes(&integration).await;
Expand Down
2 changes: 1 addition & 1 deletion object_store/src/azure/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -314,7 +314,7 @@ mod tests {
stream_get(&integration).await;
put_opts(&integration, true).await;
multipart(&integration, &integration).await;
multipart_race_condition(&integration).await;
multipart_race_condition(&integration, false).await;
signing(&integration).await;

let validate = !integration.client.config().disable_tagging;
Expand Down
1 change: 1 addition & 0 deletions object_store/src/gcp/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,7 @@ mod test {
// https://github.com/fsouza/fake-gcs-server/issues/852
stream_get(&integration).await;
multipart(&integration, &integration).await;
multipart_race_condition(&integration, true).await;
// Fake GCS server doesn't currently honor preconditions
get_opts(&integration).await;
put_opts(&integration, true).await;
Expand Down
11 changes: 8 additions & 3 deletions object_store/src/integration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1113,7 +1113,7 @@ async fn delete_fixtures(storage: &DynObjectStore) {
}

/// Tests a race condition where 2 threads are performing multipart writes to the same path
pub async fn multipart_race_condition(storage: &dyn ObjectStore) {
pub async fn multipart_race_condition(storage: &dyn ObjectStore, last_writer_wins: bool) {
let path = Path::from("test_multipart_race_condition");

let mut multipart_upload_1 = storage.put_multipart(&path).await.unwrap();
Expand Down Expand Up @@ -1165,9 +1165,14 @@ pub async fn multipart_race_condition(storage: &dyn ObjectStore) {
.unwrap();

multipart_upload_1.complete().await.unwrap();
let err = multipart_upload_2.complete().await.unwrap_err();

assert!(matches!(err, crate::Error::Generic { .. }), "{err}");
if last_writer_wins {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect this should also yield a different final result

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct. I missed that. Pushed

multipart_upload_2.complete().await.unwrap();
} else {
let err = multipart_upload_2.complete().await.unwrap_err();

assert!(matches!(err, crate::Error::Generic { .. }), "{err}");
}

let get_result = storage.get(&path).await.unwrap();
let bytes = get_result.bytes().await.unwrap();
Expand Down
Loading