refactor: bump aws sdk, use rustls in s3 client #13286

MrCroxx · 2023-11-07T09:19:29Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Bump aws sdk version to latest. Replace native-tls with rustls for s3 client because aws sdk doesn't support it anymore.

close #9503 #11842 #11852

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Signed-off-by: MrCroxx <[email protected]>

MrCroxx · 2023-11-07T09:30:55Z

src/object_store/src/object/s3.rs

+    pub fn new_http_client(config: &S3ObjectStoreConfig) -> impl HttpClient {
+        let mut http = hyper::client::HttpConnector::new();

-            // connection config
-            if let Some(keepalive_ms) = config.keepalive_ms.as_ref() {
-                http.set_keepalive(Some(Duration::from_millis(*keepalive_ms)));
-            }
+        // connection config
+        if let Some(keepalive_ms) = config.keepalive_ms.as_ref() {
+            http.set_keepalive(Some(Duration::from_millis(*keepalive_ms)));
+        }

-            if let Some(nodelay) = config.nodelay.as_ref() {
-                http.set_nodelay(*nodelay);
-            }
+        if let Some(nodelay) = config.nodelay.as_ref() {
+            http.set_nodelay(*nodelay);
+        }

-            if let Some(recv_buffer_size) = config.recv_buffer_size.as_ref() {
-                http.set_recv_buffer_size(Some(*recv_buffer_size));
-            }
+        if let Some(recv_buffer_size) = config.recv_buffer_size.as_ref() {
+            http.set_recv_buffer_size(Some(*recv_buffer_size));
+        }

-            if let Some(send_buffer_size) = config.send_buffer_size.as_ref() {
-                http.set_send_buffer_size(Some(*send_buffer_size));
-            }
+        if let Some(send_buffer_size) = config.send_buffer_size.as_ref() {
+            http.set_send_buffer_size(Some(*send_buffer_size));
+        }

-            http.enforce_http(false);
-            hyper_tls::HttpsConnector::from((http, tls.into()))
-        };
+        http.enforce_http(false);
+
+        let conn = hyper_rustls::HttpsConnectorBuilder::new()
+            .with_webpki_roots()
+            .https_or_http()
+            .enable_all_versions()
+            .wrap_connector(http);

-        aws_smithy_client::hyper_ext::Adapter::builder()
-            .hyper_builder(hyper::client::Builder::default())
-            .connector_settings(ConnectorSettings::builder().build())
-            .build(monitor_connector(native_tls, "S3"))
+        let conn = monitor_connector(conn, "S3");
+
+        HyperClientBuilder::new().build(conn)


@wenym1 @zwang28 PTAL

codecov · 2023-11-07T09:45:27Z

Codecov Report

Attention: 77 lines in your changes are missing coverage. Please review.

Comparison is base (932d694) 67.85% compared to head (b0bcbb6) 67.88%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
src/object_store/src/object/s3.rs	0.00%	49 Missing ⚠️
src/meta/src/rpc/cloud_provider.rs	0.00%	11 Missing ⚠️
src/object_store/src/object/error.rs	0.00%	10 Missing ⚠️
src/connector/src/source/kinesis/source/message.rs	0.00%	3 Missing ⚠️
src/connector/src/aws_auth.rs	0.00%	2 Missing ⚠️
...onnector/src/source/filesystem/s3/source/reader.rs	0.00%	1 Missing ⚠️
.../connector/src/source/kinesis/enumerator/client.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13286      +/-   ##
==========================================
+ Coverage   67.85%   67.88%   +0.02%     
==========================================
  Files        1526     1526              
  Lines      259938   259940       +2     
==========================================
+ Hits       176392   176456      +64     
+ Misses      83546    83484      -62

Flag	Coverage Δ
rust	`67.88% <0.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

xxchan · 2023-11-08T01:59:51Z

Cargo.lock

+[[patch.unused]]
+name = "getrandom"
+version = "0.2.9"
+source = "git+https://github.com/madsim-rs/getrandom.git?rev=8daf97e#8daf97e4142635fe28543b2db9022f5e2544bb5c"


Yep, madsim also needs to update the dependency. I'll change to madsim source when @wangrunji0408 publishes the new version.

published as madsim-aws-sdk-s3 v0.3.0

Btw, getrandom has updated to v0.2.11 some days ago. Although there's nothing important according to its changelog. If we are going to bump its version, remember to bump the patch version as well:

# Cargo.toml getrandom = { git = "https://github.com/madsim-rs/getrandom.git", rev = "e79a7ae" }

If we are going to bump its version, remember to bump the patch version as well:

It was eventually forgotten.

src/object_store/Cargo.toml

hzxa21 · 2023-11-07T09:47:00Z

Cargo.toml

+aws-config = { version = "0.57", default-features = false, features = [
  "rt-tokio",
-  "native-tls",
 ] }
-aws-credential-types = { version = "0.55", default-features = false, features = [
+aws-credential-types = { version = "0.57", default-features = false, features = [
  "hardcoded-credentials",
 ] }
-aws-sdk-kinesis = { version = "0.28", default-features = false, features = [
+aws-sdk-kinesis = { version = "0.35", default-features = false, features = [
  "rt-tokio",
-  "native-tls",
 ] }
-aws-sdk-s3 = { version = "0.28", default-features = false, features = [
+aws-sdk-s3 = { version = "0.35", default-features = false, features = [
  "rt-tokio",
-  "native-tls",
 ] }
-aws-sdk-ec2 = { version = "0.28", default-features = false, features = [
+aws-sdk-ec2 = { version = "0.35", default-features = false, features = [
  "rt-tokio",
-  "native-tls",
 ] }
-aws-sdk-sqs = { version = "0.28", default-features = false, features = [
+aws-sdk-sqs = { version = "0.35", default-features = false, features = [
  "rt-tokio",
-  "native-tls",
 ] }


Did we forget to enable the rustls feature here?

Fixed. Is there any way to test if rustls is enabled and https is working?

If you try to connect to S3 via ec2 in the same region without setting the endpoint explicitly, I think https is used by default. You can also set RW_S3_ENDPOINT explicitly with https://.....

Signed-off-by: MrCroxx <[email protected]>

hzxa21

Rest LGTM

hzxa21 · 2023-11-10T10:55:00Z

src/object_store/src/object/s3.rs

@@ -293,7 +294,9 @@ impl StreamingUploader for S3StreamingUploader {

 fn get_upload_body(data: Vec<Bytes>) -> ByteStream {
    SdkBody::retryable(move || {
-        Body::wrap_stream(stream::iter(data.clone().into_iter().map(ObjectResult::Ok))).into()
+        SdkBody::from_body_0_4(Body::wrap_stream(stream::iter(


where does from_body_0_4 come from? Any documentation?

Fixed. Now it uses hyper-0-14-x, which matches our deps.

Signed-off-by: MrCroxx <[email protected]>

TennyZhuang · 2023-11-14T13:49:18Z

The PR bumped getrandom, then we should also update the madsim patch.

risingwave/Cargo.toml

Line 244 in 2df34ee

    
           getrandom = { git = "https://github.com/madsim-rs/getrandom.git", rev = "8daf97e" }

cc @wangrunji0408

refactor: bump aws sdk, use rustls in s3 client

99609b9

Signed-off-by: MrCroxx <[email protected]>

MrCroxx requested review from hzxa21, wangrunji0408, xxchan, wenym1, wcy-fdu and zwang28 November 7, 2023 09:19

MrCroxx self-assigned this Nov 7, 2023

MrCroxx requested a review from a team as a code owner November 7, 2023 09:19

github-actions bot added the type/refactor label Nov 7, 2023

MrCroxx commented Nov 7, 2023

View reviewed changes

xxchan reviewed Nov 8, 2023

View reviewed changes

hzxa21 reviewed Nov 8, 2023

View reviewed changes

wangrunji0408 mentioned this pull request Nov 8, 2023

chore(dep): bump aws-sdk [WIP] #11852

Closed

8 tasks

MrCroxx added 2 commits November 10, 2023 15:19

chore: enable rustls feature for aws sdk, change madsim to v3.0

4a8ba89

Signed-off-by: MrCroxx <[email protected]>

Merge remote-tracking branch 'origin/main' into xx/bump-aws-sdk

3cfaffe

Signed-off-by: MrCroxx <[email protected]>

MrCroxx requested a review from hzxa21 November 10, 2023 07:27

hzxa21 approved these changes Nov 10, 2023

View reviewed changes

MrCroxx added 3 commits November 14, 2023 12:03

Merge branch 'main' into xx/bump-aws-sdk

53999da

Signed-off-by: MrCroxx <[email protected]>

fix: use hyper-0-14-x feature instead of http-body-0-4

ace24ef

Signed-off-by: MrCroxx <[email protected]>

Merge remote-tracking branch 'origin/main' into xx/bump-aws-sdk

b0bcbb6

MrCroxx added this pull request to the merge queue Nov 14, 2023

Merged via the queue into main with commit 09d312c Nov 14, 2023
27 of 28 checks passed

MrCroxx deleted the xx/bump-aws-sdk branch November 14, 2023 07:10

TennyZhuang mentioned this pull request Nov 14, 2023

chore(deps): bump getrandom to v0.2.11 and its madsim patch #13426

Merged

8 tasks

MrCroxx mentioned this pull request Nov 23, 2023

bug: performance spike without recovery or s3 failure #13584

Closed

MrCroxx restored the xx/bump-aws-sdk branch November 23, 2023 09:13

MrCroxx mentioned this pull request Nov 24, 2023

chore: bump aws sdk #13623

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: bump aws sdk, use rustls in s3 client #13286

refactor: bump aws sdk, use rustls in s3 client #13286

MrCroxx commented Nov 7, 2023 •

edited

Loading

MrCroxx Nov 7, 2023

codecov bot commented Nov 7, 2023 •

edited

Loading

xxchan Nov 8, 2023

MrCroxx Nov 8, 2023

wangrunji0408 Nov 10, 2023

wangrunji0408 Nov 10, 2023 •

edited

Loading

TennyZhuang Nov 14, 2023

wangrunji0408 Nov 14, 2023

hzxa21 Nov 7, 2023

MrCroxx Nov 10, 2023

hzxa21 Nov 13, 2023

hzxa21 left a comment

hzxa21 Nov 10, 2023

MrCroxx Nov 14, 2023

TennyZhuang commented Nov 14, 2023

refactor: bump aws sdk, use rustls in s3 client #13286

refactor: bump aws sdk, use rustls in s3 client #13286

Conversation

MrCroxx commented Nov 7, 2023 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

Choose a reason for hiding this comment

codecov bot commented Nov 7, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangrunji0408 Nov 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hzxa21 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TennyZhuang commented Nov 14, 2023

MrCroxx commented Nov 7, 2023 •

edited

Loading

codecov bot commented Nov 7, 2023 •

edited

Loading

wangrunji0408 Nov 10, 2023 •

edited

Loading