feat(sink): Support big query sink #12873

xxhZs · 2023-10-16T09:58:13Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

support big query sink. Because BigQuery has limited support for updates and deletes, we currently only support 'append only'

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

sql example
CREATE SINK s1 FROM t1 WITH (
connector = 'bigquery',
type = 'append-only',
bigquery.path= '${bigquery_service_account_json_path}',
bigquery.project= '${project_id}',
bigquery.dataset= '${dataset_id}',
bigquery.table= '${table_id}',
force_append_only='true'
);

bigquery.path: Big Query service account json file path. Can find it inhttps://console.cloud.google.com/iam-admin/serviceaccounts.
bigquery.project: Big Query project id
bigquery.dataset: Big Query dataset id
bigquery.table: Big Query table id

fix fix fmt

wenym1

Rest LGTM. Thanks for the PR!

src/connector/src/sink/mod.rs

src/connector/src/sink/encoder/mod.rs

src/connector/src/sink/big_query.rs

wenym1 · 2023-10-23T10:29:54Z

src/connector/src/sink/big_query.rs

+    }
+
+    async fn append_only(&mut self, chunk: StreamChunk) -> Result<()> {
+        let mut insert_vec = vec![];


From the current implementation, it seems that each self.insert_request only lives inside the scope of append_only? If so, instead of storing it as field, we may create a new local variable inside each append_only?

We will use this variable in two places, either when it exceeds the maximum size or when a barrier is encountered.

src/connector/src/sink/big_query.rs

codecov · 2023-10-23T12:33:55Z

Codecov Report

Merging #12873 (0cb4f51) into main (84d27ba) will decrease coverage by 0.07%.
Report is 1 commits behind head on main.
The diff coverage is 11.07%.

@@            Coverage Diff             @@
##             main   #12873      +/-   ##
==========================================
- Coverage   68.23%   68.16%   -0.07%     
==========================================
  Files        1506     1507       +1     
  Lines      255354   255634     +280     
==========================================
+ Hits       174235   174259      +24     
- Misses      81119    81375     +256

Flag	Coverage Δ
rust	`68.16% <11.07%> (-0.07%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
src/connector/src/sink/encoder/mod.rs	`89.74% <ø> (ø)`
src/connector/src/sink/mod.rs	`60.71% <ø> (ø)`
src/connector/src/sink/encoder/json.rs	`87.23% <0.00%> (-2.09%)`	⬇️
src/connector/src/sink/big_query.rs	`11.61% <11.61%> (ø)`

... and 11 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

wenym1

LGTM!

wenym1 · 2023-10-24T12:45:50Z

src/connector/src/sink/big_query.rs

+};
+
+pub const BIGQUERY_SINK: &str = "bigquery";
+const BIGQUERY_INSERT_MAX_NUMS: usize = 500;


Any reason for setting this value?

Set the maximum number of cached rows; any rows exceeding this limit will be inserted directly to prevent certain issues caused by excessive data within a single barrier.

tabVersion

rest lgtm

src/connector/src/sink/big_query.rs

fuyufjh

LGTM for Cargo.toml

gitguardian · 2023-10-25T15:34:01Z

⚠️ GitGuardian has uncovered 3 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	Secret	Commit	Filename
7648795	Generic CLI Secret	`34f07d7`	integration_tests/iceberg-cdc/run_test.sh	View secret
7648795	Generic CLI Secret	`0cb4f51`	integration_tests/iceberg-cdc/run_test.sh	View secret
7648795	Generic CLI Secret	`0cb4f51`	integration_tests/iceberg-cdc/docker-compose.yml	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}

xxhZs added 2 commits October 13, 2023 15:18

support big query sink

cec68b9

support bigquery sink

735b6d1

github-actions bot added the type/feature label Oct 16, 2023

xxhZs force-pushed the xxh/big_query_sink branch 2 times, most recently from e3ed31d to 5e80162 Compare October 16, 2023 10:01

add integration test

0469554

fix fix fmt

xxhZs force-pushed the xxh/big_query_sink branch from 5e80162 to 0469554 Compare October 16, 2023 10:06

xxhZs marked this pull request as ready for review October 16, 2023 10:07

xxhZs requested a review from a team as a code owner October 16, 2023 10:07

xxhZs requested review from hzxa21, wenym1 and tabVersion October 16, 2023 10:07

xxhZs added 2 commits October 23, 2023 16:50

Merge branch 'main' into xxh/big_query_sink

d48b976

fix cargo lock

5e3cd65

wenym1 reviewed Oct 23, 2023

View reviewed changes

fix

cb4de65

wenym1 approved these changes Oct 24, 2023

View reviewed changes

tabVersion approved these changes Oct 24, 2023

View reviewed changes

src/connector/src/sink/big_query.rs Show resolved Hide resolved

src/connector/src/sink/big_query.rs Outdated Show resolved Hide resolved

fuyufjh approved these changes Oct 25, 2023

View reviewed changes

fix

eaaca1e

xxhZs added the user-facing-changes Contains changes that are visible to users label Oct 25, 2023

xxhZs added this pull request to the merge queue Oct 25, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 25, 2023

xxhZs added this pull request to the merge queue Oct 25, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 25, 2023

Merge branch 'main' into xxh/big_query_sink

34f07d7

xxhZs enabled auto-merge October 25, 2023 15:35

Merge branch 'main' into xxh/big_query_sink

946f2a7

xxhZs disabled auto-merge October 26, 2023 05:56

add s3 file path

58818f9

xxhZs requested a review from tabVersion October 30, 2023 03:22

tabVersion approved these changes Nov 1, 2023

View reviewed changes

xxhZs enabled auto-merge November 1, 2023 03:26

xxhZs disabled auto-merge November 1, 2023 03:28

Merge branch 'main' into xxh/big_query_sink

0cb4f51

xxhZs enabled auto-merge November 1, 2023 03:28

xxhZs added this pull request to the merge queue Nov 1, 2023

Merged via the queue into main with commit 7f5d3f6 Nov 1, 2023
8 checks passed

xxhZs deleted the xxh/big_query_sink branch November 1, 2023 04:16

xxchan mentioned this pull request Nov 10, 2023

risingwave 1.4.0 risingwavelabs/homebrew-risingwave#17

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(sink): Support big query sink #12873

feat(sink): Support big query sink #12873

xxhZs commented Oct 16, 2023 •

edited

Loading

wenym1 left a comment

wenym1 Oct 23, 2023

xxhZs Oct 23, 2023

codecov bot commented Oct 23, 2023 •

edited

Loading

wenym1 left a comment

wenym1 Oct 24, 2023

xxhZs Oct 25, 2023

tabVersion left a comment

fuyufjh left a comment

gitguardian bot commented Oct 25, 2023 •

edited

Loading

feat(sink): Support big query sink #12873

feat(sink): Support big query sink #12873

Conversation

xxhZs commented Oct 16, 2023 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

wenym1 left a comment

Choose a reason for hiding this comment

wenym1 Oct 23, 2023

Choose a reason for hiding this comment

xxhZs Oct 23, 2023

Choose a reason for hiding this comment

codecov bot commented Oct 23, 2023 • edited Loading

Codecov Report

wenym1 left a comment

Choose a reason for hiding this comment

wenym1 Oct 24, 2023

Choose a reason for hiding this comment

xxhZs Oct 25, 2023

Choose a reason for hiding this comment

tabVersion left a comment

Choose a reason for hiding this comment

fuyufjh left a comment

Choose a reason for hiding this comment

gitguardian bot commented Oct 25, 2023 • edited Loading

⚠️ GitGuardian has uncovered 3 secrets following the scan of your pull request.

xxhZs commented Oct 16, 2023 •

edited

Loading

codecov bot commented Oct 23, 2023 •

edited

Loading

gitguardian bot commented Oct 25, 2023 •

edited

Loading