feat(sink): Optimizing ClickHouse Upsert #12259
Conversation
xxhZs
changed the title
Codecov Report
@@ Coverage Diff @@
## main #12259 +/- ##
==========================================
- Coverage 68.64% 68.56% -0.09%
==========================================
Files 1496 1496
Lines 251405 251440 +35
==========================================
- Hits 172588 172388 -200
- Misses 78817 79052 +235
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 21 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
| #[allow(clippy::enum_variant_names)] | ||
| #[derive(Debug)] | ||
| enum ClickHouseEngine { | ||
| MergeTree, |
There was a problem hiding this comment.
Can all these engines be supported? Or just some of them?
There was a problem hiding this comment.
Yes, here we support all of them. CollapsingMergeTree and CollapsingMergeTree support upsert, while others only support append-only. Among these, only ReplacingMergeTree can guarantee that the same primary key is inserted only once.
xxhZs
force-pushed
the
xxh/clickhouse_sink_upsert2
branch
from
9c6be25 to
13d2ba9
Compare
|
| GitGuardian id | Secret | Commit | Filename | |
|---|---|---|---|---|
| 7648795 | Generic CLI Secret | 13d2ba9 | integration_tests/iceberg-cdc/run_test.sh | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Our GitHub checks need improvements? Share your feedbacks!
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
#11967
In this PR, We adopt different sink support methods for different ClickHouse engines. For all ClickHouse engines, we support 'append-only,' where only for ReplacingMergeTree, we do not insert duplicate data. Additionally, for CollapsingMergeTree and VersionedCollapsingMergeTree, we support 'upsert,' where 'rw' will transform 'delete' into 'insert sign=1' form. We also provide support for the 'decimal' format.
Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
We only support
upsertwith clickhouse'CollapsingMergeTreeandVersionedCollapsingMergeTree