refactor(udf): switch to the latest arrow-udf versions #16619
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
BugenZhao
approved these changes
May 9, 2024
| @@ -84,7 +90,7 @@ def hex_to_dec(hex: Optional[str]) -> Optional[Decimal]: | |||
| return dec | |||
|
|
|||
|
|
|||
| @udf(input_types=["FLOAT8"], result_type="DECIMAL") | |||
| @udf(input_types=["FLOAT64"], result_type="DECIMAL") | |||
There was a problem hiding this comment.
Is this considered a breaking change that should be notable for existing users?
There was a problem hiding this comment.
Yes, it is a breaking change, but only for the new version (arrow-udf v0.2). Existing users will not be affected unless they try to upgrade to the new version.
| @@ -22,7 +22,9 @@ embedded-python-udf = ["arrow-udf-python"] | |||
| [dependencies] | |||
| anyhow = "1" | |||
| arrow-array = { workspace = true } | |||
| arrow-flight = "50" | |||
There was a problem hiding this comment.
Can arrow-udf-flight re-export this?
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 9425213 | Triggered | Generic Password | ec49c16 | e2e_test/source/cdc/cdc.validate.postgres.slt | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Signed-off-by: Runji Wang <[email protected]>
Signed-off-by: Runji Wang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Currently, decimal and jsonb types are mapped to Arrow LargeBinary and LargeString types in UDFs. However, since we have decoupled UDF code into the
arrow-udfproject, this design blocks community users from using LargeBinary and LargeString types in Arrow. (arrow-udf/arrow-udf#16) To solve this problem, we decided to use Arrow extension types for these special types. The migration on the arrow-udf side is completed. (arrow-udf/arrow-udf#17, arrow-udf/arrow-udf#18) This PR finishes the rest work on the RisingWave side, which is, switching to new arrow-udf versions and handling backward compatibility issues.In detail, this PR:
risingwave-udfcrate. the code is moved to arrow-udf-flight.arrow-udf-*versionsarrow_udf.rs)Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
The Python UDF SDK is updated. If using RisingWave 1.10, you are encouraged to update to the latest version following the migration guide (to be added later). Older versions are still supported, but may be deprecated in the future version.