feat: introduce iceberg-rust catalog #17308
Conversation
Cargo.toml
Outdated
| iceberg = { git = "https://github.com/risingwavelabs/iceberg-rust.git", rev = "e6ae6229dfd0c0a8793cde42a7d626c704ea9088" } | ||
| iceberg-catalog-rest = { git = "https://github.com/risingwavelabs/iceberg-rust.git", rev = "e6ae6229dfd0c0a8793cde42a7d626c704ea9088" } |
There was a problem hiding this comment.
Any reason why we need to use a forked version?
There was a problem hiding this comment.
for now, the upstream reader is easy to cause redundant. detail apache/iceberg-rust#398
We can refactor to use the upstream interface when the upstream PR merge. apache/iceberg-rust#401
There was a problem hiding this comment.
Can you comment the reason for fork in Cargo.toml?
ZENOTME
force-pushed
the
zj/update_iceberg_1
branch
2 times, most recently
from
71974f0 to
e93dca0
Compare
Cargo.lock
Outdated
| @@ -238,6 +238,29 @@ dependencies = [ | |||
| "backtrace", | |||
| ] | |||
|
|
|||
| [[package]] | |||
| name = "apache-avro" | |||
There was a problem hiding this comment.
We now have two different versions of avro. 😕
There was a problem hiding this comment.
I think this one is used in iceberg-rust and it will not expose the interface about apache-avro.
There was a problem hiding this comment.
We have three, another one in icelake
ZENOTME
force-pushed
the
zj/update_iceberg_1
branch
4 times, most recently
from
e958d2e to
c3ad081
Compare
ZENOTME
force-pushed
the
zj/update_iceberg_1
branch
from
c3ad081 to
eca2f64
Compare
|
@BugenZhao @xxchan Any concerns on the |
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
iceberg-rust now is the iceberg rust SDK maintained by upstream. It's under more active development now and we will graduate migrate to it from icelake gradually.
For now, iceberg-rust has powerful support for reading ability. The #17277 is aimed at introducing iceberg-rust and using it in our batch iceberg scan operator.
This PR is the first part of #17277. This PR:
the upstream is not plan to support storage catalog so we need to implement it in rw.
Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.