Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add file-level scan api for iceberg source #15783

Closed
chenzl25 opened this issue Mar 19, 2024 · 2 comments
Closed

feat: add file-level scan api for iceberg source #15783

chenzl25 opened this issue Mar 19, 2024 · 2 comments

Comments

@chenzl25
Copy link
Contributor

Is your feature request related to a problem? Please describe.

Currently, iceberg source does scan planning in the frontend node and sends the files needed to be scanned to compute nodes. Technically, we can use a file level scan API to scan those files. However, icelake lacks this API, so we need to reuse the table-level API and filter out the assigned files. I think we can add a file-level read API to icelake to avoid redundant scan planning.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

@github-actions github-actions bot added this to the release-1.8 milestone Mar 19, 2024
@chenzl25 chenzl25 removed this from the release-1.8 milestone Apr 8, 2024
@ZENOTME
Copy link
Contributor

ZENOTME commented May 29, 2024

After apache/iceberg-rust#377, I think the iceberg-rust can distribute the file scan task to different compute nodes. And the iceberg-rust has better support for reading and is under active development. I find that we can add the new interface like load_table_v2 using iceberg-rust so that we can replace the icelake implementation about read with iceberg-rust.

pub async fn load_table(&self) -> ConnectorResult<Table> {

Copy link
Contributor

github-actions bot commented Aug 1, 2024

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean.
Don't worry if you think the issue is still valuable to continue in the future.
It's searchable and can be reopened when it's time. 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants