Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance the metadata reading during the Copy From scenarios #3196

Open
WenyXu opened this issue Jan 19, 2024 · 0 comments
Open

Enhance the metadata reading during the Copy From scenarios #3196

WenyXu opened this issue Jan 19, 2024 · 0 comments

Comments

@WenyXu
Copy link
Member

WenyXu commented Jan 19, 2024

What type of enhancement is this?

Performance

What does the enhancement do?

The file reader is designed for the sync reader. They typically support relatively cheap seek operation. However, if we try to seek on stream bytes (e.g., read bytes from S3), the seek position could be costly (re-send a new get request to S3).

I checked CSV and JSON format, and these file format reader only requires Read trait to infer a schema. Therefore, we might only need to care about the Parquet and the ORC format.

See also: #3191

Implementation challenges

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant