feat(batch): support file scan a directory of parquet files #17811
Conversation
chenzl25
added
the
user-facing-changes
| Err(Error::new( | ||
| ErrorKind::DataInvalid, | ||
| format!("Invalid s3 url: {}, should start with {}", dir, prefix), | ||
| ))? |
There was a problem hiding this comment.
Why it's not an assertion? Because the code says the bucket was extracted from dir
There was a problem hiding this comment.
Because we need to ensure the URL prefix is s3:// instead of something else like http://. The bucket is the URL's host and can't guarantee the protocol.
| })?; | ||
|
|
||
| if let Some(files) = files { | ||
| // if the file location is a directory, we need to remove the last argument and add all files in the directory as arguments |
There was a problem hiding this comment.
A bit dirty to list files in binder, but acceptable to me
| complete_stages.insert( | ||
| stage.id, | ||
| Arc::new(stage.clone_with_exchange_info(exchange_info, Some(1))), |
There was a problem hiding this comment.
Why we set the parallelism to 1 previously?
There was a problem hiding this comment.
Because it only supports reading one file in file_scan() previously.
| @@ -61,7 +61,7 @@ impl Executor for S3FileScanExecutor { | |||
| impl S3FileScanExecutor { | |||
| pub fn new( | |||
| file_format: FileFormat, | |||
| location: String, | |||
| file_location: Vec<String>, | |||
There was a problem hiding this comment.
I dont see an example of how to set multiple file_locations
| // The rest of the arguments are file locations | ||
| let file_location = eval_args[5..].iter().cloned().collect_vec(); |
There was a problem hiding this comment.
@tabVersion We set file_location here.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.
Example: