feat(connector): support local fs source

[KeXiangWang]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

As title. let keep it in stealth and only used for test.
Posix fs source should only be used for testing.
For a single-CN cluster, the behavior is well-defined. It will read from the local file system.
For a multi-CN cluster, each CN will read from its own local file system under the given directory.

There are two parameters needed:

root: String, The root directly of the files to search. The files will be searched recursively.
match_pattern: Option<String>, The regex pattern to match files under root directory.

An example:

CREATE TABLE diamonds (
    carat FLOAT,
    cut TEXT,
    color TEXT,
    depth FLOAT,
) WITH (
  connector = 'posix_fs',
  match_pattern = 'data*.csv',
  posix_fs.root = '~/downloads',
) FORMAT PLAIN ENCODE CSV ( without_header = 'false', delimiter = ',' );

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

[KeXiangWang]

The main-cron test of local_fs has passed. Check details here.

[tabVersion]

so we can deny the prefix in props?

[wcy-fdu]

If this fs source is only used in testing, it's ok to ignore the miss of prefix list.
BTW, I'm not aware that OpenDAL Fs engine does not support prefix list. cc @Xuanwo

[KeXiangWang]

The prefix is not in props now. In other implemenataion, it's extracted from the match_pattern(eg, the prefix of abc*.csv is abc) and used for filtering out files with a different prefix under the directory/bucket. Since Opendal does not support this listing with prefix correctly, we simply set the prefix to empty. If the problem in opendal is solved, we can then generate the prefix here and return it, it benefits to performance.

[Xuanwo]

BTW, I'm not aware that OpenDAL Fs engine does not support prefix list. cc @Xuanwo

opendal 0.44 should fix it.

As I mentioned before, the prefix support for fs is simulated: we will list the parent dir and return the matched prefix. Since fs doesn't have prefix list support natively.

[Xuanwo]

Join the conversation on adding glob support to opendal – your input would be valuable: apache/opendal#3500

[tabVersion]

The pr LGTM as long as it satisfies your testing requirements. also please some doc for all accepted props.

[wcy-fdu]

LGTM as this change is only used for test.

[wcy-fdu]

If this fs source is only used in testing, it's ok to ignore the miss of prefix list.
BTW, I'm not aware that OpenDAL Fs engine does not support prefix list. cc @Xuanwo