Rust implementation of this blog or discussion
scalable-ls is a Rust-based command-line tool designed to list the contents of directories containing an extremely large number of files. Traditional tools like ls can struggle or fail in these environments, but scalable-ls is optimized to handle such scenarios efficiently.
Ensure you have Rust and Cargo installed on your system. Clone this repository and navigate to the project directory. Build the project using Cargo:
cargo build --releaseTo create a directory and populate it with empty files under demo-files directory, use the following command:
./create_files.shTo list the files in a directory, use the following command:
cargo run -- [OPTIONS] <PATH>Options:
-b, --buf_size <BUF_SIZE>: Set the buffer size for reading directory entries (default is 5MB).
Example:
cargo run -- ./demo_files/cargo run -- --buf-size 1000000 ./demo_files/- Handles directories with millions of files.
- Efficient directory listing without hanging or crashing.
- Customizable buffer size for performance tuning.
For 20000 files, both seem to be working. But, it could fail for larger number of files & also output is inconsistent in native ls in case of large numbers of files:
