https://github.com/siddhant-k-code/scalable-ls
Rust implementation of: http://be-n.com/spw/you-can-list-a-million-files-in-a-directory-but-not-with-ls.html
https://github.com/siddhant-k-code/scalable-ls
linux list-files ls rust rust-lang scaleable
Last synced: 11 months ago
JSON representation
Rust implementation of: http://be-n.com/spw/you-can-list-a-million-files-in-a-directory-but-not-with-ls.html
- Host: GitHub
- URL: https://github.com/siddhant-k-code/scalable-ls
- Owner: Siddhant-K-code
- Created: 2024-03-08T12:12:41.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-03-08T13:10:24.000Z (almost 2 years ago)
- Last Synced: 2025-03-25T21:14:05.268Z (11 months ago)
- Topics: linux, list-files, ls, rust, rust-lang, scaleable
- Language: Rust
- Homepage: https://x.com/Siddhant_K_code/status/1766090204241162311?s=20
- Size: 59.6 KB
- Stars: 7
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# scalable-ls
> Rust implementation of [this blog or discussion](http://be-n.com/spw/you-can-list-a-million-files-in-a-directory-but-not-with-ls.html)
## Introduction
`scalable-ls` is a Rust-based command-line tool designed to list the contents of directories containing an extremely large number of files. Traditional tools like `ls` can struggle or fail in these environments, but `scalable-ls` is optimized to handle such scenarios efficiently.
## Installation
Ensure you have Rust and Cargo installed on your system. Clone this repository and navigate to the project directory. Build the project using Cargo:
```sh
cargo build --release
```
## Usage
### Create empty files
To create a directory and populate it with empty files under `demo-files` directory, use the following command:
```sh
./create_files.sh
```
To list the files in a directory, use the following command:
```sh
cargo run -- [OPTIONS]
```
Options:
- `-b, --buf_size `: Set the buffer size for reading directory entries (default is 5MB).
Example:
```sh
cargo run -- ./demo_files/
```
```sh
cargo run -- --buf-size 1000000 ./demo_files/
```
## Features
- Handles directories with millions of files.
- Efficient directory listing without hanging or crashing.
- Customizable buffer size for performance tuning.
## Example
For 20000 files, both seem to be working. But, it could fail for larger number of files & also output is inconsistent in native `ls` in case of large numbers of files:
