Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/bevry-archive/readdir-cluster
Use multiple CPU cores via clusters for the fastest way to read nested directories and their stats
https://github.com/bevry-archive/readdir-cluster
cluster nodejs readdir
Last synced: about 9 hours ago
JSON representation
Use multiple CPU cores via clusters for the fastest way to read nested directories and their stats
- Host: GitHub
- URL: https://github.com/bevry-archive/readdir-cluster
- Owner: bevry-archive
- License: other
- Created: 2015-11-30T02:38:53.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2023-12-30T10:59:59.000Z (10 months ago)
- Last Synced: 2024-04-13T10:08:24.036Z (7 months ago)
- Topics: cluster, nodejs, readdir
- Language: TypeScript
- Size: 683 KB
- Stars: 6
- Watchers: 2
- Forks: 2
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Changelog: HISTORY.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# readdir-cluster
Create a cluster of workers to iterate through the filesystem
## Usage
[Complete API Documentation.](http://master.readdir-cluster.bevry.surge.sh/docs/)
```typescript
import readdirCluster, { Stat } from 'readdir-cluster'
// note that Stat is not the same as fs.Stats as it has functions removed, as it needed to be serialisable
function iterator(path: string, filename: string, stat: Stat) {
// skip directories and files that start with .
if (filename[0] === '.') return false
// do not recurse into directories
if (stat.directory) return false
}
const paths = await readdirCluster({ directory: '.', iterator })
console.log(paths)
```
### Performance
Benchmarks:
- Running `readdir-cluster .` returns 7388 files in 500ms
- Running [`readdir`](https://nodejs.org/api/fs.html#fsreaddirpath-options-callback) with `recursive: true` returns 7388 files in 100ms
```javascript
import { readdir } from 'fs'
readdir('.', { recursive: true }, (err, files) => {
if (err) console.error(err)
else if (files.length) process.stdout.write(files.join('\n') + '\n')
})
```
- Running [fdir](https://github.com/thecodrr/fdir) returns 6480 files in 100ms
```javascript
import { fdir } from 'fdir'
const api = new fdir().withBasePath().crawl(process.argv[2])
api.withPromise().then((files) => {
if (files.length) process.stdout.write(files.join('\n') + '\n')
})
```
Recommendations:
- if you target Node.js 18.7 and above, you should use [`fs.readdir`](https://nodejs.org/api/fs.html#fsreaddirpath-options-callback) with `recursive: true`
- if you target older Node.js versions, you should use [`@bevry/fs-list`](https://github.com/bevry/fs-list)
- if you target older Node.js versions and you want a stat object, use `readdir-cluster`
- if you target Nodejs 12 and above, and want a lot of customisation, use [fdir](https://github.com/thecodrr/fdir)
As for why this package exists, `readdir-cluster` was created in 2005, `recursive` was added to Node.js in 2023, and `fdir` was created in 2020. That said, there are [several issues](https://github.com/bevry/readdir-cluster/issues) that could potentially improve `readdir-cluster` performance.
## Install
### [npm](https://npmjs.com 'npm is a package manager for javascript')
#### Install Globally
- Install: `npm install --global readdir-cluster`
- Executable: `readdir-cluster`
#### Install Locally
- Install: `npm install --save readdir-cluster`
- Executable: `npx readdir-cluster`
- Import: `import pkg from ('readdir-cluster')`
- Require: `const pkg = require('readdir-cluster').default`
### [Editions](https://editions.bevry.me 'Editions are the best way to produce and consume packages you care about.')
This package is published with the following editions:
- `readdir-cluster` aliases `readdir-cluster/index.cjs` which uses the [Editions Autoloader](https://github.com/bevry/editions 'You can use the Editions Autoloader to autoload the appropriate edition for your consumers environment') to automatically select the correct edition for the consumer's environment
- `readdir-cluster/source/index.ts` is [TypeScript](https://www.typescriptlang.org/ 'TypeScript is a typed superset of JavaScript that compiles to plain JavaScript.') source code with [Import](https://babeljs.io/docs/learn-es2015/#modules 'ECMAScript Modules') for modules
- `readdir-cluster/edition-es2022/index.js` is [TypeScript](https://www.typescriptlang.org/ 'TypeScript is a typed superset of JavaScript that compiles to plain JavaScript.') compiled against [ES2022](https://en.wikipedia.org/wiki/ES2022 'ECMAScript 2022') for [Node.js](https://nodejs.org "Node.js is a JavaScript runtime built on Chrome's V8 JavaScript engine") 14 || 16 || 18 || 20 || 21 with [Require](https://nodejs.org/dist/latest-v5.x/docs/api/modules.html 'Node/CJS Modules') for modules
- `readdir-cluster/edition-es2017/index.js` is [TypeScript](https://www.typescriptlang.org/ 'TypeScript is a typed superset of JavaScript that compiles to plain JavaScript.') compiled against [ES2017](https://en.wikipedia.org/wiki/ES2017 'ECMAScript 2017') for [Node.js](https://nodejs.org "Node.js is a JavaScript runtime built on Chrome's V8 JavaScript engine") 8 || 10 || 12 || 14 || 16 || 18 || 20 || 21 with [Require](https://nodejs.org/dist/latest-v5.x/docs/api/modules.html 'Node/CJS Modules') for modules
- `readdir-cluster/edition-es2015/index.js` is [TypeScript](https://www.typescriptlang.org/ 'TypeScript is a typed superset of JavaScript that compiles to plain JavaScript.') compiled against [ES2015](https://babeljs.io/docs/en/learn#ecmascript-2015-features 'ECMAScript 2015') for [Node.js](https://nodejs.org "Node.js is a JavaScript runtime built on Chrome's V8 JavaScript engine") 6 || 8 || 10 || 12 || 14 || 16 || 18 || 20 || 21 with [Require](https://nodejs.org/dist/latest-v5.x/docs/api/modules.html 'Node/CJS Modules') for modules
- `readdir-cluster/edition-es5/index.js` is [TypeScript](https://www.typescriptlang.org/ 'TypeScript is a typed superset of JavaScript that compiles to plain JavaScript.') compiled against ES5 for [Node.js](https://nodejs.org "Node.js is a JavaScript runtime built on Chrome's V8 JavaScript engine") 4 || 6 || 8 || 10 || 12 || 14 || 16 || 18 || 20 || 21 with [Require](https://nodejs.org/dist/latest-v5.x/docs/api/modules.html 'Node/CJS Modules') for modules
- `readdir-cluster/edition-es2017-esm/index.js` is [TypeScript](https://www.typescriptlang.org/ 'TypeScript is a typed superset of JavaScript that compiles to plain JavaScript.') compiled against [ES2017](https://en.wikipedia.org/wiki/ES2017 'ECMAScript 2017') for [Node.js](https://nodejs.org "Node.js is a JavaScript runtime built on Chrome's V8 JavaScript engine") 12 || 14 || 16 || 18 || 20 || 21 with [Import](https://babeljs.io/docs/learn-es2015/#modules 'ECMAScript Modules') for modules
- `readdir-cluster/edition-types/index.d.ts` is [TypeScript](https://www.typescriptlang.org/ 'TypeScript is a typed superset of JavaScript that compiles to plain JavaScript.') compiled Types with [Import](https://babeljs.io/docs/learn-es2015/#modules 'ECMAScript Modules') for modules
## History
[Discover the release history by heading on over to the `HISTORY.md` file.](https://github.com/bevry/readdir-cluster/blob/HEAD/HISTORY.md#files)
## Backers
### Code
[Discover how to contribute via the `CONTRIBUTING.md` file.](https://github.com/bevry/readdir-cluster/blob/HEAD/CONTRIBUTING.md#files)
#### Authors
- [Benjamin Lupton](https://balupton.com) — Accelerating collaborative wisdom.
#### Maintainers
- [Benjamin Lupton](https://balupton.com) — Accelerating collaborative wisdom.
#### Contributors
- [Benjamin Lupton](https://github.com/balupton) — [view contributions](https://github.com/bevry/readdir-cluster/commits?author=balupton 'View the GitHub contributions of Benjamin Lupton on repository bevry/readdir-cluster')
### Finances
#### Sponsors
- [Andrew Nesbitt](https://nesbitt.io) — Software engineer and researcher
- [Balsa](https://balsa.com) — We're Balsa, and we're building tools for builders.
- [Codecov](https://codecov.io) — Empower developers with tools to improve code quality and testing.
- [Poonacha Medappa](https://poonachamedappa.com)
- [Rob Morris](https://github.com/Rob-Morris)
- [Sentry](https://sentry.io) — Real-time crash reporting for your web apps, mobile apps, and games.
- [Syntax](https://syntax.fm) — Syntax Podcast
#### Donors
- [Andrew Nesbitt](https://nesbitt.io)
- [Armen Mkrtchian](https://mogoni.dev)
- [Balsa](https://balsa.com)
- [Chad](https://opencollective.com/chad8)
- [Codecov](https://codecov.io)
- [dr.dimitru](https://veliovgroup.com)
- [Elliott Ditman](https://elliottditman.com)
- [entroniq](https://gitlab.com/entroniq)
- [GitHub](https://github.com/about)
- [Hunter Beast](https://cryptoquick.com)
- [Jean-Luc Geering](https://github.com/jlgeering)
- [Michael Duane Mooring](https://mdm.cc)
- [Michael Harry Scepaniak](https://michaelscepaniak.com)
- [Mohammed Shah](https://github.com/smashah)
- [Mr. Henry](https://mrhenry.be)
- [Nermal](https://arjunaditya.vercel.app)
- [Pleo](https://pleo.io)
- [Poonacha Medappa](https://poonachamedappa.com)
- [Rob Morris](https://github.com/Rob-Morris)
- [Robert de Forest](https://github.com/rdeforest)
- [Sentry](https://sentry.io)
- [ServieJS](https://github.com/serviejs)
- [Skunk Team](https://skunk.team)
- [Syntax](https://syntax.fm)
- [WriterJohnBuck](https://github.com/WriterJohnBuck)
## License
Unless stated otherwise all works are:
- Copyright © [Benjamin Lupton](https://balupton.com)
and licensed under:
- [Artistic License 2.0](http://spdx.org/licenses/Artistic-2.0.html)