https://github.com/f5io/pool
A highly flexible process pooling library for Node.js
https://github.com/f5io/pool
concurrency csp nodejs pooling
Last synced: 2 months ago
JSON representation
A highly flexible process pooling library for Node.js
- Host: GitHub
- URL: https://github.com/f5io/pool
- Owner: f5io
- License: mit
- Created: 2017-09-03T17:47:06.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2022-03-26T21:32:01.000Z (over 3 years ago)
- Last Synced: 2023-08-01T05:59:10.601Z (about 2 years ago)
- Topics: concurrency, csp, nodejs, pooling
- Language: TypeScript
- Homepage:
- Size: 427 KB
- Stars: 19
- Watchers: 11
- Forks: 1
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# @f5io/pool
A highly flexible process pooling library for Node.js. Built with [@f5io/csp](https://github.com/f5io/csp).
[](https://badge.fury.io/js/%40f5io%2Fpool)
## Installation
```
$ npm install --save @f5io/pool
```
or
```
$ yarn add @f5io/pool
```
## API
This library exposes a single factory method for creating pools.
### `createPool({ poolSize = 5, createProcess, createAsyncProcess, handler })` -> `Pool|Promise`
The pool factory takes an options object containing 3 of 4 properties:
- `poolSize` - defaults to 5, determines the size of the pool
- `createProcess` - defines a `process` factory function which can return anything
- `createAsyncProcess` - defines an async `process` factory which can return anything, useful if your process requires time to become active.
- `handler(process, input)` -> `Promise` - defines a function which handles a unit of work. The handler must return a `Promise` and receives a `process` (as defined by the `process` factory) and the `input` from a call to `run` on the pool
You must supply only one of `createProcess` or `createAsyncProcess`! If you supply `createAsyncProcess` the return value of the `createPool` factory will be a `Promise`.
A returned `Pool` exposes 2 methods:
- `Pool.run(input)` -> `Promise` - The interface defined to run against the pool of processes, supplied input can be of any type as the handler supplied at `pool` creation defines how the input interacts which the underlying process
- `Pool.close` -> `Promise` - A mechanism for destroying the `pool` when it is no longer needed
## Example Usage
### Network I/O parallelisation
By defining our process as a plain `Symbol`, or `true` for that matter, we can limit request parallelisation to the size of the pool.
```javascript
const assert = require('assert');
const fetch = require('node-fetch');
const createPool = require('@f5io/pool');
const { run, close } = createPool({
poolSize: 2,
createProcess: () => Symbol('process'),
handler: (_, query) => {
console.log(`🚀 running request with query: ${query}`);
return fetch(`https://postman-echo.com/get?q=${query}`)
.then(res => res.json())
.then(res => assert.equal(res.args.q, query));
.then(_ => console.log(`👌 request completed successfully`));
}
});
(async () => {
const queries = Array.from({ length: 20 }, (_, i) => run(`${++i}`));
await Promise.all(queries);
close();
})();
```
### Child process pooling
For spawning multiple child processes and spreading work across processes in the pool.
```javascript
const assert = require('assert');
const { spawn } = require('child_process');
const createPool = require('@f5io/pool');
const { run, close } = createPool({
poolSize: 10,
createProcess: () => {
const p = spawn('cat', [ '-' ]);
p.stdin.setEncoding('utf-8');
p.stdout.setEncoding('utf-8');
return p;
},
handler: (p, input) =>
new Promise(resolve => {
p.stdout.once('data', d => {
assert(d, input);
console.log(`👌 received data: ${d.trim()} from pid: ${p.pid}`);
resolve(d);
});
console.log(`🚀 sending data: ${input.trim()} to pid: ${p.pid}`);
p.stdin.write(input);
}),
});
(async () => {
const inputs = Array.from({ length: 100 }, (_, i) => run(`${++i}\n`));
await Promise.all(inputs);
close();
})();
```
## Contributions
Contributions are welcomed and appreciated!
1. Fork this repository.
1. Make your changes, documenting your new code with comments.
1. Submit a pull request with a sane commit message.
Feel free to get in touch if you have any questions.
## License
Please see the `LICENSE` file for more information.