Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/allegro/node-worker-nodes
A node.js library to run cpu-intensive tasks in a separate processes and not block the event loop.
https://github.com/allegro/node-worker-nodes
cluster hacktoberfest nodejs processes workers
Last synced: about 22 hours ago
JSON representation
A node.js library to run cpu-intensive tasks in a separate processes and not block the event loop.
- Host: GitHub
- URL: https://github.com/allegro/node-worker-nodes
- Owner: allegro
- License: apache-2.0
- Created: 2016-12-19T12:31:12.000Z (almost 8 years ago)
- Default Branch: main
- Last Pushed: 2024-09-17T13:38:14.000Z (about 2 months ago)
- Last Synced: 2024-11-06T01:15:38.418Z (8 days ago)
- Topics: cluster, hacktoberfest, nodejs, processes, workers
- Language: JavaScript
- Homepage:
- Size: 431 KB
- Stars: 492
- Watchers: 15
- Forks: 33
- Open Issues: 12
-
Metadata Files:
- Readme: Readme.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://www.npmjs.org/package/worker-nodes)
[](https://github.com/allegro/node-worker-nodes/actions/workflows/ci.yml)
# worker-nodes
A node.js library to run cpu-intensive tasks in a separate processes and to not to block the event loop.
## Installation
```bash
$ npm install worker-nodes
```
Node.js greater than 14.0.0 is *required*
# API Reference
## WorkerNodes
**Kind**: global class
* [WorkerNodes](#WorkerNodes)
* [new WorkerNodes(path, [options])](#new_WorkerNodes_new)
* [.call](#WorkerNodes+call) : Proxy
* [.ready()](#WorkerNodes+ready) ⇒ Promise
* [.terminate()](#WorkerNodes+terminate) ⇒ Promise
* [.profiler(duration)](#WorkerNodes+profiler) ⇒ void
* [.takeSnapshot()](#WorkerNodes+takeSnapshot) ⇒ void
* [.getUsedWorkers()](#WorkerNodes+getUsedWorkers) ⇒ Array.<Worker>
### new WorkerNodes(path, [options])
| Param | Type | Description |
| --- | --- | --- |
| path | String | An absolute path to the module that will be run in the workers. |
| [options] | Object | See [WorkerNodesOptions](#WorkerNodesOptions) for a detailed description. |
### workerNodes.call : Proxy
This exposes the api of a module that the worker nodes are working on. If the module is a function, you
can call this directly. If the module exports multiple functions, you can call them as they were properties
of this proxy.
**Kind**: instance property of [WorkerNodes](#WorkerNodes)
### workerNodes.ready() ⇒ Promise
A method to check if the minimum required number of workers are ready to serve the calls.
**Kind**: instance method of [WorkerNodes](#WorkerNodes)
**Returns**: Promise - resolves with a [WorkerNodes](#WorkerNodes) instance
### workerNodes.terminate() ⇒ Promise
Starts the process of terminating this instance.
**Kind**: instance method of [WorkerNodes](#WorkerNodes)
**Returns**: Promise - - resolved when the instance is terminated.
### workerNodes.profiler(duration) ⇒ void
Run CPU Profiler and save result on main process directory
**Kind**: instance method of [WorkerNodes](#WorkerNodes)
| Param | Type |
| --- | --- |
| duration | number |
### workerNodes.takeSnapshot() ⇒ void
Take Heap Snapshot and save result on main process directory
**Kind**: instance method of [WorkerNodes](#WorkerNodes)
### workerNodes.getUsedWorkers() ⇒ Array.<Worker>
Return list with used workers in pool
**Kind**: instance method of [WorkerNodes](#WorkerNodes)
## WorkerNodesOptions
Describes a WorkerNodes options.
**Kind**: global class
* [WorkerNodesOptions](#WorkerNodesOptions)
* [.autoStart](#WorkerNodesOptions+autoStart) : Boolean
* [.lazyStart](#WorkerNodesOptions+lazyStart) : Boolean
* [.asyncWorkerInitialization](#WorkerNodesOptions+asyncWorkerInitialization) : Boolean
* [.minWorkers](#WorkerNodesOptions+minWorkers) : Number
* [.maxWorkers](#WorkerNodesOptions+maxWorkers) : Number
* [.maxTasks](#WorkerNodesOptions+maxTasks) : Number
* [.maxTasksPerWorker](#WorkerNodesOptions+maxTasksPerWorker) : Number
* [.taskTimeout](#WorkerNodesOptions+taskTimeout) : Number
* [.taskMaxRetries](#WorkerNodesOptions+taskMaxRetries) : Number
* [.workerEndurance](#WorkerNodesOptions+workerEndurance) : Number
* [.workerStopTimeout](#WorkerNodesOptions+workerStopTimeout) : Number
* [.resourceLimits](#WorkerNodesOptions+resourceLimits) : Object
* [.workerType](#WorkerNodesOptions+workerType) : string
### options.autoStart : Boolean
Whether should initialize the workers before a first call.
If true, depending on the [lazyStart](#WorkerNodesOptions+lazyStart) option, it will start the
[min](#WorkerNodesOptions+minWorkers) or [max](#WorkerNodesOptions+maxWorkers) number of workers.
**Kind**: instance property of [WorkerNodesOptions](#WorkerNodesOptions)
**Default**: false
### options.lazyStart : Boolean
Whether should start a new worker only if all the others are busy.
**Kind**: instance property of [WorkerNodesOptions](#WorkerNodesOptions)
**Default**: false
### options.asyncWorkerInitialization : Boolean
Enables async initialization of worker.
To start handling task over worker, need to invoke `sendWorkerMessage('ready')` function when it fully initialized.
For examples please refer to [the test cases](https://github.com/allegro/node-worker-nodes/blob/master/e2e/async-initialization.spec.js)
**Kind**: instance property of [WorkerNodesOptions](#WorkerNodesOptions)
**Default**: false
### options.minWorkers : Number
The minimum number of workers that needs to be running to consider the whole pool as operational.
**Kind**: instance property of [WorkerNodesOptions](#WorkerNodesOptions)
**Default**: 0
### options.maxWorkers : Number
The maximum number of workers that can be running at the same time.
Defaults to the number of cores the operating system sees.
**Kind**: instance property of [WorkerNodesOptions](#WorkerNodesOptions)
### options.maxTasks : Number
The maximum number of calls that can be handled at the same time.
Exceeding this limit causes MaxConcurrentCallsError to be thrown.
**Kind**: instance property of [WorkerNodesOptions](#WorkerNodesOptions)
**Default**: Infinity
### options.maxTasksPerWorker : Number
The number of calls that can be given to a single worker at the same time.
**Kind**: instance property of [WorkerNodesOptions](#WorkerNodesOptions)
**Default**: 1
### options.taskTimeout : Number
The number milliseconds after which a call is considered to be lost.
Exceeding this limit causes TimeoutError to be thrown and a worker that performed that task to be killed.
**Kind**: instance property of [WorkerNodesOptions](#WorkerNodesOptions)
**Default**: Infinity
### options.taskMaxRetries : Number
The maximum number of retries that will be performed over a task before reporting it as incorrectly terminated.
Exceeding this limit causes ProcessTerminatedError to be thrown.
**Kind**: instance property of [WorkerNodesOptions](#WorkerNodesOptions)
**Default**: 0
### options.workerEndurance : Number
The maximum number of calls that a single worker can handle during its whole lifespan.
Exceeding this limit causes the termination of the worker.
**Kind**: instance property of [WorkerNodesOptions](#WorkerNodesOptions)
**Default**: Infinity
### options.workerStopTimeout : Number
The timeout value (in milliseconds) for the worker to stop before sending SIGKILL.
**Kind**: instance property of [WorkerNodesOptions](#WorkerNodesOptions)
**Default**: 100
### options.resourceLimits : Object
Provides the set of JS engine resource constraints inside this Worker thread. (Usable when using `workerType: thread` only)
**Kind**: instance property of [WorkerNodesOptions](#WorkerNodesOptions)
**Properties**
| Name | Type | Description |
| --- | --- | --- |
| maxYoungGenerationSizeMb | Number | The maximum size of a heap space for recently created objects |
| maxOldGenerationSizeMb | Number | The maximum size of the main heap in MB |
| codeRangeSizeMb | Number | The size of a pre-allocated memory range used for generated code |
| stackSizeMb | Number | The default maximum stack size for the thread. Small values may lead to unusable Worker instances |
### options.workerType : string
Can be either `process` or `thread` (default), that controls the underlying implementation used, either `child_process` or `worker_threads`.
Most usecases are perfectly fine with `thread` implementation, some work loads though, might need to use `process`, for example, if you are using
`process.chdir()` call which is [not supported](https://github.com/nodejs/node/issues/41673) in `worker_threads`.
## Example
Given `/home/joe.doe/workspace/my-module.js`:
```javascript
module.exports = function myTask() {
return 'hello from separate process!';
};
```
you can run it through the worker nodes as follows:
```javascript
const WorkerNodes = require('worker-nodes');
const myModuleWorkerNodes = new WorkerNodes('/home/joe.doe/workspace/my-module');
myModuleWorkerNodes.call().then(msg => console.log(msg)); // -> 'hello from separate process!'
```
For more advanced examples please refer to [the test cases](https://github.com/allegro/node-worker-nodes/tree/main/e2e).
## Running tests
Check out the library code and then:
```bash
$ npm install
$ npm test
```
## Benchmarks
To run tests, type:
```bash
$ npm install
$ npm run benchmark
```
It will run a performance test against the selected libraries:
- data in: an object that consists of a single field that is a 0.5MB random string
- data out: received object stringified and concatenated with another 1MB string
Example results:
```bash
results for 100 executions
name time: total [ms] time usr [ms] time sys [ms] worker usr [ms] worker sys [ms] mem rss [MB] worker rss [MB] errors
------------------ ---------------- ------------- ------------- --------------- --------------- ------------ --------------- ------
no-workers 148 203 37 0 0 98 0 0
[email protected] 362 390 143 389 143 213 210 0
[email protected] 367 495 185 492 182 236 245 0
[email protected] 1095 520 207 592 243 216 86 0
[email protected] 1886 749 276 947 299 221 70 0
[email protected] 2002 847 285 986 309 219 74 0
[email protected] 13775 7129 5236 1891 952 363 63 0
os : Darwin / 19.5.0 / x64
cpu : Intel(R) Core(TM) i7-7660U CPU @ 2.50GHz × 4
node : 14.3.0 / v8: 8.1.307.31-node.33
```
## See also
sources of inspiration:
* [Worker Farm](https://github.com/rvagg/node-worker-farm)
* [workerpool](https://github.com/josdejong/workerpool)
* [process-pool](https://github.com/ohjames/process-pool)
* [node-worker-pool](https://github.com/jeffmo/node-worker-pool)
## License
Copyright Allegro Sp. z o.o.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.