https://github.com/tiagogrosso/ts-basic-cluster
Run multiple async tasks in parallel
https://github.com/tiagogrosso/ts-basic-cluster
async cluster lib nodejs parallel pooling typescript
Last synced: about 2 months ago
JSON representation
Run multiple async tasks in parallel
- Host: GitHub
- URL: https://github.com/tiagogrosso/ts-basic-cluster
- Owner: TiagoGrosso
- License: mit
- Created: 2023-12-01T19:53:46.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2025-09-11T09:11:36.000Z (6 months ago)
- Last Synced: 2025-09-11T12:33:20.621Z (6 months ago)
- Topics: async, cluster, lib, nodejs, parallel, pooling, typescript
- Language: TypeScript
- Homepage: https://www.npmjs.com/package/basic-cluster
- Size: 811 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Basic Cluster
[](https://www.npmjs.org/package/basic-cluster)
[](https://codecov.io/gh/TiagoGrosso/ts-basic-cluster)
[](https://packagephobia.com/result?p=basic-cluster)
[](http://npm-stat.com/charts.html?package=basic-cluster)
A library for performing multiple tasks in parallel with control over resource
usage.
## Installation
```
npm install basic-cluster
```
## Concepts
### Cluster
The cluster controls the execution of multiple async tasks. It has a size which
will control how many [instances](#instances) it can manage.
The cluster also accepts options for how it retries obtaining an instance to run
submitted tasks. Retries are managed via
[exponential-backoff](https://github.com/coveooss/exponential-backoff) so check
the docs over there for options.
### Instance
An instance is simply an object managed by the cluster and acts as a context for
tasks to run in. While running a task, an instance is considered `busy` and it
cannot accept a new task until the current one completes.
### Task
Task is what you submit to the cluster for it to run when possible.
## Usage
Here's what you do:
1. Create a Cluster.
1. Submit tasks to it.
1. When you are done, you can shutdown the cluster.
### Basic usage
The `BasicCluster` class makes it easy to run a bunch of parallel tasks which
don't depend on a managed instance object.
```typescript
import { BasicCluster } from './src/cluster/BasicCluster';
const clusterSize = 3;
const cluster: BasicCluster = new BasicCluster(clusterSize);
const result = cluster.submit(async () => {
// Do something
});
```
### Shutdown
If you no longer require the cluster, you can shut it down. There are two
options for this:
1. `Cluster#shutdown()` will attempt to wait for any running task to complete
before shutting down.
1. After shutdown is requested, new task submissions are immediately
rejected. Tasks waiting to be picked up will also be rejected.
1. If there are running tasks, the cluster will retry to shutdown at a later
time.
1. When gracefully shutdown retries are exhausted, the cluster will
forcefully shutdown.
1. `Cluster#shutdownNow()` will forcefully shutdown the cluster, calling
`shutdown()` on all its instances immediately.
1. This does not cancel running tasks, so depending on how their built and
what stage their in, they might still complete successfully.
### Usage with instances
You can create clusters that use (potentially complex) instance objects, reusing
them for new tasks. The example that inspired this package was a cluster of
[Puppeteer](https://www.npmjs.com/package/puppeteer) browser instances, which
take some time to initiate and, as such, are a prime candidate for pooling.
To do so, you can implement the `Instance` interface and use the `Cluster` class
directly with your instance.
### SimpleInstance
The `SimpleInstance` is a utility for when you need an instance with some state
and the shutdown is a no-op. Here's an example:
```typescript
let i = 0;
const cluster: Cluster = new Cluster(3, () => new SimpleInstance(++i));
cluster.submit((instance) => {
console.log(`Running task on instance ${instance.getValue()}`);
// do something with the instance
});
```