https://github.com/not-empty/hydra-publisher
High Availability Concurrency Publisher
https://github.com/not-empty/hydra-publisher
Last synced: 2 months ago
JSON representation
High Availability Concurrency Publisher
- Host: GitHub
- URL: https://github.com/not-empty/hydra-publisher
- Owner: not-empty
- Created: 2024-07-02T19:10:57.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-07-17T17:48:17.000Z (almost 2 years ago)
- Last Synced: 2025-04-12T05:36:55.043Z (about 1 year ago)
- Language: JavaScript
- Size: 85 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.MD
Awesome Lists containing this project
README
# Hydra Async Pool Pulisher
[](http://makeapullrequest.com)
**Hydra Async Pool** is an open-source project designed to manage and limit asynchronous concurrency in high availability and high resource-demanding environments. This tool is essential for applications that need to restrict the number of simultaneous executions of certain tasks due to rate limits or resource constraints.
## Introduction
In high-demand environments, managing the execution of asynchronous concurrent tasks is crucial and hard. Whether dealing with services that impose rate limits, databases with connection limits, or systems with constrained resources, you need a solution that can handle asynchronous concurrency gracefully. **Hydra Async Pool** provides a robust and flexible way to manage and limit the number of concurrent asynchronous jobs, ensuring your system remains stable and performs optimally.
## Features
- **Concurrency Limiting**: Define the maximum number of asynchronous concurrent jobs to prevent brake rate limits or resource exhaustion.
- **Redis Integration**: Uses Redis for efficient and reliable job queue management.
- **Job States Management**: Track and manage job states, including pending, executing, and finished jobs.
- **Real-time Updates**: Subscribe to job updates for real-time monitoring and control.
- **Scalability**: Designed to scale with your application's needs, making it suitable for both small and large deployments.
## Hydra Async Pool
You will need to setup and install the **Hydra Async Pool** from [here](https://github.com/not-empty/hydra.
## instalation
Install the publisher in all your projects that need to publish and finish jobs
```sh
npm install hydra-publisher
```
## Publish/Finish
Now just publish and finish jobs as you want to, The **Hydra Async Pool** will manage the concurrence and assures that you always have your maximum limit respected.
The Hydra Publisher handles the communication between job publishers, worker nodes, and the job manager. By publishing new jobs and reporting finished jobs, the manager can effectively orchestrate job pooling and ensure that concurrency limits are respected
## Usage
## Publishing
```javascript
import { HydraPublisher } from 'hydra-publisher';
const publisher = new HydraPublisher({ redisOptions: { url: 'redis://localhost:6379' } });
const jobId = await publisher.addJob({ task: 'exampleTask' });
console.log(`Job added with ID: ${jobId}`);
```
## Finishing
```javascript
import { HydraPublisher } from 'hydra-publisher';
const publisher = new HydraPublisher({ redisOptions: { url: 'redis://localhost:6379' } });
// After completing the job
await publisher.finishJob(jobId);
console.log(`Job ${jobId} finished`);
```
By following this setup, the Hydra Async Pool will be able to orchestrate the job pooling, manage concurrency, and ensure efficient job execution across your system.
**Not Empty Foundation - Free codes, full minds**