https://github.com/masroorahmed402/distributed-task-queue
Scalable distributed task queue built with Node.js, PostgreSQL, and Redis supporting asynchronous processing, retries, priority queues, and multi-worker execution.
https://github.com/masroorahmed402/distributed-task-queue
asynchronous-processing backend distributed-systems express nodejs postgresql redis scalable-systems system-design task-queue worker-architecture
Last synced: 7 days ago
JSON representation
Scalable distributed task queue built with Node.js, PostgreSQL, and Redis supporting asynchronous processing, retries, priority queues, and multi-worker execution.
- Host: GitHub
- URL: https://github.com/masroorahmed402/distributed-task-queue
- Owner: masroorahmed402
- Created: 2026-04-28T03:00:14.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2026-04-28T03:18:27.000Z (about 2 months ago)
- Last Synced: 2026-04-28T05:10:23.675Z (about 2 months ago)
- Topics: asynchronous-processing, backend, distributed-systems, express, nodejs, postgresql, redis, scalable-systems, system-design, task-queue, worker-architecture
- Language: JavaScript
- Homepage:
- Size: 964 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Distributed Task Queue System
A scalable backend system built using Node.js, PostgreSQL, and Redis to process asynchronous tasks using a distributed worker architecture.
## Features
- REST API for task submission
- Redis-based queue for fast task dispatch
- PostgreSQL for persistent storage
- Multiple workers for parallel processing
- Retry logic with exponential backoff
- Priority-based task scheduling (HIGH, MEDIUM, LOW)
## Tech Stack
- Node.js (Express)
- PostgreSQL
- Redis
## How to Run
1. Install dependencies:
npm install
2. Start PostgreSQL and Redis
3. Run the server:
node server.js
4. Run workers (in separate terminals):
node worker.js
node worker.js
## API Example
Create a task:
curl -X POST http://localhost:3000/tasks \
-H "Content-Type: application/json" \
-d '{"type":"EMAIL","payload":{"message":"hello"},"priority":"HIGH"}'
## Architecture
Client → API → PostgreSQL → Redis Queue → Workers → Status Updates
## Key Concepts
- Distributed systems
- Asynchronous processing
- Queue-based architecture
- Fault tolerance and retries
- Horizontal scaling with multiple workers
## Author
Mohammed Masroor Ahmed