{"id":27998624,"url":"https://github.com/nadimnesar/distributed-job-queue","last_synced_at":"2026-02-23T00:07:23.379Z","repository":{"id":279948884,"uuid":"940438623","full_name":"nadimnesar/distributed-job-queue","owner":"nadimnesar","description":"Distributed job queue system using Redis and Spring Boot, designed to efficiently distribute computational tasks across multiple worker nodes.","archived":false,"fork":false,"pushed_at":"2025-04-10T19:18:11.000Z","size":315,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"dev","last_synced_at":"2025-05-08T22:54:21.582Z","etag":null,"topics":["docker","java","liquibase","nginx-load-balancer","postgresql","redis-queue","spring-boot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nadimnesar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-28T07:11:20.000Z","updated_at":"2025-04-12T07:49:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"5f68f9bb-361f-4b5d-954e-d6fddf1a7800","html_url":"https://github.com/nadimnesar/distributed-job-queue","commit_stats":null,"previous_names":["nadimnesar/distributed-job-queue-system-with-redis","nadimnesar/distributed-job-queue"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nadimnesar%2Fdistributed-job-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nadimnesar%2Fdistributed-job-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nadimnesar%2Fdistributed-job-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nadimnesar%2Fdistributed-job-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nadimnesar","download_url":"https://codeload.github.com/nadimnesar/distributed-job-queue/tar.gz/refs/heads/dev","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253160814,"owners_count":21863625,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","java","liquibase","nginx-load-balancer","postgresql","redis-queue","spring-boot"],"created_at":"2025-05-08T22:54:25.068Z","updated_at":"2026-02-23T00:07:23.345Z","avatar_url":"https://github.com/nadimnesar.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Distributed Job Queue System with Redis and Spring Boot\n\n## Overview\n\nThis project implements a scalable distributed job queue system using Redis and Spring Boot, designed to efficiently\ndistribute computational tasks across multiple worker nodes. The system provides robust job tracking, fault tolerance,\nand handles job dependencies and failures gracefully.\n\n### Features\n\n- Deploy multiple worker and producer nodes across different machines.\n- Use Nginx to load balance producer requests.\n- Enqueue jobs with priorities for efficient task processing.\n- Use Directed Acyclic Graph (DAG) based dependency management to ensure proper execution order.\n- Track job statuses and support job cancellation.\n- Support proper distributed locking to prevent duplicate job processing.\n- Support virtual threads for concurrent job execution.\n- Automatic retry mechanism for failed jobs.\n- Dead Letter Queue (DLQ) for unprocessable jobs and support revive dead jobs.\n- Monitoring APIs for worker system metrics.\n\n## System Design\n\n```mermaid\nflowchart TD\n    User((User)) --\u003e Frontend[Frontend UI]\n    Frontend --\u003e Nginx[Load Balancer: Nginx]\n\n    subgraph Producers[Producer Service Cluster]\n        direction TB\n        Producer1[Producer 1]\n        Producer2[Producer 2]\n        Producer3[Producer 3]\n    end\n\n    Nginx --\u003e Producers\n    Producers --\u003e|Enqueue Job| Redis[(\"Redis: Job Queue\")]\n\n    subgraph Workers[Worker Cluster]\n        direction TB\n        Worker1[Worker 1]\n        Worker2[Worker 2]\n        Worker3[Worker 3]\n        Worker4[Worker 4]\n        Worker5[Worker 5]\n    end\n\n    Redis --\u003e|Consume Job| Workers\n    Workers --\u003e|Update Job| PostgreSQL[(\"PostgreSQL: Job Metadata\")]\n    Producers --\u003e|Store Job Metadata| PostgreSQL\n    Workers --\u003e Decision1{Successful?}\n    Decision1 --\u003e|Yes| Done([Done])\n    Decision1 --\u003e|No| Decision2{Limit Exceeded?}\n    Decision2 --\u003e|Yes: Move to DLQ| Redis\n    Decision2 --\u003e|No| Redis\n    \n    classDef producer fill: #8E44AD, stroke: #6C3483, color: white, stroke-width: 2px, font-weight: bold\n    classDef worker fill: #F57C00, stroke: #E65100, color: white, stroke-width: 2px, font-weight: bold\n    classDef database fill: #336791, stroke: #274472, color: white, stroke-width: 2px, font-weight: bold, stroke-dasharray: 5 2\n    classDef cache fill: #D32F2F, stroke: #B71C1C, color: white, stroke-width: 2px, font-weight: bold, stroke-dasharray: 5 2\n    classDef loadbalancer fill: #388E3C, stroke: #2E7D32, color: white, stroke-width: 2px, font-weight: bold\n    \n    class Producer1,Producer2,Producer3 producer\n    class Worker1,Worker2,Worker3,Worker4,Worker5 worker\n    class PostgreSQL database\n    class Redis cache\n    class Nginx loadbalancer\n```\n\n## Technical Specifications\n\n* **Backend**: Spring Boot\n* **Queue:** Redis\n* **Load Balancer:** Nginx\n* **Database:** PostgreSQL\n* **Database Migration:** Liquibase\n\n## Getting Started\n\n### Prerequisites\n\n* Java 21\n* Docker 20.10.13 or higher\n\n### Build and Run the Project\n\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/nadimnesar/distributed-job-queue-with-redis-and-spring.git\n   cd distributed-job-queue-with-redis-and-spring\n   ```\n\n2. Start the services:\n   ```bash\n   make up\n   ```\n\n3. Stop the services:\n   ```bash\n   make down\n   ```\n\n### API Endpoints\n\nDownload the Postman collection and environment for testing the APIs.\n\n[Postman Collection](docs/postman/distributed-job-queue.postman_collection.json)\n[Postman Environment](docs/postman/distributed-job-queue.postman_environment.json)\n\n### Logs\n\nAll logs are stored in the following directory: `docs/logs/`\n\nUse the following commands to view logs in real-time:\n\n```bash\n# View log files for all producer instances\nls docs/logs/producer-*.log\n# View logs for a specific producer instance (example: producer-062f8e6ef23)\ntail -f docs/logs/producer-062f8e6ef23.log\n```\n\n## Future Enhancements\n\n- [ ] Implement horizontal scaling for worker nodes based on queue length\n- [ ] Improve job distribution algorithm for better worker utilization\n- [ ] Implement job scheduling capabilities\n- [ ] Implement Redis Cluster for high availability\n- [ ] Add PostgreSQL replication for database redundancy\n- [ ] Add advanced monitoring with Prometheus and Grafana\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnadimnesar%2Fdistributed-job-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnadimnesar%2Fdistributed-job-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnadimnesar%2Fdistributed-job-queue/lists"}