https://github.com/kasyap1234/distributed-task-queue
A highly scalable distributed task queue system in Go, utilizing Redis for job storage, PostgreSQL for result persistence, and concurrent workers with exponential backoff for efficient job processing and scheduling.
https://github.com/kasyap1234/distributed-task-queue
distributed-systems go golang redis
Last synced: 4 months ago
JSON representation
A highly scalable distributed task queue system in Go, utilizing Redis for job storage, PostgreSQL for result persistence, and concurrent workers with exponential backoff for efficient job processing and scheduling.
- Host: GitHub
- URL: https://github.com/kasyap1234/distributed-task-queue
- Owner: kasyap1234
- Created: 2024-07-29T16:33:09.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2024-08-07T05:14:00.000Z (11 months ago)
- Last Synced: 2025-01-21T11:44:55.683Z (6 months ago)
- Topics: distributed-systems, go, golang, redis
- Language: Go
- Homepage:
- Size: 24.4 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Distributed Task Queue
A robust, scalable distributed task queue system implemented in Go, designed to handle various types of jobs efficiently.
## Features
- Concurrent job processing
- Redis-based queue for job storage and retrieval
- Job scheduling for recurring tasks
- RESTful API for job submission
- Configurable worker count and timeout
- Exponential backoff strategy for efficient queue polling
- Support for multiple job types (email, image resizing, data processing)
- Persistent storage for job results
## Components
### Worker (worker.go)
The worker component is responsible for processing jobs from the queue. It supports:
- Concurrent job processing
- Custom job type handlers
- Exponential backoff for efficient queue polling
### Scheduler (scheduler.go)
The scheduler allows for the creation of recurring jobs. It uses a cron-like syntax for job scheduling.
### Queue (queue.go)
Implements a Redis-based queue for job storage and retrieval. Supports operations like:
- Enqueue
- Dequeue
- Update job status
### Job (job.go)
Defines the structure for different job types, including:
- Email jobs
- Image resizing jobs
- Data processing jobs
### Handlers (handler.go)
Contains the logic for processing different job types:
- Email sending
- Image resizing
- Data processing
### Storage (storage.go)
Manages persistent storage of job results using PostgreSQL.
### Config (config.go)
Handles application configuration using Viper, allowing for easy management of:
- Redis connection details
- Worker count and timeout
- API server address
## Getting Started
1. Ensure Redis and PostgreSQL are installed and running.
2. Set up the configuration in `config.go` or use environment variables.
3. Run the main application:
4. The API server will start, and you can submit jobs via POST requests to `/submit`.
## API Usage
Submit a job:
POST /submit Content-Type: application/json
{ "type": "email", "payload": { "to": "[email protected]", "subject": "Test Email", "body": "This is a test email." } }
## Configuration
The application can be configured using environment variables:
- `REDISADDR`: Redis server address (default: "127.0.0.1:6379")
- `MAXRETRIES`: Maximum job retry attempts (default: 5)
- `WORKERCOUNT`: Number of concurrent workers (default: 10)
- `APIADDR`: API server address (default: "127.0.0.1:8080")
- `WORKERTIMEOUT`: Worker timeout in seconds (default: 10)
## Extending the System
To add new job types:
1. Define the job structure in `job.go`
2. Implement the job handler in `handler.go`
3. Register the new handler in `main.go`
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
## License
This project is licensed under the MIT License.