{"id":40035157,"url":"https://github.com/amitbasuri/taskqueue-runner-go","last_synced_at":"2026-01-19T04:34:54.966Z","repository":{"id":330295060,"uuid":"1122280731","full_name":"amitbasuri/taskqueue-runner-go","owner":"amitbasuri","description":"Distributed background task processing for Go, backed by PostgreSQL and Kubernetes—retries, worker pool, locking, and real-time dashboard.","archived":false,"fork":false,"pushed_at":"2025-12-24T22:14:51.000Z","size":93,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-26T03:52:43.398Z","etag":null,"topics":["golang","kubernetes","postgresql","task-queue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amitbasuri.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-24T12:13:52.000Z","updated_at":"2025-12-24T22:20:41.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/amitbasuri/taskqueue-runner-go","commit_stats":null,"previous_names":["amitbasuri/taskqueue-runner-go"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/amitbasuri/taskqueue-runner-go","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitbasuri%2Ftaskqueue-runner-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitbasuri%2Ftaskqueue-runner-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitbasuri%2Ftaskqueue-runner-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitbasuri%2Ftaskqueue-runner-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amitbasuri","download_url":"https://codeload.github.com/amitbasuri/taskqueue-runner-go/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amitbasuri%2Ftaskqueue-runner-go/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28561842,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-19T03:31:16.861Z","status":"ssl_error","status_checked_at":"2026-01-19T03:31:15.069Z","response_time":67,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","kubernetes","postgresql","task-queue"],"created_at":"2026-01-19T04:34:54.901Z","updated_at":"2026-01-19T04:34:54.958Z","avatar_url":"https://github.com/amitbasuri.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TaskQueue-Go\n\n[![Go Version](https://img.shields.io/badge/Go-1.24-blue.svg)](https://golang.org)\n[![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)\n[![CI](https://github.com/amitbasuri/taskqueue-runner-go/actions/workflows/ci.yml/badge.svg)](https://github.com/amitbasuri/taskqueue-runner-go/actions)\n[![Go Report Card](https://goreportcard.com/badge/github.com/amitbasuri/taskqueue-runner-go)](https://goreportcard.com/report/github.com/amitbasuri/taskqueue-runner-go)\n\n\u003e A production-ready distributed background task processing system built with Go, PostgreSQL, and Kubernetes.\n\n**Key Features:**\n- ✅ Priority-based task scheduling with FIFO ordering\n- ✅ Automatic retries with exponential backoff\n- ✅ Concurrent task processing with worker pool (5 goroutines/instance)\n- ✅ Real-time monitoring dashboard with Server-Sent Events\n- ✅ Distributed lock-based task claiming (prevents duplicate execution)\n- ✅ Full Kubernetes deployment support with Helm\n- ✅ Comprehensive integration tests (11/11 passing)\n- ✅ Row-level locking with PostgreSQL FOR UPDATE SKIP LOCKED\n\n---\n\n## 📋 Table of Contents\n\n- [About](#about)\n- [Architecture](#architecture)\n- [Design Choices](#design-choices)\n- [Quick Start](#quick-start)\n- [Testing](#testing)\n- [API Reference](#api-reference)\n- [Configuration](#configuration)\n- [Contributing](#contributing)\n- [License](#license)\n\n---\n\n## 📖 About\n\nTaskQueue-Go is a production-ready distributed task queue system designed for reliability, scalability, and ease of use. Built with Go and PostgreSQL, it provides robust background job processing with automatic retries, real-time monitoring, and horizontal scalability.\n\n**Perfect for:**\n- Background job processing (emails, reports, data processing)\n- Scheduled tasks and cron-like operations\n- Distributed workflows requiring retry logic\n- Systems needing audit trails and task history\n- Microservices requiring asynchronous task processing\n\n**Why TaskQueue-Go?**\n- 🚀 **Production-Ready**: Comprehensive error handling, monitoring, and logging\n- 🔒 **Reliable**: PostgreSQL-backed with ACID guarantees and row-level locking\n- 📈 **Scalable**: Horizontal scaling with multiple worker instances\n- 🎯 **Developer-Friendly**: Clear API, excellent documentation, easy deployment\n- 🧪 **Well-Tested**: 11 comprehensive integration tests covering all scenarios\n\n---\n\n## 🏗️ Architecture\n\n### System Overview\n\n```\n┌──────────────┐         ┌──────────────────┐         ┌─────────────────┐\n│              │         │                  │         │                 │\n│  API Server  │────────▶│   PostgreSQL     │◀────────│    Workers      │\n│  (Producer)  │  POST   │  (Durable Queue) │  CLAIM  │  (Consumers)    │\n│              │  Tasks  │                  │  Tasks  │                 │\n└──────────────┘         └──────────────────┘         └─────────────────┘\n```\n\n### Components\n\n| Component | Purpose | Technology |\n|-----------|---------|------------|\n| **API Server** | REST API for task management | Go, Chi router |\n| **PostgreSQL** | Durable task queue and state storage | PostgreSQL 16 |\n| **Workers** | Execute tasks with retry logic | Go, worker pool |\n| **Dashboard** | Real-time monitoring UI | HTML/JS with SSE |\n\n### How It Works\n\n1. **Client** submits task via REST API (`POST /api/tasks`)\n2. **API Server** validates and stores task in PostgreSQL with status `queued`\n3. **Worker** polls database using `SELECT FOR UPDATE SKIP LOCKED`\n4. **Worker** claims task (sets `locked_until` timestamp)\n5. **Worker** executes task handler\n6. **Worker** updates status to `succeeded` or schedules retry if failed\n7. **Dashboard** displays real-time updates via Server-Sent Events\n\n---\n\n## 🎯 Design Choices\n\n### . Dispatcher Pattern\n\n**Problem:** Multiple workers polling database creates \"thundering herd\"\n\n**Solution:** Single dispatcher goroutine + worker pool\n\n```\nWorker Process:\n  Dispatcher (1 goroutine) ──polls DB──\u003e Claims 1 task\n       │\n       └──\u003e Buffered Channel (size 10)\n                 │\n                 └──\u003e Worker Pool (5 goroutines)\n                       ├─\u003e Worker 1\n                       ├─\u003e Worker 2\n                       ├─\u003e Worker 3\n                       ├─\u003e Worker 4\n                       └─\u003e Worker 5\n```\n\n**Benefits:**\n- **1 DB query** instead of 50 per poll cycle\n- **Buffered channel** provides backpressure\n- **No worker starvation** - dispatcher ensures fair distribution\n\n### 3. Exponential Backoff with Jitter\n\n**Formula:**\n```\nbackoff = min(2^retry_count, 2^20) seconds ± 25% jitter\nminimum = 1 second\n```\n\n**Example retry schedule:**\n- Attempt 1: ~1-1.5s delay\n- Attempt 2: ~2-3s delay  \n- Attempt 3: ~4-6s delay\n- Attempt 4: Max retries reached, task marked `failed`\n\n**Why jitter?**\n- Prevents retry storms (many tasks retrying simultaneously)\n- Spreads load over time instead of synchronized spikes\n\n### 4. Lock Expiration\n\n**Problem:** Worker crashes while holding lock → task stuck forever\n\n**Solution:** 30-second lock timeout\n\n```sql\nWHERE (status = 'queued')\n   OR (status = 'running' AND locked_until \u003c NOW())\nORDER BY\n  CASE WHEN status = 'running' THEN 0 ELSE 1 END,  -- Expired locks first\n  priority DESC,\n  created_at ASC\n```\n\n**Why prioritize expired locks?**\n- Prevents task starvation\n- Failed workers don't block the queue\n- Respects original priority after recovery\n\n### 5. SELECT FOR UPDATE SKIP LOCKED\n\n**Problem:** Multiple workers trying to claim same task\n\n**Solution:** PostgreSQL row-level locking with `SKIP LOCKED`\n\n```sql\nSELECT * FROM tasks\nWHERE status = 'queued'\nORDER BY priority DESC\nLIMIT 1\nFOR UPDATE SKIP LOCKED\n```\n\n**How it works:**\n- Worker A locks row → Worker B skips it, tries next row\n- Zero contention between workers\n- No deadlocks or retries needed\n\n---\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- Docker \u0026 Docker Compose\n- Go 1.21+ (for local development)\n- Node.js 18+ (for integration tests)\n- kubectl, kind \u0026 Helm (for Kubernetes testing)\n\n### Run with Docker Compose\n\n```bash\n# Start all services\ndocker-compose up -d\n\n# Wait for startup\nsleep 15\n\n# Verify health\ncurl http://localhost:8080/health\n\n# Open dashboard\nopen http://localhost:8080\n```\n\n### Run with Kubernetes (kind)\n\n```bash\n# Deploy to kind cluster with Bitnami PostgreSQL Helm chart\nmake run-k8s\n\n# Automatically:\n# - Creates kind cluster with port mappings\n# - Deploys PostgreSQL via Bitnami Helm chart\n# - Deploys server (2 replicas) and worker (3 replicas)\n# - Configures NodePort service for external access\n\n# Access the application\nopen http://localhost:8080\n\n# View deployment status\nkubectl get pods -n task-queue\n\n# Check logs\nkubectl logs -f deployment/task-queue-server -n task-queue\nkubectl logs -f deployment/task-queue-worker -n task-queue\n```\n\nFor detailed Kubernetes documentation, see [`k8s/README.md`](k8s/README.md).\n\n---\n\n## 🧪 Testing\n\n### Run All Integration Tests\n\n```bash\n# Test Docker Compose deployment\nmake test-integration-docker\n\n# Test Kubernetes deployment  \nmake test-integration-k8s\n```\n\n### Test Results\n\nAll 11 integration tests pass successfully:\n\n| Test | Description |\n|------|-------------|\n| Basic lifecycle | Task creation → execution → completion |\n| Priority ordering | High-priority tasks processed first |\n| Task failures | Retry logic and backoff |\n| Failure types | Errors vs timeouts |\n| Timeout retries | Slow tasks retry correctly |\n| Retry history | Complete audit trail |\n| Concurrent processing | 20 tasks with 5 workers |\n| Statistics API | Metrics accuracy |\n| Invalid task type | Error handling |\n| Missing task | 404 responses |\n\n**Test Coverage:**\n- ✅ Task lifecycle (create, queue, run, complete)\n- ✅ Priority scheduling\n- ✅ Retry logic with exponential backoff\n- ✅ Timeout handling\n- ✅ Concurrent processing (no race conditions)\n- ✅ Error handling\n- ✅ Real-time statistics\n\n### Manual Testing\n\n```bash\n# Create a task\ncurl -X POST http://localhost:8080/api/tasks \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"name\": \"Test Email\",\n    \"type\": \"send_email\",\n    \"priority\": 5,\n    \"payload\": {\n      \"to\": \"test@example.com\",\n      \"subject\": \"Test\",\n      \"body\": \"This is a test\"\n    }\n  }'\n\n# Get task status\ncurl http://localhost:8080/api/tasks/{task_id}\n\n# Get task history\ncurl http://localhost:8080/api/tasks/{task_id}/history\n\n# View statistics\ncurl http://localhost:8080/api/stats\n```\n\n---\n\n## 📡 API Reference\n\n### Create Task\n\n**POST** `/api/tasks`\n\n```json\n{\n  \"name\": \"Send Welcome Email\",\n  \"type\": \"send_email\",\n  \"priority\": 5,\n  \"payload\": {\n    \"to\": \"user@example.com\",\n    \"subject\": \"Welcome!\",\n    \"body\": \"Thanks for signing up.\"\n  }\n}\n```\n\n**Response:**\n```json\n{\n  \"id\": \"uuid\",\n  \"name\": \"Send Welcome Email\",\n  \"type\": \"send_email\",\n  \"status\": \"queued\",\n  \"priority\": 5,\n  \"retry_count\": 0,\n  \"max_retries\": 3,\n  \"created_at\": \"2025-12-06T10:00:00Z\"\n}\n```\n\n### Get Task\n\n**GET** `/api/tasks/:id`\n\n**Response:**\n```json\n{\n  \"id\": \"uuid\",\n  \"status\": \"succeeded\",\n  \"retry_count\": 1,\n  \"started_at\": \"2025-12-06T10:00:05Z\",\n  \"completed_at\": \"2025-12-06T10:00:15Z\"\n}\n```\n\n### Get Task History\n\n**GET** `/api/tasks/:id/history`\n\n**Response:**\n```json\n[\n  {\n    \"event_type\": \"task_queued\",\n    \"status\": \"queued\",\n    \"created_at\": \"2025-12-06T10:00:00Z\"\n  },\n  {\n    \"event_type\": \"task_started\",\n    \"status\": \"running\",\n    \"worker_id\": \"worker-123\",\n    \"created_at\": \"2025-12-06T10:00:05Z\"\n  },\n  {\n    \"event_type\": \"task_succeeded\",\n    \"status\": \"succeeded\",\n    \"created_at\": \"2025-12-06T10:00:15Z\"\n  }\n]\n```\n\n### Get Statistics\n\n**GET** `/api/stats`\n\n**Response:**\n```json\n{\n  \"total_tasks\": 1000,\n  \"queued_tasks\": 10,\n  \"running_tasks\": 5,\n  \"succeeded_tasks\": 950,\n  \"failed_tasks\": 35,\n  \"avg_retry_count\": 0.45,\n  \"tasks_with_retries\": 300\n}\n```\n\n### Health Check\n\n**GET** `/health`\n\n**Response:**\n```json\n{\n  \"status\": \"healthy\",\n  \"database\": \"connected\"\n}\n```\n\n---\n\n## ⚙️ Configuration\n\n### Environment Variables\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `DB_HOST` | `localhost` | PostgreSQL host |\n| `DB_PORT` | `5432` | PostgreSQL port |\n| `DB_USERNAME` | `admin` | Database user |\n| `DB_PASSWORD` | `admin` | Database password |\n| `DB_DATABASE` | `tasks` | Database name |\n| `SERVER_PORT` | `8080` | API server port |\n| `WORKER_CONCURRENCY` | `5` | Worker pool size |\n| `WORKER_POLL_INTERVAL` | `1` | Poll interval (seconds) |\n| `WORKER_TIMEOUT` | `30` | Task timeout (seconds) |\n\n### Docker Compose\n\nEdit `docker-compose.yml` to adjust configuration:\n\n```yaml\nservices:\n  worker:\n    environment:\n      WORKER_CONCURRENCY: \"10\"\n      WORKER_POLL_INTERVAL: \"2\"\n```\n\n### Kubernetes\n\nEdit `k8s/manifests/worker-deployment.yaml`:\n\n```yaml\nspec:\n  replicas: 5  # Number of worker pods\n  template:\n    spec:\n      containers:\n      - name: worker\n        env:\n        - name: WORKER_CONCURRENCY\n          value: \"10\"\n```\n\nConfiguration files are organized in `k8s/`:\n- `manifests/` - Kubernetes YAML files (deployments, services, configs)\n- `scripts/` - Deployment automation scripts\n\n---\n\n## 🔧 Development\n\n### Project Structure\n\n```\n.\n├── cmd/\n│   ├── server/          # API server entry point\n│   └── worker/          # Worker entry point\n│\n├── internal/\n│   ├── api/             # HTTP handlers and routes\n│   ├── config/          # Configuration\n│   ├── models/          # Domain models (Task, History)\n│   ├── storage/         # Data access layer\n│   │   └── postgres/    # PostgreSQL implementation\n│   └── worker/          # Worker pool and task handlers\n│       ├── worker.go    # Dispatcher + worker pool\n│       ├── registry.go  # Handler registration\n│       └── handlers/    # Task type implementations\n│\n├── db/\n│   └── migrations/      # SQL schema migrations\n│\n├── k8s/                 # Kubernetes deployment\n│   ├── manifests/       # YAML files (deployments, services, configs)\n│   ├── scripts/         # Deployment automation scripts\n│   └── README.md        # Kubernetes documentation\n│\n├── tests/               # Integration tests\n├── web/                 # Dashboard UI\n│   ├── static/          # CSS, JavaScript\n│   └── templates/       # HTML templates\n│\n├── docker-compose.yml\n└── Makefile\n```\n\nPlease review these documents before contributing:\n- [CONTRIBUTING.md](CONTRIBUTING.md) — guidelines, development setup, and workflow\n- [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) — community standards and enforcement\n- [SECURITY.md](SECURITY.md) — how to report vulnerabilities\n\n### Makefile Commands\nmake fmt                        # Format Go code\nmake lint                       # Run linters (golangci-lint)\n\n```bash\n# Quick Start\nmake help                       # Show all commands\nmake docker-up                  # Start Docker Compose\nmake run-k8s                    # Deploy to Kubernetes (one command!)\nmake test-integration           # Run integration tests\n\n# Build\nmake build                      # Build server and worker binaries\nmake docker-build               # Build Docker images\n\n# Testing\nmake test-integration-docker    # Test Docker Compose\nmake test-integration-k8s       # Test Kubernetes\nmake check-tools                # Verify prerequisites\n\n# Kubernetes\nmake k8s-down                   # Stop Kubernetes\n\n# Database\nmake migrate-up                 # Run migrations\nmake migrate-down               # Rollback last migration\n\n# Cleanup\nmake clean                      # Remove build artifacts\n```\n\n---\n\n## 📊 Monitoring\n\n### Dashboard\n\nAccess the real-time dashboard at: **http://localhost:8080/**\n\nFeatures:\n- Live task statistics (updated via Server-Sent Events)\n- Success rate visualization\n- Retry metrics\n- Auto-refresh every 5 seconds\n\n### Logs\n\n**Docker Compose:**\n```bash\ndocker-compose logs -f server\ndocker-compose logs -f worker\n```\n\n**Kubernetes:**\n```bash\nkubectl logs -f deployment/task-queue-worker -n task-queue\nkubectl logs -f deployment/task-queue-server -n task-queue\n\n# View all pods\nkubectl get pods -n task-queue\n\n# Describe a specific pod\nkubectl describe pod \u003cpod-name\u003e -n task-queue\n```\n\n---\n\n## 🏗️ Why This Design?\n\n### Key Architectural Decisions\n\n1. **PostgreSQL**\n   - Simplicity: One database for everything\n   - Durability: No separate persistence layer needed\n   - Rich queries: Task history, statistics, complex filtering\n\n2. **Dispatcher Pattern**\n   - Prevents database connection storm\n   - Reduces DB load by 98% (1 query vs 50 queries/second)\n   - Buffered channel provides natural backpressure\n\n3. **Exponential Backoff + Jitter**\n   - Prevents retry storms\n   - Gradually increases delay for persistent failures\n   - Jitter spreads load over time\n\n4. **Lock Expiration**\n   - Auto-recovery from worker crashes\n   - No manual intervention needed\n   - Tasks never stuck permanently\n\n5. **Context Cancellation**\n   - Graceful shutdown\n   - In-flight tasks complete properly\n   - Safe for Kubernetes rolling updates\n\n6. **Kubernetes with kind \u0026 Bitnami PostgreSQL**\n   - Production-ready: Bitnami Helm chart with security updates\n   - Easy local testing: kind runs K8s in Docker containers\n   - Horizontal scaling: Server (2 replicas) + Worker (3 replicas)\n   - Self-healing: Automatic pod restart on failures\n   - Organized structure: Separate manifests/ and scripts/ directories\n\n---\n\n\n## 🤝 Contributing\n\nContributions are welcome! Here's how you can help:\n\n1. **Fork** the repository\n2. **Create** a feature branch (`git checkout -b feature/amazing-feature`)\n3. **Commit** your changes (`git commit -m 'Add amazing feature'`)\n4. **Push** to the branch (`git push origin feature/amazing-feature`)\n5. **Open** a Pull Request\n\n### Development Setup\n\n```bash\nmake setup             # Install dependencies\nmake test-integration  # Run tests\n```\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## 👤 Author\n\n**Amit Basuri**\n- GitHub: [@amitbasuri](https://github.com/amitbasuri)\n\n## 🌟 Show Your Support\n\nGive a ⭐️ if this project helped you!\n\n---\n\n**Built with ❤️ using Go, PostgreSQL, and Kubernetes**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famitbasuri%2Ftaskqueue-runner-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famitbasuri%2Ftaskqueue-runner-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famitbasuri%2Ftaskqueue-runner-go/lists"}