{"id":50717527,"url":"https://github.com/prathamreet/ct-project-176-191","last_synced_at":"2026-06-09T20:01:12.723Z","repository":{"id":355454130,"uuid":"1228148115","full_name":"prathamreet/ct-project-176-191","owner":"prathamreet","description":"A Production-Grade Microservices Task Queue with Kubernetes Native Scaling, Observability \u0026 CI/CD","archived":false,"fork":false,"pushed_at":"2026-05-24T17:58:16.000Z","size":917,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-24T19:24:39.172Z","etag":null,"topics":["cd","ci","docker","grafana","html","nginx","nodejs","prometheus","redis","sqlite"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prathamreet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-03T16:53:58.000Z","updated_at":"2026-05-24T17:58:20.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/prathamreet/ct-project-176-191","commit_stats":null,"previous_names":["prathamreet/ct-project"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/prathamreet/ct-project-176-191","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prathamreet%2Fct-project-176-191","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prathamreet%2Fct-project-176-191/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prathamreet%2Fct-project-176-191/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prathamreet%2Fct-project-176-191/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prathamreet","download_url":"https://codeload.github.com/prathamreet/ct-project-176-191/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prathamreet%2Fct-project-176-191/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34123172,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-09T02:00:06.510Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cd","ci","docker","grafana","html","nginx","nodejs","prometheus","redis","sqlite"],"created_at":"2026-06-09T20:01:11.266Z","updated_at":"2026-06-09T20:01:12.715Z","avatar_url":"https://github.com/prathamreet.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"https://capsule-render.vercel.app/api?type=soft\u0026color=gradient\u0026height=10\u0026section=header\" width=\"1080\" align=\"center\"/\u003e\n\n\u003e Om Ji Rao (1NH23CS176)  ·  Pratham Reet (1NH23CS191)\n\n\u003cimg src=\"https://capsule-render.vercel.app/api?type=soft\u0026color=gradient\u0026height=10\u0026section=header\" width=\"1080\" align=\"center\"/\u003e\n\n# TaskFlow - Scalable Asynchronous Job Engine\n\nTaskFlow is an industry-standard, high-performance asynchronous task processing system built using a decoupled microservices architecture. It demonstrates modern DevOps engineering principles, including horizontal scaling, self-healing infrastructures, dynamic metric scrapers, and automated integration pipelines.\n\n### Core Objectives\n1. **Multi-Service Architecture:** Design and implement an asynchronous task queue with separation of concerns across Nginx frontend, Express backend API, Redis message broker, and background workers.\n2. **Multi-Stage Containerization:** Containerize all services using optimized multi-stage Docker builds to achieve an 80% package footprint reduction for production environments.\n3. **Orchestrated Deployment:** Deploy the complete system on Kubernetes using Horizontal Pod Autoscalers (HPA) for dynamic scaling and Persistent Volume Claims (PVC) for broker state safety.\n4. **Cloud-Native Observability:** Establish a production-grade observability pipeline using Prometheus Service Discovery (via Kubernetes RBAC API permissions) and custom Grafana dashboards.\n5. **CI/CD Automation:** Automate the build, test, and validation lifecycle using fully integrated GitHub Actions workflows.\n\n```\n                  ┌────────────────────────┐\n                  │      Frontend UI       │\n                  │   (Nginx Web Server)   │\n                  └───────────┬────────────┘\n                              │ (Reverse Proxy /api/*)\n                              ▼\n                  ┌────────────────────────┐\n                  │    Backend API Core    │  ◄──── [Prometheus Pull]\n                  │   (Node.js Express)    │\n                  └───────────┬────────────┘\n                              │\n                    (LPUSH)   ▼   (RPOP)\n                  ┌────────────────────────┐\n                  │  Redis Message Broker  │\n                  │     (Queue \u0026 AOF)      │\n                  └───────────┬────────────┘\n                              │\n                              ▼\n                  ┌────────────────────────┐\n                  │   Background Worker    │  ◄──── [Prometheus Pull]\n                  │    (Node.js Engine)    │\n                  └────────────────────────┘\n```\n\n### Decoupled Core Services\n*   **Frontend UI (Nginx):** A high-performance Alpine-based reverse proxy that serves the dark-themed dashboard and forwards API routes cleanly, separating external client paths from internal networks.\n*   **Backend API (Express):** Exposes RESTful endpoints for task creation/retrieval and generates dynamic, standard Prometheus metrics from process resource usage and active state.\n*   **Worker Engine (Node.js):** Consumes queue jobs using an event-loop-safe non-blocking polling mechanism and updates state in Redis.\n*   **Message Broker (Redis):** Manages task lists (`queue` and `all_tasks`) backed by a Persistent Volume Claim using **Append-Only File (AOF)** persistence.\n\n\n## Key Engineering Highlights\n\n### Docker Multi-Stage Optimization\nBoth Backend and Worker images implement multi-stage production builds separating builder compilers from lightweight lean runtimes, resulting in an **80% package footprint reduction**:\n\n| Service Container  | Single-Stage Size | Multi-Stage Size | Reduction % | Pull/Startup Time |\n| :----------------- | :---------------: | :--------------: | :---------: | :---------------: |\n| **Backend Core**   |      913 MB       |    **184 MB**    |  **79.8%**  |  ~45s → **12s**   |\n| **Worker Engine**  |      913 MB       |    **184 MB**    |  **79.8%**  |  ~45s → **12s**   |\n| **Frontend Proxy** |      24.7 MB      |   **24.7 MB**    | *Baseline*  |       \u003c 3s        |\n\n### Kubernetes Native Elastic Scaling (HPA)\nThe cluster deploys automated **Horizontal Pod Autoscalers (HPA)** configured with responsive, low limits for demonstrable scaling behavior during workloads:\n*   **Worker Auto-Scalability:** Dynamically scales from **1 to 10 replicas** based on target CPU utilization limits.\n*   **Backend Auto-Scalability:** Scales from **1 to 5 replicas** dynamically to manage high API traffic.\n*   **Performance Metrics:** Handles **580 tasks/sec** at peak capacity with 10 replicas (an **8.9x** throughput increase over single-pod baselines).\n\n### Self-Healing \u0026 State Persistence\n*   **Probes:** Equipped with custom `livenessProbes` to trigger container restarts on thread blockages and `readinessProbes` to route user traffic only when the Redis network link is active.\n*   **Data Integrity:** Redis leverages active AOF logging mapped to a `PersistentVolumeClaim` (PVC), preserving queue states across sudden pod restarts and replacements.\n\n### RBAC Observability Pipeline\n*   **Prometheus Service Discovery:** Configured with a Kubernetes `ClusterRole` ServiceAccount to query pod endpoints dynamically via K8s APIs, scraping metrics automatically as HPAs create or terminate replicas.\n*   **Grafana Dashboards:** Pre-provisioned dashboards plotting panel statistics for *Active Pods*, *CPU Load*, *Task Throughput*, *Processing Speed*, and *Queue Pressure*.\n\n\n## Technology Stack\n\n*   **Runtime:** Node.js (v18 LTS), Express, Redis client\n*   **Broker:** Redis (Alpine, AOF persistence)\n*   **Proxy \u0026 Web Server:** Nginx (Alpine)\n*   **Orchestration:** Docker Compose (Local Dev) / Kubernetes (Production minikube)\n*   **Metrics \u0026 Visualization:** Prometheus OSS \u0026 Grafana Dashboards\n*   **CI/CD:** GitHub Actions (HTMLHint, Compose Linter, Automated Multi-stage builds)\n\n\n## Project Directory Structure\n\n```\ntaskflow/\n├── docker-compose.yml       # Local multi-container development environment\n├── README.md                # System Overview \u0026 report summaries\n├── .gitignore               # Configured to exclude agent and graph files\n├── services/\n│   ├── backend/             # Express API \u0026 Metrics Server + Multi-stage Dockerfile\n│   ├── frontend/            # Dark-theme HTML/CSS UI \u0026 Nginx Reverse Proxy Config\n│   └── worker/              # Queue processing consumer \u0026 Metrics + Multi-stage Dockerfile\n├── infra/\n│   ├── k8s/                 # Kubernetes YAML configs (Deployments, Services, HPAs, PVCs)\n│   └── monitoring/          # Prometheus configuration, RBAC, Grafana dashboards\n├── scripts/\n│   ├── run-k8s.ps1 / .sh    # Automation deployment scripts (starts, ports, and registers)\n│   ├── stop-k8s.ps1 / .sh   # Automation teardown scripts\n│   └── stress.js            # Interactive high-concurrency stress testing script\n└── .github/workflows/       # GitHub Actions CI pipelines\n```\n\n\n## Quick Start Guide\n\n### 1. Local Container Development (Docker Compose)\nBuild and spin up the complete local environment in detached mode:\n```bash\n# Start all services\ndocker-compose up --build -d\n\n# Verify local running containers\ndocker-compose ps\n\n# Stop all services\ndocker-compose down\n```\n\n### 2. Kubernetes Orchestration Deployment (minikube / kubectl)\nRun the automated powershell/bash pipeline scripts to spin up, build, deploy K8s manifests, install Metrics Servers, and configure local port forwards in one command:\n```bash\n# Run automated setup\n./scripts/run-k8s.sh\n\n# Run interactive load generator to trigger scaling (Input 500 tasks)\nnode scripts/stress.js\n\n# Teardown local cluster resources\n./scripts/stop-k8s.sh\n```\n\n---\n\n## Port \u0026 Endpoint Reference\n\n| Interface Dashboard        | Service Engine    | Access URL                | Port Type               |\n| :------------------------- | :---------------- | :------------------------ | :---------------------- |\n| **Frontend Web Dashboard** | Nginx Proxy       | **http://localhost:8080** | LoadBalancer / External |\n| **Backend REST API**       | Express Core      | **http://localhost:5000** | ClusterIP / Internal    |\n| **Prometheus Targets**     | Metrics Scraper   | **http://localhost:9090** | ClusterIP / Internal    |\n| **Grafana Visualizations** | System Dashboards | **http://localhost:3000** | LoadBalancer / External |\n\n---\n\n## CLI Cheat Sheet\n\n```bash\n# Verify active pods \u0026 services\nkubectl get pods -o wide\nkubectl get services\n\n# Watch Autoscaling (HPA) live\nkubectl get hpa -w\n\n# Check container health and persistent storage mounts\nkubectl describe pod backend-\u003cname\u003e\nkubectl get pvc\n\n# Aggregate application logs\nkubectl logs -l app=backend --tail=100\n```\n\u003cimg src=\"https://capsule-render.vercel.app/api?type=soft\u0026color=gradient\u0026height=10\u0026section=header\" width=\"1080\" align=\"center\"/\u003e\n\n\n*   **Interactive Code Dependency Graph:** [TaskFlow System Dependency Graph](https://exquisite-marzipan-5c18ba.netlify.app/graph.html)\n*   **Interactive Codebase Call-Flow Map:** [TaskFlow Dynamic Call-Flow Diagrams](https://exquisite-marzipan-5c18ba.netlify.app/ctmp-callflow.html)\n\n\u003cimg src=\"https://capsule-render.vercel.app/api?type=soft\u0026color=gradient\u0026height=10\u0026section=header\" width=\"1080\" align=\"center\"/\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprathamreet%2Fct-project-176-191","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprathamreet%2Fct-project-176-191","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprathamreet%2Fct-project-176-191/lists"}