{"id":29860292,"url":"https://github.com/imdeepmind/gotaskq","last_synced_at":"2026-02-08T18:04:46.192Z","repository":{"id":301734125,"uuid":"1010119010","full_name":"imdeepmind/GoTaskQ","owner":"imdeepmind","description":"GoTaskQ is a distributed task queue system built in Go, designed for high performance, reliability, and full visibility into asynchronous workloads. ","archived":false,"fork":false,"pushed_at":"2025-06-28T13:14:38.000Z","size":3,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-28T14:24:59.804Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/imdeepmind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-28T11:40:04.000Z","updated_at":"2025-06-28T13:15:04.000Z","dependencies_parsed_at":"2025-06-28T14:25:01.993Z","dependency_job_id":"da09621e-b87e-4082-98c2-a1827b05a4ff","html_url":"https://github.com/imdeepmind/GoTaskQ","commit_stats":null,"previous_names":["imdeepmind/gotaskq"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/imdeepmind/GoTaskQ","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imdeepmind%2FGoTaskQ","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imdeepmind%2FGoTaskQ/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imdeepmind%2FGoTaskQ/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imdeepmind%2FGoTaskQ/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/imdeepmind","download_url":"https://codeload.github.com/imdeepmind/GoTaskQ/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imdeepmind%2FGoTaskQ/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267803409,"owners_count":24146518,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-30T03:18:29.886Z","updated_at":"2026-02-08T18:04:46.183Z","avatar_url":"https://github.com/imdeepmind.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# GoTaskQ\n\n**GoTaskQ** is a distributed task queue system built in Go, designed for high performance, reliability, and full visibility into asynchronous workloads. It aims to provide a clean, extensible architecture that supports multiple queue backends, a simple function registration mechanism, and powerful monitoring tools.\n\nThis project is currently in development.\n\n## What is GoTaskQ?\n\nGoTaskQ is a general-purpose task execution framework that allows you to offload, schedule, and monitor background jobs using a queue-based system. It supports submitting tasks via REST, gRPC, and a built-in dashboard, and is designed to integrate easily with modern backend services.\n\nAt its core, GoTaskQ connects a message queue to a user-defined task registry and provides full lifecycle management of task execution — including retries, timeouts, cancellation, and detailed status tracking.\n\n## Goals\n\n- **Queue Backend Agnostic**: Support RabbitMQ, AWS SQS, and Redis Streams with a clean abstraction layer.\n- **Multi-Protocol Submission**: Tasks can be submitted via REST, gRPC, or directly through the dashboard.\n- **Function Registry**: Define and register task functions by name, and let the system dispatch messages to the correct handler.\n- **Visibility**: Real-time dashboard for monitoring queues, workers, logs, errors, execution metadata, and more.\n- **Reliability**: Built-in retry mechanism, dead-letter queue (DLQ) support, and task status persistence.\n- **Security**: API keys for REST/gRPC, and authentication for dashboard access.\n- **Control**: User-defined max concurrency, task timeouts, and cancellation support.\n\n## Planned Features\n\n### Task Submission\n\n- Submit tasks via REST, gRPC, or the dashboard\n- Support for batch submission\n- Each task includes:\n  - A `function_key` that maps to a registered Go function\n  - Arbitrary JSON payload\n  - Optional task-level settings like timeout, retries, and priority\n\n### Task Execution\n\n- Register custom functions in Go at startup\n- Dispatcher routes tasks based on the `function_key`\n- Task-level timeout and cancellation support\n- Configurable worker pool size\n- Retries with backoff strategy\n- Dead Letter Queue for permanently failed tasks\n\n### Queue Backends\n\n- RabbitMQ\n- AWS SQS\n- Redis Streams\n\n### Status Tracking \u0026 Metadata\n\n- Persistent task status:\n  - `PENDING`, `STARTED`, `SUCCESS`, `FAILURE`, `CANCELED`\n- Stores:\n  - Execution time\n  - Retry count\n  - Timestamps (created, started, completed)\n  - Result or error messages\n- Pluggable storage backends (Redis, SQL)\n\n### Dashboard\n\n- Web-based UI for:\n  - Submitting new tasks\n  - Viewing live task status\n  - Filtering and searching tasks\n  - Viewing execution logs and errors\n  - Requeuing or canceling tasks\n- Authenticated access for operators\n\n### APIs and Access Control\n\n- REST and gRPC APIs for task submission and status querying\n- API key authentication\n- Logging for all task lifecycle events\n\n## Architecture\n\n```mermaid\nflowchart LR\n    subgraph Clients\n        UI[Web UI]\n        API[REST / gRPC Clients]\n    end\n\n    subgraph Producer\n        P[Task Producer]\n    end\n\n    subgraph Broker\n        Q[(RabbitMQ / SQS Queue)]\n    end\n\n    subgraph Workers\n        W1[Worker 1]\n        W2[Worker 2]\n        WN[Worker N]\n    end\n\n    subgraph Analytics\n        DB[(Tracking / Analytics DB)]\n    end\n\n    %% Client to Producer\n    UI --\u003e P\n    API --\u003e P\n\n    %% Producer actions\n    P --\u003e|Publish Task| Q\n    P -.-\u003e|Best-effort Store Metadata| DB\n\n    %% Queue to Workers\n    Q --\u003e|Exclusive Delivery| W1\n    Q --\u003e|Exclusive Delivery| W2\n    Q --\u003e|Exclusive Delivery| WN\n\n    %% Worker lifecycle\n    W1 --\u003e|Execute Task| W1\n    W2 --\u003e|Execute Task| W2\n    WN --\u003e|Execute Task| WN\n\n    %% Worker tracking\n    W1 -.-\u003e|Best-effort Store Result| DB\n    W2 -.-\u003e|Best-effort Store Result| DB\n    WN -.-\u003e|Best-effort Store Result| DB\n\n    %% Acknowledgment\n    W1 --\u003e|ACK| Q\n    W2 --\u003e|ACK| Q\n    WN --\u003e|ACK| Q\n```\n\nThis system is a modular, self-hosted distributed task queue platform built in Go, designed for high-performance internal use.\n\n### Components\n\n#### Dashboard / API Layer\n\n  Provides a Web UI and REST/gRPC APIs for submitting tasks and viewing task-related information such as submission history, execution status, and logs.\n\n  This layer acts purely as an entry point and observability interface. It does **not** participate in task execution and does not make any execution decisions.\n\n#### Producer\n\n  The Producer receives task submission requests from the API layer and is responsible for:\n\n  * Validating incoming task payloads and metadata\n  * Assigning a unique task identifier\n  * Publishing the task message to the configured message broker (RabbitMQ or SQS)\n\n  The Producer may **best-effort persist task metadata** to the database for tracking and analytics.\n  The database is **not** a source of truth, and failures to persist metadata do not affect task execution.\n\n#### Message Queue (Broker)**\n\n  A pluggable message broker such as RabbitMQ or AWS SQS that serves as the **authoritative source of truth** for task delivery.\n\n  Responsibilities include:\n\n  * Durable storage of task messages\n  * Exclusive delivery of each task to exactly one worker at a time\n  * Preventing concurrent execution of the same task\n  * Redelivery of tasks if a worker crashes or fails to acknowledge\n\n  Tasks are removed from the queue **only after successful acknowledgment** by a worker.\n  The system provides **at-least-once delivery semantics**.\n\n#### Worker(s)\n\n  Workers are long-running processes that continuously poll the message queue for tasks.\n\n  Each worker:\n\n  * Receives tasks exclusively from the broker\n  * Executes the corresponding registered task function\n  * Best-effort persists execution metadata (status, logs, timing, errors) to the database\n  * Acknowledges the task after successful execution\n\n  Workers are stateless, do not coordinate with each other, and **never read from the database** to determine execution behavior.\n\n#### Database (Tracking / Analytics)\n\n  A persistence layer used solely for **observability and analytics**, including:\n\n  * Task submission metadata\n  * Execution status and timestamps\n  * Logs and error details\n\n  The database is **non-authoritative** and **eventually consistent**.\n  Missing or inconsistent records do not impact correctness or execution of tasks.\n\n## Status\n\nGoTaskQ is under active development and not yet ready for production use. This repository is currently a staging ground for architectural decisions and implementation work.\n\nIf you’re interested in this project or want to collaborate, feel free to open an issue or reach out.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimdeepmind%2Fgotaskq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimdeepmind%2Fgotaskq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimdeepmind%2Fgotaskq/lists"}