{"id":34599188,"url":"https://github.com/flyingrobots/go-redis-work-queue","last_synced_at":"2025-12-24T12:07:57.537Z","repository":{"id":314495947,"uuid":"1055751053","full_name":"flyingrobots/go-redis-work-queue","owner":"flyingrobots","description":"Go-based job queue system backed by Redis. Provides producer, worker, and all-in-one modes with robust resilience, observability, and configurable behavior.","archived":false,"fork":false,"pushed_at":"2025-09-20T13:08:18.000Z","size":149246,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-20T14:49:04.320Z","etag":null,"topics":["go","golang","jobqueue","jobs","redis","workers"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flyingrobots.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-12T18:40:07.000Z","updated_at":"2025-09-20T12:55:59.000Z","dependencies_parsed_at":"2025-09-12T21:07:14.481Z","dependency_job_id":"fddba902-82c8-410a-af03-e02d02b14e50","html_url":"https://github.com/flyingrobots/go-redis-work-queue","commit_stats":null,"previous_names":["flyingrobots/go-redis-work-queue"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/flyingrobots/go-redis-work-queue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyingrobots%2Fgo-redis-work-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyingrobots%2Fgo-redis-work-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyingrobots%2Fgo-redis-work-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyingrobots%2Fgo-redis-work-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flyingrobots","download_url":"https://codeload.github.com/flyingrobots/go-redis-work-queue/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flyingrobots%2Fgo-redis-work-queue/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28002250,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-24T02:00:07.193Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","jobqueue","jobs","redis","workers"],"created_at":"2025-12-24T12:07:40.667Z","updated_at":"2025-12-24T12:07:57.532Z","avatar_url":"https://github.com/flyingrobots.png","language":"Go","readme":"# Go Redis Work Queue\n\n\u003e Redis job queue system in Go. \n\nProvides producer, worker, and all-in-one modes with robust resilience, observability, and configurable behavior via YAML.\n\n- Single binary with multi-role execution\n- Priority queues with reliable processing and retries\n- Graceful shutdown, reaper for stuck jobs, circuit breaker\n- Prometheus metrics, structured logging, optional tracing\n\nSee the [Feature Matrix](docs/features-ledger.md) for the latest capability status (stable, experimental, deprecated).\n\nSee `docs/` to learn more. A sample configuration is provided in `config/config.example.yaml`.\n\nDeveloper tools and automation: see `docs/tools/README.md` for:\n- Progress automation for the Features Ledger and README\n- Local pre-commit hook and CI auto-update\n- Script to extract CodeRabbit PR comments and “Prompt for AI Agents” sections\n\n----\n\n## Progress\n\nFor full details, see the Features Ledger at [docs/features-ledger.md](docs/features-ledger.md).\n\n\u003c!-- progress:begin --\u003e\n```text\n██████████████████████▓░░░░░░░░░░░░░░░░░ 56%\n---------|---------|---------|---------|\n        MVP      Alpha     Beta  v1.0.0 \n```\n\u003c!-- progress:end --\u003e\n\n----\n\n## Quick start\n\n1. Clone the repo\n2. Ensure Redis is available (e.g., Docker container `redis:latest` on port `6379`)\n3. Follow the instructions to run in producer, worker, or all-in-one modes\n\n### Build and run\n\n1. Copy example config\n\n```bash\ncp config/config.example.yaml config/config.yaml\n```\n\n2. Build (Go 1.25+)\n\n```bash\nmake build\n```\n\n3. Run in one of the following modes:\n\nRun all-in-one\n\n```bash\n./bin/job-queue-system --role=all --config=config/config.yaml\n```\n\nRun producer only\n\n```bash\n./bin/job-queue-system --role=producer --config=config/config.yaml\n```\n\nRun worker only\n\n```bash\n./bin/job-queue-system --role=worker --config=config/config.yaml\n```\n\n### TUI (Bubble Tea)\n\nAn interactive TUI is available for observing and administering the job queue. It uses `Charmbracelet`’s Bubble Tea stack and renders queue stats, keys, peeks, a simple benchmark, and charts.\n\nBefore the first run, download dependencies:\n\n```bash\ngo mod download\n```\n\nRun it:\n\n```\ngo run ./cmd/tui --config config/config.yaml\n```\n\nOr build it:\n\n```\ngo build -o bin/tui ./cmd/tui\n./bin/tui --config config/config.yaml\n```\n\nFlags:\n\n- `--config`: Path to YAML config (defaults to `config/config.yaml`).\n- `--refresh`: Stats refresh interval (default `2s`).\n\nKeybindings:\n\n- `q`: quit\n- `esc`: toggle help overlay (when not in a modal/input)\n- `tab`: switch Queues/Keys\n- `r`: refresh\n- `j/k`: move selection\n- `p`: peek selected queue\n- `b`: open bench form (tab cycles fields, enter runs, esc exits)\n- `c`: charts view (time-series for queue lengths)\n- `f` or `/`: filter queues (fuzzy, case-insensitive); `esc` clears\n- `D`: purge dead-letter queue (modal confirm)\n- `A`: purge ALL managed keys (modal confirm)\n\nMouse:\n\n- Wheel scrolls, hover highlights row, left-click selects, right-click peeks.\n\nNotes:\n\n- The TUI calls internal admin APIs, so it reflects the same Redis keys as the CLI admin mode.\n- When a confirmation modal is open, the background dims and a full-screen scrim appears for focus.\n\nScreenshots (examples):\n\n![Queues View](docs/images/tui-queues.png)\n\n![Peek Modal](docs/images/tui-peek.png)\n\n![Charts View](docs/images/tui-charts.png)\n\n### Admin Commands\n\nThe CLI provides `--admin-cmd` flags that help you inspect the system.\n\n```bash\n# Stats\n./bin/job-queue-system --role=admin --admin-cmd=stats --config=config/config.yaml\n\n# Peek\n./bin/job-queue-system --role=admin --admin-cmd=peek --queue=low --n=10 --config=config/config.yaml\n\n# Purge DLQ\n./bin/job-queue-system --role=admin --admin-cmd=purge-dlq --yes --config=config/config.yaml\n\n# Purge all (test keys) — DEV ONLY; requires explicit --dev + --yes safeguards\n./bin/job-queue-system --role=admin --admin-cmd=purge-all --dev --yes --config=config/config.yaml\n\n# Stats (keys)\n./bin/job-queue-system --role=admin --admin-cmd=stats-keys --config=config/config.yaml\n\n# Version\n./bin/job-queue-system --version\n```\n\n### Metrics\n\nPrometheus metrics exposed at \u003chttp://localhost:9091/metrics\u003e by default (override via `observability.metrics_port` to avoid conflicts with local Prometheus).\n\n### Health and Readiness\n\n- Liveness: \u003chttp://localhost:9090/healthz\u003e returns 200 when the process is up\n- Readiness: \u003chttp://localhost:9090/readyz\u003e returns 200 only when Redis is reachable\n\n### Priority Fetching\n\nWorkers emulate prioritized multi-queue blocking fetch by looping priorities (e.g., high then low) and issuing `BRPOPLPUSH` per-queue with a short timeout (default 1s). This preserves atomic move semantics within each queue, prefers higher priority at sub-second granularity, and avoids job loss. Lower-priority jobs may incur up to the timeout in extra latency when higher-priority queues are empty.\n\n### Rate Limiting\n\nProducer rate limiting uses a fixed-window counter (`INCR` + 1s `EXPIRE`) and sleeps precisely until the end of the window (`TTL`), with small jitter to avoid thundering herd.\n\n### Docker\n\nTo make it easy to use, a `Dockerfile` has been provided. The following demonstrate how to use it:\n\n#### Build\n\n```bash\ndocker build -t job-queue-system:latest .\n```\n\n#### Run\n\n```bash\ndocker run --rm \\\n  -p 9091:9091 \\\n  -v $(pwd)/config/config.yaml:/app/config/config.yaml:ro \\\n  --env-file env.list \\\n  job-queue-system:latest --role=all --config=/app/config/config.yaml\n```\n\nEnsure `config/config.yaml` exists locally and `env.list` provides credentials (see `config/config.example.yaml` for keys).\n\n#### Compose\n\n```bash\ndocker compose -f deploy/docker-compose.yml up --build\n```\n\n----\n\n## `v0.4.0-alpha` Coming Soon\n\nRelease branch open for `v0.4.0-alpha`: see PR \u003chttps://github.com/flyingrobots/go-redis-work-queue/pull/1\u003e\n\nPromotion gates and confidence summary (details in `docs/15_promotion_checklists.md`):\n\n### Release Roadmap\n\n- **Alpha → Beta**: overall confidence `~0.85` (functional/observability/CI strong; perf and coverage improvements planned)\n- **Beta → RC**: overall confidence `~0.70` (needs controlled perf run, chaos tests, soak)\n- **RC → GA**: overall confidence `~0.70` (release flow ready; soak and rollback rehearsal pending)\n\n### Evidence artifacts (`docs/evidence/`):\n\n- `ci_run.json` (CI URL), \n- `bench.json` (throughput/latency), \n- `metrics_before.txt`/`metrics_after.txt`, \n- `config.alpha.yaml`\n\nTo reproduce evidence locally, see `docs/evidence/README.md`.\n\n----\n\n## Testing\n\nSee `docs/testing-guide.md` for a package-by-package overview and copy/paste commands to run individual tests or the full suite with the race detector.\n\n## Module Status Map\n\n| Module | Status | Summary |\n| --- | --- | --- |\n| [internal/automatic-capacity-planning](internal/automatic-capacity-planning/README.md) | BROKEN | Forecasting/simulator tests expect legacy behaviour; new algorithms need history seeding. |\n| [internal/canary-deployments](internal/canary-deployments/README.md) | BUILDS | Compiles; API endpoints still stubbed for worker/rollback flows. |\n| [internal/collaborative-session](internal/collaborative-session/README.md) | BUILDS | Compiles; control handoff still returns a not-implemented error. |\n| [internal/distributed-tracing-integration](internal/distributed-tracing-integration/README.md) | BUILDS | Compiles; tests still expect legacy OpenTelemetry helpers. |\n| [internal/dlq-remediation-pipeline](internal/dlq-remediation-pipeline/README.md) | BUILDS | Compiles; remediation actions still need implementation. |\n| [internal/event-hooks](internal/event-hooks/README.md) | BUILDS | Compiles; handlers remain TODO for replay/test flows. |\n| [internal/exactly_once](internal/exactly_once/README.md) | BROKEN | Outbox manager tests panic due to retry bookkeeping regressions. |\n| [internal/forecasting](internal/forecasting/README.md) | BROKEN | Holt-Winters and recommendation tests fail with new defaults. |\n| [internal/job-budgeting](internal/job-budgeting/README.md) | BUILDS | Compiles; budgeting enforcement still TODO. |\n| [internal/json-payload-studio](internal/json-payload-studio/README.md) | BUILDS | Compiles; handlers still rely on in-memory stubs. |\n| [internal/calendar-view](internal/calendar-view/README.md) | BUILDS | Compiles; calendar panel still awaits TUI wiring. |\n| [internal/kubernetes-operator](internal/kubernetes-operator/README.md) | BUILDS | Compiles; controllers/webhooks still stubbed out. |\n| [internal/long-term-archives](internal/long-term-archives/README.md) | BUILDS | Compiles; exporters/storage integrations still TODO. |\n| [internal/storage-backends](internal/storage-backends/README.md) | BUILDS | Compiles; remaining test harness needs go-redis v9 cleanup. |\n| [internal/tui](internal/tui/README.md) | BUILDS | Legacy view builds cleanly; enhanced view remains behind `tui_experimental`. |\n| [internal/trace-drilldown-log-tail](internal/trace-drilldown-log-tail/README.md) | BUILDS | Compiles; log streaming endpoints still placeholders. |\n| [internal/worker-fleet-controls](internal/worker-fleet-controls/README.md) | BUILDS | Compiles; action execution remains scaffolded for now. |\n| [internal/worker](internal/worker/README.md) | BUILDS | Runtime compiles; lacks dedicated unit tests. |\n\n----\n\n## Contributing\n\nWant to help? Here's how:\n\n1. Please report issues that you discover. \n2. If you solve any problems, PRs are welcome. \n\n### DX Tools\n\nPlease be sure to enable the following developer tools to enhance your development experience and align with the project's standard development practices.\n\n#### Enable Git hooks\n\n```bash\nmake hooks\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflyingrobots%2Fgo-redis-work-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflyingrobots%2Fgo-redis-work-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflyingrobots%2Fgo-redis-work-queue/lists"}