{"id":19324309,"url":"https://github.com/justrach/kew","last_synced_at":"2025-09-04T05:11:25.776Z","repository":{"id":260396869,"uuid":"881185482","full_name":"justrach/kew","owner":"justrach","description":"🚀 Kew - A Fast, Redis-backed Task Queue Manager for Python","archived":false,"fork":false,"pushed_at":"2025-09-01T20:00:02.000Z","size":614,"stargazers_count":46,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-01T20:15:01.278Z","etag":null,"topics":["async","asyncio","circuit-breaker","distrubuted-systems","priority-queue","python","python-library","queue-management","redis","task-queue"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/kew/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/justrach.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-31T03:54:40.000Z","updated_at":"2025-09-01T20:00:06.000Z","dependencies_parsed_at":"2025-04-22T19:42:39.290Z","dependency_job_id":"a9f976b7-441e-4b14-8acb-9edd54101d0a","html_url":"https://github.com/justrach/kew","commit_stats":null,"previous_names":["justrach/kew"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/justrach/kew","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/justrach%2Fkew","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/justrach%2Fkew/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/justrach%2Fkew/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/justrach%2Fkew/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/justrach","download_url":"https://codeload.github.com/justrach/kew/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/justrach%2Fkew/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273555459,"owners_count":25126316,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-04T02:00:08.968Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","asyncio","circuit-breaker","distrubuted-systems","priority-queue","python","python-library","queue-management","redis","task-queue"],"created_at":"2024-11-10T02:04:38.317Z","updated_at":"2025-09-04T05:11:25.767Z","avatar_url":"https://github.com/justrach.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/justrach/kew/blob/main/kew_logo.jpg\" alt=\"Kew Logo\" width=\"200\"/\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003eKew: Modern Async Task Queue\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pypi.org/project/kew\"\u003e\n    \u003cimg src=\"https://static.pepy.tech/badge/kew\" alt=\"PyPI Downloads\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/justrach/kew/actions/workflows/python-package.yml\"\u003e\n    \u003cimg src=\"https://github.com/justrach/kew/actions/workflows/python-package.yml/badge.svg\" alt=\"Github Actions\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\nA Redis-backed task queue built for modern async Python applications. Handles background processing with precise concurrency control, priority queues, and circuit breakers - all running in your existing async process.\n\n## Why Kew?\n\nBuilding async applications often means dealing with background tasks. Existing solutions like Celery require separate worker processes and complex configuration. Kew takes a different approach:\n\n- **Runs in Your Process**: No separate workers to manage - tasks run in your existing async process\n- **True Async**: Native async/await support - no sync/async bridges needed\n- **Precise Control**: Semaphore-based concurrency ensures exact worker limits\n- **Simple Setup**: Just Redis and a few lines of code to get started\n\n## How It Works\n\nKew manages task execution using a combination of Redis for persistence and asyncio for processing:\n```mermaid\ngraph LR\n    A[Application] --\u003e|Submit Task| B[Task Queue]\n    B --\u003e|Semaphore Control| C[Worker Pool]\n    C --\u003e|Execute Task| D[Task Processing]\n    D --\u003e|Success| E[Complete]\n    D --\u003e|Error| F[Circuit Breaker]\n    F --\u003e|Reset| B\n    style A fill:#f9f,stroke:#333\n    style B fill:#bbf,stroke:#333\n    style C fill:#bfb,stroke:#333\n    style D fill:#fbb,stroke:#333\n```\nTasks flow through several states with built-in error handling:\n```mermaid\nstateDiagram-v2\n    [*] --\u003e Submitted: Task Created\n    Submitted --\u003e Queued: Priority Assignment\n    Queued --\u003e Processing: Worker Available\n    Processing --\u003e Completed: Success\n    Processing --\u003e Failed: Error\n    Failed --\u003e CircuitOpen: Multiple Failures\n    CircuitOpen --\u003e Queued: Circuit Reset\n    Completed --\u003e [*]\n```\n## Quick Start\n\n1. Install Kew:\n```bash\npip install kew\n```\n\n2. Create a simple task processor:\n```python\nimport asyncio\nfrom kew import TaskQueueManager, QueueConfig, QueuePriority\n\nasync def process_order(order_id: str):\n    # Simulate order processing\n    await asyncio.sleep(1)\n    return f\"Order {order_id} processed\"\n\nasync def main():\n    # Initialize queue manager\n    manager = TaskQueueManager(redis_url=\"redis://localhost:6379\")\n    await manager.initialize()\n    \n    # Create processing queue\n    await manager.create_queue(QueueConfig(\n        name=\"orders\",\n        max_workers=4,  # Only 4 concurrent tasks\n        max_size=1000\n    ))\n    \n    # Submit some tasks\n    tasks = []\n    for i in range(10):\n        task = await manager.submit_task(\n            task_id=f\"order-{i}\",\n            queue_name=\"orders\",\n            task_type=\"process_order\",\n            task_func=process_order,\n            priority=QueuePriority.MEDIUM,\n            order_id=str(i)\n        )\n        tasks.append(task)\n    \n    # Check results\n    # Small delay to allow tasks to complete in this simple example\n    await asyncio.sleep(1.2)\n    for task in tasks:\n        status = await manager.get_task_status(task.task_id)\n        print(f\"{task.task_id}: {status.result}\")\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n## Real-World Examples\n\n### Async Web Application\n```python\nfrom fastapi import FastAPI\nfrom kew import TaskQueueManager, QueueConfig, QueuePriority\n\napp = FastAPI()\nmanager = TaskQueueManager()\n\n@app.on_event(\"startup\")\nasync def startup():\n    await manager.initialize()\n    await manager.create_queue(QueueConfig(\n        name=\"emails\",\n        max_workers=2\n    ))\n\n@app.post(\"/signup\")\nasync def signup(email: str):\n    # Handle signup immediately\n    user = await create_user(email)\n    \n    # Queue welcome email for background processing\n    await manager.submit_task(\n        task_id=f\"welcome-{user.id}\",\n        queue_name=\"emails\",\n        task_type=\"send_welcome_email\",\n        task_func=send_welcome_email,\n        priority=QueuePriority.MEDIUM,\n        user_id=user.id\n    )\n    return {\"status\": \"success\"}\n```\n\n### Data Processing Script\n```python\nasync def process_batch(items: list):\n    manager = TaskQueueManager()\n    await manager.initialize()\n    \n    # Create high and low priority queues\n    await manager.create_queue(QueueConfig(\n        name=\"critical\",\n        max_workers=4,\n        priority=QueuePriority.HIGH\n    ))\n    \n    await manager.create_queue(QueueConfig(\n        name=\"batch\",\n        max_workers=2,\n        priority=QueuePriority.LOW\n    ))\n    \n    # Process priority items first\n    for item in filter(is_priority, items):\n        await manager.submit_task(\n            task_id=f\"item-{item.id}\",\n            queue_name=\"critical\",\n            task_type=\"process_item\",\n            task_func=process_item,\n            priority=QueuePriority.HIGH,\n            item=item\n        )\n    \n    # Queue remaining items\n    for item in filter(lambda x: not is_priority(x), items):\n        await manager.submit_task(\n            task_id=f\"item-{item.id}\",\n            queue_name=\"batch\",\n            task_type=\"process_item\",\n            task_func=process_item,\n            priority=QueuePriority.LOW,\n            item=item\n        )\n```\n\n## Key Features\n\n### Concurrency Control\n```python\n# Strictly enforce 4 concurrent tasks max\nawait manager.create_queue(QueueConfig(\n    name=\"api_calls\",\n    max_workers=4  # Guaranteed not to exceed\n))\n```\n\n### Priority Queues\n```python\n# High priority queue for urgent tasks\nawait manager.create_queue(QueueConfig(\n    name=\"urgent\",\n    priority=QueuePriority.HIGH\n))\n\n# Lower priority for batch processing\nawait manager.create_queue(QueueConfig(\n    name=\"batch\",\n    priority=QueuePriority.LOW\n))\n```\n\n### Circuit Breakers\nBuilt-in per-queue circuit breaker tracks consecutive failures and temporarily opens the circuit to protect downstreams.\n\n- Defaults: `max_failures=3`, `reset_timeout=60s`\n- Note: Currently not configurable via `QueueConfig`.\n\n### Task Monitoring\n```python\n# Check task status\nstatus = await manager.get_task_status(\"task-123\")\nprint(f\"Status: {status.status}\")\nprint(f\"Result: {status.result}\")\nprint(f\"Error: {status.error}\")\n\n# Monitor queue health\nqueue_status = await manager.get_queue_status(\"api_calls\")\nprint(f\"Active Tasks: {queue_status['current_workers']}\")\nprint(f\"Circuit Breaker: {queue_status['circuit_breaker_status']}\")\n```\n\n## Performance \u0026 Reliability (v0.1.5)\n\n- Reduced worker-loop idle delay (from 100ms to 20ms) for faster task pickup and lower test flakiness.\n- Graceful shutdown: awaits active tasks, flushes callbacks, and uses Redis `aclose()` to avoid deprecation warnings and lost updates.\n- Requires Redis 7 locally and in CI.\n\n## Version Differences\n\nSee the full changelog in [CHANGELOG.md](CHANGELOG.md).\n\n- 0.1.5 (current)\n  - Faster task pickup (idle delay 100ms → 20ms)\n  - More reliable shutdown (await tasks, flush callbacks, Redis `aclose()`)\n  - Tests: 7/7 passing; coverage gate restored to 80% (total ~87%)\n  - Docs: examples aligned with async API (`task_type`, `priority`)\n  - CI: Redis 7 service; pip cache; simplified install\n- 0.1.4\n  - Stable async queues, priorities, and concurrency control\n  - Circuit breaker defaults (3 failures, 60s reset)\n  - Known issues: longer idle delay could leave tasks in PROCESSING briefly; Redis `close()` deprecation warnings\n\n## Roadmap\n\n- Make circuit breaker settings configurable per queue (max failures, reset timeout)\n- Configurable task expiry and queue polling intervals (idle delay)\n- Retry and backoff policies with dead-letter queue\n- Pause/resume controls and basic admin/health endpoints\n- Metrics and observability (Prometheus/OpenTelemetry), richer `get_queue_status()`\n- Distributed workers with coordination (locks) for multi-process scaling\n- Rate limiting per queue and burst control\n- CLI tooling for inspection and maintenance\n\n## Configuration\n\n### Redis Settings\n```python\nmanager = TaskQueueManager(\n    redis_url=\"redis://username:password@hostname:6379/0\",\n    cleanup_on_start=True  # Optional: clean stale tasks\n)\n```\n\n### Task Expiration\nTasks expire after 24 hours by default. This value is currently not configurable.\n\n## Error Handling\n\nKew provides comprehensive error handling:\n\n- `TaskAlreadyExistsError`: Task ID already in use\n- `TaskNotFoundError`: Task doesn't exist\n- `QueueNotFoundError`: Queue not configured\n- `QueueProcessorError`: Task processing failed\n\n```python\ntry:\n    await manager.submit_task(...)\nexcept TaskAlreadyExistsError:\n    # Handle duplicate task\nexcept QueueProcessorError as e:\n    # Handle processing error\n    print(f\"Task failed: {e}\")\n```\n\n## Contributing\n\nWe welcome contributions! Please check our [Contributing Guide](CONTRIBUTING.md) for details.\n\n## License\n\nMIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustrach%2Fkew","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjustrach%2Fkew","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustrach%2Fkew/lists"}