{"id":31871723,"url":"https://github.com/xtream1101/celery-throttle","last_synced_at":"2026-07-22T03:04:31.806Z","repository":{"id":317999073,"uuid":"1066846682","full_name":"xtream1101/celery-throttle","owner":"xtream1101","description":"Advanced rate limiting and queue management for Celery workers using Redis-based token bucket algorithms","archived":false,"fork":false,"pushed_at":"2025-10-04T12:29:15.000Z","size":185,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-04T20:45:55.030Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xtream1101.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-30T03:31:59.000Z","updated_at":"2025-10-04T12:25:36.000Z","dependencies_parsed_at":"2025-10-12T20:56:06.450Z","dependency_job_id":null,"html_url":"https://github.com/xtream1101/celery-throttle","commit_stats":null,"previous_names":["xtream1101/celery-throttle"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/xtream1101/celery-throttle","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xtream1101%2Fcelery-throttle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xtream1101%2Fcelery-throttle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xtream1101%2Fcelery-throttle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xtream1101%2Fcelery-throttle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xtream1101","download_url":"https://codeload.github.com/xtream1101/celery-throttle/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xtream1101%2Fcelery-throttle/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35744650,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-22T02:00:06.236Z","response_time":124,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-12T20:55:58.642Z","updated_at":"2026-07-22T03:04:31.801Z","avatar_url":"https://github.com/xtream1101.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Celery Throttle\n\n[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n**Advanced rate limiting and queue management for Celery workers using Redis-based token bucket algorithms.**\n\nCelery Throttle provides a robust solution for processing tasks with strict rate controls, ensuring efficient resource usage and compliance with API rate limits or processing constraints.\n\n## Why Celery Throttle?\n\n- **🎯 Works with Your Existing Tasks** - Apply rate limiting to any `@app.task` function without modification\n- **📝 Named Queues** - Use meaningful names like `\"email_notifications\"` instead of auto-generated UUIDs\n- **⚡ Precise Rate Limiting** - Redis Lua scripts ensure atomic, thread-safe rate control\n- **🔄 Dynamic Queue Management** - Create, modify, and remove queues on-the-fly\n- **💥 Burst Control** - Optional burst allowance for handling traffic spikes\n- **🏷️ Multi-Service Support** - Redis key prefixing enables multiple isolated services\n- **⚙️ Modern Configuration** - Streamlined setup with pydantic-settings\n\n## 📦 Installation\n\n```bash\npip install celery-throttle\n```\n\n**Requirements:**\n\n- Python 3.12+\n- Redis server\n- Celery 5.5+\n\n## 🚀 Quick Start\n\n### 1. Use Your Existing Celery Tasks\n\n```python\nfrom celery import Celery\nfrom celery_throttle import CeleryThrottle\n\n# Your existing Celery app and tasks\napp = Celery('myapp')\n\n@app.task\ndef send_email(to_email, subject, body):\n    # Your email sending logic\n    return {\"status\": \"sent\", \"to\": to_email}\n\n@app.task\ndef call_external_api(endpoint, data):\n    # Your API calling logic\n    return {\"status\": \"success\", \"endpoint\": endpoint}\n\n# Add rate limiting\nthrottle = CeleryThrottle(celery_app=app)\n\n# Create named queues with different rate limits\nthrottle.create_queue(\"10/1m\", \"email_queue\")      # 10 emails per minute\nthrottle.create_queue(\"100/1h\", \"api_queue\")       # 100 API calls per hour\n\n# Submit tasks with rate limiting\nthrottle.submit_task(\"email_queue\", \"myapp.send_email\", \"user@example.com\", \"Hello!\", \"Welcome!\")\nthrottle.submit_task(\"api_queue\", \"myapp.call_external_api\", \"/users/123\", {\"action\": \"update\"})\n```\n\n### 2. Run the Workers\n\nYou need two components:\n\n```bash\n# Terminal 1: Celery worker (processes the actual tasks)\ncelery -A myapp worker --loglevel=info --prefetch-multiplier=1\n\n# Terminal 2: Rate limiter dispatcher (manages the queues)\ncelery-throttle dispatcher --celery-app=myapp:app\n```\n\n**Important Worker Settings:**\n\n- `--prefetch-multiplier=1` - **Required** to prevent task prefetching which defeats rate limiting\n\nThat's it! Your tasks are now rate-limited.\n\n## 📋 Rate Limit Formats\n\nFlexible rate limit syntax with various time units and optional burst allowances.\n\n### Time Units\n\n- **Seconds**: `\"10/60s\"` (10 requests per 60 seconds)\n- **Minutes**: `\"10/5m\"` (10 requests per 5 minutes)\n- **Hours**: `\"4000/3h\"` (4000 requests per 3 hours)\n\n### Format Pattern\n\n```plaintext\nrequests/period[time_unit][:burst_allowance]\n```\n\n- `requests`: Number of requests allowed\n- `period`: Time period value\n- `time_unit`: `s` (seconds), `m` (minutes), or `h` (hours)\n- `burst_allowance`: Optional burst token capacity (defaults to 1)\n\n### Examples\n\n```python\n# Basic rate limits\nthrottle.create_queue(\"10/60s\", \"queue1\")     # 10 requests per 60 seconds\nthrottle.create_queue(\"5/2m\", \"queue2\")       # 5 requests per 2 minutes\nthrottle.create_queue(\"1000/1h\", \"queue3\")    # 1000 requests per 1 hour\n\n# Rate limits with burst allowance\nthrottle.create_queue(\"10/60s:5\", \"queue4\")   # 10/min with 5 burst tokens\nthrottle.create_queue(\"100/1h:20\", \"queue5\")  # 100/hour with 20 burst tokens\n```\n\n### Distribution Modes\n\n#### **Smooth Distribution (Default)**\n\nTasks are distributed evenly over time to prevent resource bursting:\n\n```python\nthrottle.create_queue(\"10/60s\")   # 1 task every 6 seconds\nthrottle.create_queue(\"100/1h\")   # 1 task every 36 seconds\n```\n\n#### **Burst Allowance (Optional)**\n\nAllow traffic spikes while maintaining overall rate limits:\n\n```python\nthrottle.create_queue(\"10/60s:5\")   # Allow up to 5 immediate tasks, then smooth distribution\n```\n\n## 🔧 Configuration\n\n### Simple Configuration\n\n```python\nfrom celery_throttle import CeleryThrottle\n\n# Load from environment variables (CELERY_THROTTLE_* prefix)\nthrottle = CeleryThrottle(celery_app=app)\n\n# Or use kwargs\nthrottle = CeleryThrottle(\n    celery_app=app,\n    target_queue=\"my-rate-limited-queue\",\n    queue_prefix=\"myapp\"\n)\n```\n\n### Advanced Configuration\n\n```python\nfrom celery_throttle import CeleryThrottle\nfrom celery_throttle.config import CeleryThrottleConfig, RedisConfig\n\nconfig = CeleryThrottleConfig(\n    app_name=\"my-rate-limited-app\",\n    target_queue=\"rate-limited-queue\",\n    queue_prefix=\"my_app\",\n    redis=RedisConfig(\n        host=\"localhost\",\n        port=6379,\n        db=1\n    )\n)\n\nthrottle = CeleryThrottle(celery_app=app, config=config)\n```\n\n### Environment Variables\n\n```bash\nexport CELERY_THROTTLE_REDIS_HOST=localhost\nexport CELERY_THROTTLE_REDIS_PORT=6379\nexport CELERY_THROTTLE_REDIS_DB=1\nexport CELERY_THROTTLE_APP_NAME=my-app\nexport CELERY_THROTTLE_TARGET_QUEUE=rate-limited-tasks\nexport CELERY_THROTTLE_QUEUE_PREFIX=my_app\n```\n\n**Configuration Precedence:**\n\n1. Kwargs (highest priority)\n2. Config object\n3. Environment variables\n4. Default values\n\nSee [CONFIGURATION.md](CONFIGURATION.md) for detailed configuration options.\n\n## 🎮 Command Line Interface\n\n### Dispatcher\n\n```bash\n# Start the task dispatcher\ncelery-throttle dispatcher --celery-app=myapp:app\n\n# With custom Redis settings\ncelery-throttle --redis-host=localhost --redis-port=6379 dispatcher --celery-app=myapp:app\n\n# With custom interval (default: 0.1s)\ncelery-throttle dispatcher --celery-app=myapp:app --interval=0.5\n```\n\n### Queue Management\n\n```bash\n# Create queues\ncelery-throttle queue create \"10/1m\"                  # Auto-generated name\ncelery-throttle queue create \"5/30s:3\"                # With burst allowance\n\n# List all queues\ncelery-throttle queue list\n\n# Show queue details\ncelery-throttle queue show \u003cqueue-name\u003e\n\n# Update rate limit\ncelery-throttle queue update \u003cqueue-name\u003e \"20/1m\"\n\n# Activate/deactivate queues\ncelery-throttle queue activate \u003cqueue-name\u003e\ncelery-throttle queue deactivate \u003cqueue-name\u003e\n\n# Remove queues\ncelery-throttle queue remove \u003cqueue-name\u003e\ncelery-throttle queue cleanup-empty                   # Remove empty queues\ncelery-throttle queue cleanup-all                     # Remove all queues\n```\n\n## 📊 Monitoring \u0026 Statistics\n\n```python\n# Get queue statistics\nstats = throttle.get_queue_stats(\"email_queue\")\nprint(f\"Waiting: {stats.tasks_waiting}\")\nprint(f\"Processing: {stats.tasks_processing}\")\nprint(f\"Completed: {stats.tasks_completed}\")\nprint(f\"Failed: {stats.tasks_failed}\")\n\n# Get rate limit status\nstatus = throttle.get_rate_limit_status(\"email_queue\")\nprint(f\"Available tokens: {status['available_tokens']}\")\nprint(f\"Next token in: {status['next_token_in']} seconds\")\n\n# List all queues\nfor queue in throttle.list_queues():\n    status = \"active\" if queue.active else \"inactive\"\n    print(f\"{queue.name}: {queue.rate_limit} ({status})\")\n```\n\n## 🏗️ Architecture\n\n### How It Works\n\n1. **Queue Creation** - Queues are created with rate limits and stored in Redis\n2. **Task Submission** - Tasks are processed immediately or queued based on token availability\n3. **Token Management** - Redis Lua scripts atomically manage token buckets for precise rate limiting\n4. **Task Dispatch** - Background dispatcher efficiently schedules queued tasks when tokens become available\n5. **Worker Processing** - Celery workers process tasks with strict rate limit compliance\n\n### Components\n\n- **TokenBucketRateLimiter** - Atomic Redis-based rate limiting with Lua scripts\n- **UniversalQueueManager** - Dynamic queue creation and lifecycle management\n- **RateLimitedTaskProcessor** - Celery integration with injectable app support\n- **RateLimitedTaskSubmitter** - Task submission with rate limit checking\n- **RateLimitedTaskDispatcher** - Efficient scheduling of queued tasks\n- **CLI** - Complete command-line management interface\n\n## 📚 Common Use Cases\n\n### API Rate Limiting\n\n```python\n# Different API endpoints with different limits\nthrottle.create_queue(\"300/15m\", \"twitter_api\")       # Twitter limit\nthrottle.create_queue(\"5000/1h\", \"github_api\")        # GitHub limit\nthrottle.create_queue(\"1/1s\", \"slack_api\")            # Slack limit\n\n# Submit tasks\nthrottle.submit_task(\"twitter_api\", \"myapp.post_tweet\", \"Hello world\")\nthrottle.submit_task(\"github_api\", \"myapp.create_issue\", \"bug\", \"Fix this\")\nthrottle.submit_task(\"slack_api\", \"myapp.send_message\", \"general\", \"Hi!\")\n```\n\n### Batch Processing\n\n```python\n# Process large dataset over time\nthrottle.create_queue(\"100/5m\", \"batch_processing\")\n\n# Submit batch of tasks\ntasks = [\n    (\"myapp.process_item\", (i, f\"item_{i}\"), {})\n    for i in range(5000)\n]\nresults = throttle.submit_multiple_tasks(\"batch_processing\", tasks)\n\nprint(f\"Immediately processed: {results['submitted']}\")\nprint(f\"Queued for later: {results['queued']}\")\n```\n\n### Multiple Services\n\n```python\n# Service A - Email notifications\nemail_config = CeleryThrottleConfig(\n    app_name=\"email_service\",\n    target_queue=\"email_rate_limited\",\n    queue_prefix=\"email\"\n)\nemail_throttle = CeleryThrottle(celery_app=app, config=email_config)\nemail_throttle.create_queue(\"100/1m\", \"notifications\")\n\n# Service B - API calls\napi_config = CeleryThrottleConfig(\n    app_name=\"api_service\",\n    target_queue=\"api_rate_limited\",\n    queue_prefix=\"api\"\n)\napi_throttle = CeleryThrottle(celery_app=app, config=api_config)\napi_throttle.create_queue(\"1000/1h\", \"external_calls\")\n\n# Each service has isolated Redis keys and Celery queues\n```\n\nSee [EXAMPLES.md](EXAMPLES.md) for more detailed examples.\n\n## 🔌 Advanced: Custom Task Processing\n\nYou can extend `RateLimitedTaskProcessor` to add instrumentation, metrics, or custom logging:\n\n```python\nfrom celery_throttle import CeleryThrottle, RateLimitedTaskProcessor\nimport time\n\nclass MetricsTaskProcessor(RateLimitedTaskProcessor):\n    def execute_task(self, queue_name: str, task_name: str, args: tuple, kwargs: dict):\n        # Add metrics, timing, custom logging, etc.\n        start_time = time.time()\n        logger.info(f\"⏱️  Starting {task_name} from {queue_name}\")\n\n        try:\n            result = super().execute_task(queue_name, task_name, args, kwargs)\n            duration = time.time() - start_time\n            logger.info(f\"✅ Completed {task_name} in {duration:.2f}s\")\n            # Send metrics to your monitoring system\n            return result\n        except Exception as e:\n            logger.error(f\"❌ Failed {task_name}: {e}\")\n            # Send error metrics\n            raise\n\n# Use custom processor\nthrottle = CeleryThrottle(celery_app=app, task_processor_cls=MetricsTaskProcessor)\n```\n\n**Note:** This is optional and only needed if you want to instrument task execution. For most use cases, the default processor is sufficient.\n\n## 🛡️ Best Practices\n\n### Worker Configuration\n\nFor optimal rate limiting, use these Celery worker settings:\n\n```bash\ncelery -A myapp worker \\\n  --prefetch-multiplier=1 \\\n  --without-mingle \\\n  --without-gossip \\\n  --loglevel=info\n```\n\n**Why these settings?**\n\n- `--prefetch-multiplier=1` - **Required** - Prevents workers from prefetching multiple tasks\n- `--without-mingle` - Improves startup time\n- `--without-gossip` - Reduces network chatter\n\n### Queue-Specific Workers\n\nFor better isolation, run dedicated workers:\n\n```bash\n# Worker for rate-limited tasks only\ncelery -A myapp worker --queues=rate-limited-queue --prefetch-multiplier=1\n\n# Worker for regular tasks\ncelery -A myapp worker --queues=celery\n```\n\n### Multiple Services on Same Redis\n\nUse different queue prefixes to isolate services:\n\n```python\n# Service A\nthrottle_a = CeleryThrottle(celery_app=app, queue_prefix=\"service_a\")\n\n# Service B\nthrottle_b = CeleryThrottle(celery_app=app, queue_prefix=\"service_b\")\n\n# Redis keys: service_a:* and service_b:* are completely isolated\n```\n\n## 🧪 Testing\n\n```bash\n# Install development dependencies\npip install celery-throttle[dev]\n\n# Run tests\npytest\n\n# Run with coverage\npytest --cov=celery_throttle\n\n# Run specific test file\npytest tests/test_rate_limiter.py -v\n```\n\n## 🤝 Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## 📖 Documentation\n\n- [Quick Reference](QUICKREF.md) - Fast lookup for common operations\n- [Configuration Guide](CONFIGURATION.md) - Detailed configuration options\n- [Examples](EXAMPLES.md) - Comprehensive usage examples\n- [Changelog](CHANGELOG.md) - Version history and changes\n- [API Reference](docs/API.md) - Complete API documentation (coming soon)\n\n## 🔗 Links\n\n- [GitHub Repository](https://github.com/xtream1101/celery-throttle)\n- [Issue Tracker](https://github.com/xtream1101/celery-throttle/issues)\n- [PyPI Package](https://pypi.org/project/celery-throttle/) (coming soon)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxtream1101%2Fcelery-throttle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxtream1101%2Fcelery-throttle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxtream1101%2Fcelery-throttle/lists"}