{"id":35177837,"url":"https://github.com/nolantrem/toller","last_synced_at":"2026-04-02T03:00:15.222Z","repository":{"id":286352521,"uuid":"961156056","full_name":"NolanTrem/toller","owner":"NolanTrem","description":"Lightweight Python library to make your async calls to microservices, LLMs, external APIs, etc., more robust and reliable.","archived":false,"fork":false,"pushed_at":"2025-05-13T07:10:20.000Z","size":1574,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-31T13:48:34.590Z","etag":null,"topics":["ai-workflows","async","async-workflows","asyncio","concurrency","flow-control","rate-limiting"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NolanTrem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-04-05T21:54:14.000Z","updated_at":"2025-12-02T15:25:58.000Z","dependencies_parsed_at":"2025-04-05T22:26:09.403Z","dependency_job_id":null,"html_url":"https://github.com/NolanTrem/toller","commit_stats":null,"previous_names":["nolantrem/toller"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/NolanTrem/toller","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NolanTrem%2Ftoller","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NolanTrem%2Ftoller/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NolanTrem%2Ftoller/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NolanTrem%2Ftoller/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NolanTrem","download_url":"https://codeload.github.com/NolanTrem/toller/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NolanTrem%2Ftoller/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31294808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T01:43:37.129Z","status":"online","status_checked_at":"2026-04-02T02:00:08.535Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-workflows","async","async-workflows","asyncio","concurrency","flow-control","rate-limiting"],"created_at":"2025-12-28T23:14:49.783Z","updated_at":"2026-04-02T03:00:15.050Z","avatar_url":"https://github.com/NolanTrem.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"logo.png\" alt=\"Toller Logo\" width=\"400\"/\u003e\n\u003c/p\u003e\n\n## What is Toller?\n\nToller is a lightweight Python library designed to make your asynchronous calls to microservices, GenAI solutions, external APIs, etc., more robust and reliable. It provides a simple yet powerful decorator to add **rate limiting**, **retries** (with backoff \u0026 jitter), and **circuit breaking** to your `async` functions with minimal boilerplate.\n\nJust as the [Nova Scotia Duck Tolling Retriever](https://www.akc.org/dog-breeds/nova-scotia-duck-tolling-retriever/) lures and guides ducks, Toller \"lures\" unruly asynchronous tasks into well-managed, predictable flows, guiding the overall execution path and making concurrency easier to reason about.\n\n## Why Toller?\n\nModern applications that integrate with numerous LLMs, vector databases, and other microservices, face a constant challenge: external services can be unreliable. They might be temporarily down, enforce rate limits, or return transient errors.\n\nBuilding robust applications in this environment means every external call needs careful handling, but repeating this logic for every API call leads to boilerplate, inconsistency, and often, poorly managed asynchronous processes. **Toller was built to solve this.** It provides a declarative way to add these resilience patterns.\n\nToller offers this standard, both for client-side calls and potentially for protecting server-side resources.\n\n## Features\n\n*   **`@toller.task` Decorator:** A single, easy-to-use decorator to apply all resilience patterns.\n*   **Rate Limiting:**\n    *   Async-safe Token Bucket-based `CallRateLimiter`.\n    *   Configurable call rates and burst capacity.\n    *   Automatic asynchronous waiting when limits are hit.\n*   **Retries:**\n    *   Strategies: Max attempts, fixed delay, exponential backoff with jitter.\n    *   Conditional retries on specific exceptions (e.g., `TransientError`).\n    *   Conditional stopping on specific exceptions (e.g., `FatalError`).\n    *   Raises `MaxRetriesExceeded` wrapping the last encountered error.\n*   **Circuit Breaker:**\n    *   Standard states: CLOSED, OPEN, HALF_OPEN.\n    *   Configurable failure thresholds and recovery timeouts.\n    *   Trips on specified exceptions (e.g., `MaxRetriesExceeded`, or custom fatal errors).\n    *   Prevents calls to a failing service, allowing it time to recover.\n*   **Custom Exception Hierarchy:** Clear exceptions like `OpenCircuitError`, `TransientError`, `FatalError` for better error handling.\n*   **Async Native:** Built for `asyncio`.\n*   **Lightweight:** Minimal dependencies.\n\n## Installation\n\n```bash\npip install toller\n```\n\n## Usage and Examples\n\n### Example 1: Basic Resilience for Generative AI Calls\n\u003cdetails open\u003e\n    For a function that calls out to an LLM, we want to handle rate limits, retry on temporary server issues, and stop if the service is truly down.\n\n```python\nimport asyncio\nimport random\nimport toller\nfrom toller import TransientError, FatalError, MaxRetriesExceeded, OpenCircuitError\n\n# Define potential API errors\nclass LLMRateLimitError(TransientError): pass\nclass LLMServerError(TransientError): pass\nclass LLMInputError(FatalError): pass # e.g., prompt too long\n\n# Simulate an LLM call\nLLM_DOWN_FOR_DEMO = 0 # Counter for demoing circuit breaker\nasync def call_llm_api(prompt: str):\n    global LLM_DOWN_FOR_DEMO\n    print(f\"LLM API: Processing '{prompt[:20]}...' (Attempt for this task)\")\n    await asyncio.sleep(random.uniform(0.1, 0.3)) # Network latency\n\n    if LLM_DOWN_FOR_DEMO \u003e 0:\n        LLM_DOWN_FOR_DEMO -=1\n        print(\"LLM API: Simulating 503 Service Unavailable\")\n        raise LLMServerError(\"LLM service is temporarily down\")\n    if random.random() \u003c 0.2: # 20% chance of a transient rate limit error\n        print(\"LLM API: Simulating 429 Rate Limit\")\n        raise LLMRateLimitError(\"Hit LLM rate limit\")\n    if len(prompt) \u003c 5:\n        print(\"LLM API: Simulating 400 Bad Request (prompt too short)\")\n        raise LLMInputError(\"Prompt is too short\")\n    \n    return f\"LLM Response for '{prompt[:20]}...': Generated text.\"\n\n# Apply Toller\n@toller.task(\n    # Rate Limiter: 60 calls per minute (1 per sec), burst 5\n    rl_calls_per_second=1.0,  # 60 RPM / 60s\n    rl_max_burst_calls=5,\n    \n    # Retries: 3 attempts on transient LLM errors\n    retry_max_attempts=3,\n    retry_delay=1.0, # Start with 1s delay for LLM errors\n    retry_backoff=2.0,\n    retry_on_exception=(LLMRateLimitError, LLMServerError),\n    retry_stop_on_exception=(LLMInputError,), # Don't retry bad input\n\n    # Circuit Breaker: Opens if retries fail 2 times consecutively\n    cb_failure_threshold=2, # Low threshold for demo\n    cb_recovery_timeout=20.0, # Wait 20s before one test call\n    cb_expected_exception=MaxRetriesExceeded # CB trips when all retries are exhausted\n)\nasync def get_llm_completion(prompt: str):\n    return await call_llm_api(prompt)\n\nasync def run_example1():\n    print(\"Example 1: Basic Resilience for Generative AI Calls\")\n    prompts = [\n        \"Tell me a story about a brave duck.\",\n        \"Explain async programming.\",\n        \"Short\", # This will cause a FatalError (LLMInputError)\n        \"Another valid prompt after fatal.\",\n        \"Prompt to trigger server errors 1\", # Will hit retry then MaxRetriesExceeded\n        \"Prompt to trigger server errors 2\", # Will hit retry then MaxRetriesExceeded, tripping CB\n        \"Prompt after CB should open\", # Should hit OpenCircuitError\n    ]\n\n    # Simulated LLM service downtime for the relevant prompts\n    global LLM_DOWN_FOR_DEMO\n    LLM_DOWN_FOR_DEMO = 4\n\n    for i, p in enumerate(prompts):\n        print(f\"\\Sending request: '{p}'\")\n        try:\n            result = await get_llm_completion(p)\n            print(f\"Success: {result}\")\n        except MaxRetriesExceeded as e:\n            print(f\"Toller: Max retries exceeded. Last error: {type(e.last_exception).__name__}: {e.last_exception}\")\n        except OpenCircuitError as e:\n            print(f\"Toller: Circuit is open! {e}. Further calls blocked temporarily.\")\n            if i == len(prompts) - 2: # If this is the call just before the last one\n                print(\"Waiting for circuit breaker recovery timeout for demo...\")\n                await asyncio.sleep(21) # Wait for CB to go HALF_OPEN\n        except FatalError as e:\n            print(f\"Toller: Fatal error, no retries. Error: {type(e).__name__}: {e}\")\n        except Exception as e:\n            print(f\"Toller: Unexpected error. Type: {type(e).__name__}, Error: {e}\")\n        \n        await asyncio.sleep(0.3) # Small pause between top-level requests to see rate limiter too\n\nif __name__ == \"__main__\":\n    asyncio.run(run_example1())\n```\n\u003c/details\u003e\n\n\n### Example 2: Shared Rate Limiter for Multiple Related API Calls\n\u003cdetails\u003e\n    Often, different API endpoints for the same service share an overall rate limit.\n\n```python\nimport asyncio\nimport time\nimport toller\nfrom toller import CallRateLimiter # For creating a shared instance\n\n# Assume these two functions call endpoints that share a single rate limit pool\nshared_api_rl = CallRateLimiter(calls_per_second=2, max_burst_calls=2, name=\"MyServiceSharedRL\")\n\n@toller.task(\n    rate_limiter_instance=shared_api_rl,\n    # Disable retry/CB for this simple RL demo\n    enable_retry=False, enable_circuit_breaker=False \n)\nasync def call_endpoint_a(item_id: int):\n    print(f\"Calling A for {item_id}...\")\n    await asyncio.sleep(0.1)\n    return f\"A {item_id} done\"\n\n@toller.task(\n    rate_limiter_instance=shared_api_rl,\n    enable_retry=False, enable_circuit_breaker=False\n)\nasync def call_endpoint_b(item_id: int):\n    print(f\"Calling B for {item_id}...\")\n    await asyncio.sleep(0.1)\n    return f\"B {item_id} done\"\n\nasync def run_example2():\n    print(\"\\nExample 2: Shared Rate Limiter\")\n    tasks = []\n    # These 4 calls will exceed the burst of 2 for the shared limiter (rate 2/sec), so, some will be delayed.\n    tasks.append(call_endpoint_a(1))\n    tasks.append(call_endpoint_b(1))\n    tasks.append(call_endpoint_a(2))\n    tasks.append(call_endpoint_b(2))\n\n    start_time = time.time()\n    results = await asyncio.gather(*tasks)\n    duration = time.time() - start_time\n\n    for res in results:\n        print(f\"Shared RL Result: {res}\")\n    print(f\"Total time for 4 calls with shared RL (2/sec, burst 2): {duration:.2f}s (expected \u003e ~1.0s)\")\n\nif __name__ == \"__main__\":\n    asyncio.run(run_example2())\n```\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnolantrem%2Ftoller","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnolantrem%2Ftoller","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnolantrem%2Ftoller/lists"}