{"id":35432570,"url":"https://github.com/joshrotenberg/tower-resilience","last_synced_at":"2026-03-27T20:51:47.996Z","repository":{"id":318711119,"uuid":"1071877745","full_name":"joshrotenberg/tower-resilience","owner":"joshrotenberg","description":"Resilience features for tower","archived":false,"fork":false,"pushed_at":"2026-03-04T01:54:57.000Z","size":1218,"stargazers_count":75,"open_issues_count":2,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-04T06:18:09.392Z","etag":null,"topics":["resilience","tokio","tower"],"latest_commit_sha":null,"homepage":"https://docs.rs/tower-resilience/latest/tower_resilience/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joshrotenberg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-08T00:11:26.000Z","updated_at":"2026-03-04T01:55:00.000Z","dependencies_parsed_at":"2025-10-10T14:27:55.274Z","dependency_job_id":"773de110-a638-497c-b1c8-2978bc46511f","html_url":"https://github.com/joshrotenberg/tower-resilience","commit_stats":null,"previous_names":["joshrotenberg/tower-resilience"],"tags_count":216,"template":false,"template_full_name":null,"purl":"pkg:github/joshrotenberg/tower-resilience","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshrotenberg%2Ftower-resilience","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshrotenberg%2Ftower-resilience/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshrotenberg%2Ftower-resilience/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshrotenberg%2Ftower-resilience/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joshrotenberg","download_url":"https://codeload.github.com/joshrotenberg/tower-resilience/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshrotenberg%2Ftower-resilience/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30246874,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-08T05:41:50.788Z","status":"ssl_error","status_checked_at":"2026-03-08T05:41:39.075Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["resilience","tokio","tower"],"created_at":"2026-01-02T21:15:24.727Z","updated_at":"2026-03-08T06:03:05.343Z","avatar_url":"https://github.com/joshrotenberg.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tower-resilience\n\n[![Crates.io](https://img.shields.io/crates/v/tower-resilience.svg)](https://crates.io/crates/tower-resilience)\n[![Documentation](https://docs.rs/tower-resilience/badge.svg)](https://docs.rs/tower-resilience)\n[![License](https://img.shields.io/badge/license-MIT%2FApache--2.0-blue.svg)](LICENSE-MIT)\n[![Rust Version](https://img.shields.io/badge/rust-1.64.0%2B-blue.svg)](https://www.rust-lang.org)\n\nA comprehensive resilience and fault-tolerance toolkit for [Tower](https://github.com/tower-rs/tower) services, inspired by [Resilience4j](https://resilience4j.readme.io/).\n\n## About\n\nResilience patterns for [Tower](https://docs.rs/tower) services, inspired by [Resilience4j](https://resilience4j.readme.io/). Includes circuit breaker, bulkhead, retry with backoff, rate limiting, and more.\n\n## Resilience Patterns\n\n- **Adaptive Concurrency** - Dynamic concurrency limiting using AIMD or Vegas algorithms\n- **Bulkhead** - Isolates resources to prevent system-wide failures\n- **Cache** - Response memoization to reduce load\n- **Chaos** - Inject failures and latency for testing resilience (development/testing only)\n- **Circuit Breaker** - Prevents cascading failures by stopping calls to failing services\n- **Coalesce** - Deduplicates concurrent identical requests (singleflight pattern)\n- **Executor** - Delegates request processing to dedicated executors for parallelism\n- **Fallback** - Graceful degradation when services fail\n- **Hedge** - Reduces tail latency by racing redundant requests\n- **Health Check** - Proactive health monitoring with intelligent resource selection\n- **Outlier Detection** - Fleet-aware instance ejection based on consecutive error tracking\n- **Rate Limiter** - Controls request rate with fixed or sliding window algorithms\n- **Reconnect** - Automatic reconnection with configurable backoff strategies\n- **Retry** - Intelligent retry with exponential backoff, jitter, and retry budgets\n- **Router** - Weighted traffic routing for canary deployments and progressive rollout\n- **Time Limiter** - Advanced timeout handling with cancellation support\n\n## Quick Start\n\n```toml\n[dependencies]\ntower-resilience = \"0.9\"\ntower = \"0.5\"\n```\n\n```rust\nuse tower::{Layer, ServiceBuilder};\nuse tower_resilience::prelude::*;\n\nlet circuit_breaker = CircuitBreakerLayer::builder()\n    .failure_rate_threshold(0.5)\n    .build();\n\nlet service = ServiceBuilder::new()\n    .layer(circuit_breaker)\n    .layer(BulkheadLayer::builder()\n        .max_concurrent_calls(10)\n        .build())\n    .service(my_service);\n```\n\n## Presets: Get Started in One Line\n\nEvery pattern includes **preset configurations** with sensible defaults. Start immediately without tuning parameters - customize later when you need to:\n\n```rust\nuse tower_resilience::retry::RetryLayer;\nuse tower_resilience::circuitbreaker::CircuitBreakerLayer;\nuse tower_resilience::ratelimiter::RateLimiterLayer;\nuse tower_resilience::bulkhead::BulkheadLayer;\nuse tower_resilience::timelimiter::TimeLimiterLayer;\nuse tower_resilience::hedge::HedgeLayer;\n\n// Retry with exponential backoff (3 attempts, 100ms base)\nlet retry = RetryLayer::\u003c(), (), MyError\u003e::exponential_backoff().build();\n\n// Circuit breaker with balanced defaults\nlet breaker = CircuitBreakerLayer::standard().build();\n\n// Rate limit to 100 requests per second\nlet limiter = RateLimiterLayer::per_second(100).build();\n\n// Limit to 50 concurrent requests\nlet bulkhead = BulkheadLayer::medium().build();\n\n// 5 second timeout with cancellation\nlet timeout = TimeLimiterLayer::standard().build();\n\n// Reduce tail latency with hedged requests\nlet hedge = HedgeLayer::standard();\n```\n\n### Available Presets\n\n| Pattern | Presets | Description |\n|---------|---------|-------------|\n| **Bulkhead** | `small()` | 10 concurrent calls |\n| | `medium()` | 50 concurrent calls |\n| | `large()` | 200 concurrent calls |\n| **Circuit Breaker** | `standard()` | 50% threshold, 100 calls - balanced |\n| | `fast_fail()` | 25% threshold, 20 calls - fail fast |\n| | `tolerant()` | 75% threshold, 200 calls - high tolerance |\n| **Hedge** | `conservative()` | 500ms delay, 2 attempts |\n| | `standard()` | 100ms delay, 3 attempts |\n| | `aggressive()` | 50ms delay, 5 attempts |\n| **Rate Limiter** | `per_second(n)` | n requests per second |\n| | `per_minute(n)` | n requests per minute |\n| | `burst(rate, size)` | Sustained rate with burst capacity |\n| **Retry** | `exponential_backoff()` | 3 attempts, 100ms base - balanced default |\n| | `aggressive()` | 5 attempts, 50ms base - fast recovery |\n| | `conservative()` | 2 attempts, 500ms base - minimal overhead |\n| **Time Limiter** | `fast()` | 1s timeout, cancel on timeout |\n| | `standard()` | 5s timeout, cancel on timeout |\n| | `slow()` | 30s timeout, cancel on timeout |\n| | `streaming()` | 60s timeout, no cancellation |\n\nPresets return builders, so you can customize any setting:\n\n```rust\n// Start with a preset, override what you need\nlet breaker = CircuitBreakerLayer::fast_fail()\n    .name(\"payment-api\")           // Add observability\n    .wait_duration_in_open(Duration::from_secs(30))  // Custom recovery time\n    .build();\n```\n\n## Examples\n\n### Adaptive Concurrency\n\nDynamically adjust concurrency limits based on observed latency and error rates:\n\n```rust\nuse tower_resilience::adaptive::{AdaptiveLimiterLayer, Aimd, Vegas};\nuse tower::ServiceBuilder;\nuse std::time::Duration;\n\n// AIMD: Classic TCP-style congestion control\n// Increases limit on success, decreases on failure/high latency\nlet layer = AdaptiveLimiterLayer::new(\n    Aimd::builder()\n        .initial_limit(10)\n        .min_limit(1)\n        .max_limit(100)\n        .increase_by(1)                           // Add 1 on success\n        .decrease_factor(0.5)                     // Halve on failure\n        .latency_threshold(Duration::from_millis(100))\n        .build()\n);\n\n// Vegas: More stable, uses RTT to estimate queue depth\nlet layer = AdaptiveLimiterLayer::new(\n    Vegas::builder()\n        .initial_limit(10)\n        .alpha(3)    // Increase when queue \u003c 3\n        .beta(6)     // Decrease when queue \u003e 6\n        .build()\n);\n\nlet service = ServiceBuilder::new()\n    .layer(layer)\n    .service(my_service);\n```\n\nUse cases:\n- **Auto-tuning**: No manual concurrency limit configuration needed\n- **Variable backends**: Adapts to changing downstream capacity\n- **Load shedding**: Automatically reduces load when backends struggle\n\n**Full examples:** [adaptive.rs](examples/adaptive.rs)\n\n### Bulkhead\n\nLimit concurrent requests to prevent resource exhaustion:\n\n```rust\nuse tower_resilience::bulkhead::BulkheadLayer;\nuse std::time::Duration;\n\nlet layer = BulkheadLayer::builder()\n    .name(\"worker-pool\")\n    .max_concurrent_calls(10)                    // Max 10 concurrent\n    .max_wait_duration(Duration::from_secs(5))        // Wait up to 5s\n    .on_call_permitted(|concurrent| {\n        println!(\"Request permitted (concurrent: {})\", concurrent);\n    })\n    .on_call_rejected(|max| {\n        println!(\"Request rejected (max: {})\", max);\n    })\n    .build();\n\nlet service = layer.layer(my_service);\n```\n\n**Full examples:** [bulkhead.rs](examples/bulkhead.rs) | [bulkhead_advanced.rs](crates/tower-resilience-bulkhead/examples/bulkhead_advanced.rs)\n\n### Cache\n\nCache responses to reduce load on expensive operations:\n\n```rust\nuse tower_resilience::cache::{CacheLayer, EvictionPolicy};\nuse std::time::Duration;\n\nlet layer = CacheLayer::builder()\n    .max_size(1000)\n    .ttl(Duration::from_secs(300))                 // 5 minute TTL\n    .eviction_policy(EvictionPolicy::Lru)          // LRU, LFU, or FIFO\n    .key_extractor(|req: \u0026Request| req.id.clone())\n    .on_hit(|| println!(\"Cache hit!\"))\n    .on_miss(|| println!(\"Cache miss\"))\n    .build();\n\nlet service = layer.layer(my_service);\n```\n\n**Full examples:** [cache.rs](examples/cache.rs) | [cache_example.rs](crates/tower-resilience-cache/examples/cache_example.rs)\n\n### Chaos (Testing Only)\n\nInject failures and latency to test your resilience patterns:\n\n```rust\nuse tower_resilience::chaos::ChaosLayer;\nuse std::time::Duration;\n\n// Types inferred from closure signature - no type parameters needed!\nlet chaos = ChaosLayer::builder()\n    .name(\"test-chaos\")\n    .error_rate(0.1)                               // 10% of requests fail\n    .error_fn(|_req: \u0026String| std::io::Error::new(\n        std::io::ErrorKind::Other, \"chaos!\"\n    ))\n    .latency_rate(0.2)                             // 20% delayed\n    .min_latency(Duration::from_millis(50))\n    .max_latency(Duration::from_millis(200))\n    .seed(42)                                      // Deterministic chaos\n    .build();\n\nlet service = chaos.layer(my_service);\n```\n\n**WARNING**: Only use in development/testing environments. Never in production.\n\n**Full examples:** [chaos.rs](examples/chaos.rs) | [chaos_example.rs](crates/tower-resilience-chaos/examples/chaos_example.rs)\n\n### Circuit Breaker\n\nPrevent cascading failures by opening the circuit when error rate exceeds threshold:\n\n```rust\nuse tower::Layer;\nuse tower_resilience::circuitbreaker::CircuitBreakerLayer;\nuse std::time::Duration;\n\nlet layer = CircuitBreakerLayer::builder()\n    .name(\"api-circuit\")\n    .failure_rate_threshold(0.5)          // Open at 50% failure rate\n    .sliding_window_size(100)              // Track last 100 calls\n    .wait_duration_in_open(Duration::from_secs(60))  // Stay open 60s\n    .on_state_transition(|from, to| {\n        println!(\"Circuit breaker: {:?} -\u003e {:?}\", from, to);\n    })\n    .build();\n\nlet service = layer.layer(my_service);\n```\n\n**Full examples:** [circuitbreaker.rs](examples/circuitbreaker.rs) | [circuitbreaker_fallback.rs](crates/tower-resilience-circuitbreaker/examples/circuitbreaker_fallback.rs) | [circuitbreaker_health_check.rs](crates/tower-resilience-circuitbreaker/examples/circuitbreaker_health_check.rs)\n\n### Coalesce\n\nDeduplicate concurrent identical requests (singleflight pattern):\n\n```rust\nuse tower_resilience::coalesce::CoalesceLayer;\nuse tower::ServiceBuilder;\n\n// Coalesce by request ID - concurrent requests for same ID share one execution\nlet layer = CoalesceLayer::new(|req: \u0026Request| req.id.clone());\n\nlet service = ServiceBuilder::new()\n    .layer(layer)\n    .service(my_service);\n\n// Use with cache to prevent stampede on cache miss\nlet service = ServiceBuilder::new()\n    .layer(cache_layer)      // Check cache first\n    .layer(coalesce_layer)   // Coalesce cache misses\n    .service(backend);\n```\n\nUse cases:\n- **Cache stampede prevention**: When cache expires, only one request refreshes it\n- **Expensive computations**: Deduplicate identical report generation requests\n- **Rate-limited APIs**: Reduce calls to external APIs by coalescing identical requests\n\n**Note:** Response and error types must implement `Clone` to be shared with all waiters.\n\n### Executor\n\nDelegate request processing to dedicated executors for parallel execution:\n\n```rust\nuse tower_resilience::executor::ExecutorLayer;\nuse tower::ServiceBuilder;\n\n// Use a dedicated runtime for CPU-heavy work\nlet compute_runtime = tokio::runtime::Builder::new_multi_thread()\n    .worker_threads(8)\n    .thread_name(\"compute\")\n    .build()\n    .unwrap();\n\nlet layer = ExecutorLayer::new(compute_runtime.handle().clone());\n\n// Or use the current runtime\nlet layer = ExecutorLayer::current();\n\nlet service = ServiceBuilder::new()\n    .layer(layer)\n    .service(my_service);\n```\n\nUse cases:\n- **CPU-bound processing**: Parallelize CPU-intensive request handling\n- **Runtime isolation**: Process requests on a dedicated runtime\n- **Thread pool delegation**: Use specific thread pools for certain workloads\n\n### Fallback\n\nProvide fallback responses when the primary service fails:\n\n```rust\nuse tower_resilience::fallback::FallbackLayer;\n\n// Return a static fallback value on error\nlet layer = FallbackLayer::\u003cRequest, Response, MyError\u003e::value(\n    Response::default()\n);\n\n// Or compute fallback from the error\nlet layer = FallbackLayer::\u003cRequest, Response, MyError\u003e::from_error(|err| {\n    Response::error_response(err)\n});\n\n// Or use a backup service\nlet layer = FallbackLayer::\u003cRequest, Response, MyError\u003e::service(|req| async {\n    backup_service.call(req).await\n});\n\nlet service = layer.layer(primary_service);\n```\n\n### Hedge\n\nReduce tail latency by firing backup requests after a delay:\n\n```rust\nuse tower_resilience::hedge::HedgeLayer;\nuse std::time::Duration;\n\n// Fire a hedge request if primary takes \u003e 100ms\nlet layer = HedgeLayer::builder()\n    .delay(Duration::from_millis(100))\n    .max_hedged_attempts(2)\n    .build();\n\n// Or fire all requests in parallel (no delay)\nlet layer = HedgeLayer::\u003c(), String, MyError\u003e::builder()\n    .no_delay()\n    .max_hedged_attempts(3)\n    .build();\n\nlet service = layer.layer(my_service);\n```\n\n**Note:** Hedge requires `Req: Clone` (requests are cloned for parallel execution) and `E: Clone` (for error handling). If your types don't implement Clone, consider wrapping them in `Arc`.\n\n**Full examples:** [hedge.rs](examples/hedge.rs)\n\n### Health Check\n\nProactive health monitoring with intelligent resource selection:\n\n```rust\nuse tower_resilience::healthcheck::{HealthCheckWrapper, HealthStatus, SelectionStrategy};\nuse std::time::Duration;\n\n// Create wrapper with multiple resources\nlet wrapper = HealthCheckWrapper::builder()\n    .with_context(primary_db, \"primary\")\n    .with_context(secondary_db, \"secondary\")\n    .with_checker(|db| async move {\n        match db.ping().await {\n            Ok(_) =\u003e HealthStatus::Healthy,\n            Err(_) =\u003e HealthStatus::Unhealthy,\n        }\n    })\n    .with_interval(Duration::from_secs(5))\n    .with_selection_strategy(SelectionStrategy::RoundRobin)\n    .build();\n\n// Start background health checking\nwrapper.start().await;\n\n// Get a healthy resource\nif let Some(db) = wrapper.get_healthy().await {\n    // Use healthy database\n}\n```\n\n**Note:** Health Check is not a Tower layer - it's a wrapper pattern for managing multiple resources with automatic failover.\n\n**Full examples:** [healthcheck_basic.rs](crates/tower-resilience-healthcheck/examples/healthcheck_basic.rs)\n\n### Outlier Detection\n\nFleet-aware instance ejection that tracks per-instance health and routes around unhealthy backends:\n\n```rust\nuse tower_resilience::outlier::{OutlierDetectionLayer, OutlierDetector};\nuse tower::{ServiceBuilder, service_fn};\nuse std::time::Duration;\n\n// Shared detector coordinates ejection across all instances\nlet detector = OutlierDetector::new()\n    .max_ejection_percent(50)\n    .base_ejection_duration(Duration::from_secs(30));\n\n// Register instances (eject after 5 consecutive errors)\ndetector.register(\"backend-1\", 5);\ndetector.register(\"backend-2\", 5);\n\n// Per-instance layers share the detector\nlet layer = OutlierDetectionLayer::builder()\n    .detector(detector.clone())\n    .instance_name(\"backend-1\")\n    .build();\n\nlet service = ServiceBuilder::new()\n    .layer(layer)\n    .service(service_fn(|req: String| async move { Ok::\u003c_, std::io::Error\u003e(req) }));\n```\n\nKey features:\n- **Backpressure mode** (default): `poll_ready` returns `Pending` for ejected instances, integrating naturally with Tower load balancers\n- **Fleet protection**: `max_ejection_percent` prevents cascading ejections\n- **Exponential backoff**: Repeatedly-ejected instances stay out longer\n\n### Rate Limiter\n\nControl request rate to protect downstream services:\n\n```rust\nuse tower_resilience::ratelimiter::RateLimiterLayer;\nuse std::time::Duration;\n\nlet layer = RateLimiterLayer::builder()\n    .limit_for_period(100)                      // 100 requests\n    .refresh_period(Duration::from_secs(1))     // per second\n    .timeout_duration(Duration::from_millis(500))  // Wait up to 500ms\n    .on_permit_acquired(|wait| {\n        println!(\"Request permitted (waited {:?})\", wait);\n    })\n    .build();\n\nlet service = layer.layer(my_service);\n```\n\n**Full examples:** [ratelimiter.rs](examples/ratelimiter.rs) | [ratelimiter_example.rs](crates/tower-resilience-ratelimiter/examples/ratelimiter_example.rs)\n\n### Reconnect\n\nAutomatically reconnect on connection failures with configurable backoff:\n\n```rust\nuse tower_resilience::reconnect::{ReconnectLayer, ReconnectConfig, ReconnectPolicy};\nuse std::time::Duration;\n\nlet layer = ReconnectLayer::new(\n    ReconnectConfig::builder()\n        .policy(ReconnectPolicy::exponential(\n            Duration::from_millis(100),  // Start at 100ms\n            Duration::from_secs(5),       // Max 5 seconds\n        ))\n        .max_attempts(10)\n        .retry_on_reconnect(true)         // Retry request after reconnecting\n        .connection_errors_only()          // Only reconnect on connection errors\n        .on_state_change(|from, to| {\n            println!(\"Connection: {:?} -\u003e {:?}\", from, to);\n        })\n        .build()\n);\n\nlet service = layer.layer(my_service);\n```\n\n**Full examples:** [reconnect.rs](examples/reconnect.rs) | [reconnect_basic.rs](crates/tower-resilience-reconnect/examples/reconnect_basic.rs) | [reconnect_custom_policy.rs](crates/tower-resilience-reconnect/examples/reconnect_custom_policy.rs)\n\n### Retry\n\nRetry failed requests with exponential backoff and jitter:\n\n```rust\nuse tower_resilience::retry::RetryLayer;\nuse std::time::Duration;\n\nlet layer = RetryLayer::\u003c(), (), MyError\u003e::builder()\n    .max_attempts(5)\n    .exponential_backoff(Duration::from_millis(100))\n    .on_retry(|attempt, delay| {\n        println!(\"Retrying (attempt {}, delay {:?})\", attempt, delay);\n    })\n    .on_success(|attempts| {\n        println!(\"Success after {} attempts\", attempts);\n    })\n    .build();\n\nlet service = layer.layer(my_service);\n```\n\n**Full examples:** [retry.rs](examples/retry.rs) | [retry_example.rs](crates/tower-resilience-retry/examples/retry_example.rs)\n\n### Router\n\nWeighted traffic routing for canary deployments and progressive rollout:\n\n```rust\nuse tower::util::BoxService;\nuse tower_resilience::router::WeightedRouter;\n\n// Route 90% to stable, 10% to canary\nlet svc_v1: BoxService\u003cString, String, MyError\u003e =\n    BoxService::new(tower::service_fn(|req: String| async move {\n        Ok(format!(\"v1: {}\", req))\n    }));\nlet svc_v2: BoxService\u003cString, String, MyError\u003e =\n    BoxService::new(tower::service_fn(|req: String| async move {\n        Ok(format!(\"v2: {}\", req))\n    }));\n\nlet router = WeightedRouter::builder()\n    .name(\"canary-deploy\")\n    .route(svc_v1, 90)\n    .route(svc_v2, 10)\n    .on_request_routed(|idx, weight| {\n        println!(\"Routed to backend {} (weight: {})\", idx, weight);\n    })\n    .build();\n```\n\nKey features:\n- **Deterministic** (default): Atomic counter for exact, repeatable distribution\n- **Random**: Probabilistic selection for high-volume statistical distribution\n- **Composable**: Wrap each backend with circuit breakers, bulkheads, etc.\n\n**Note:** Router is a standalone `Service`, not a `Layer`. Use `BoxService` to type-erase different backend implementations.\n\n**Full examples:** [router.rs](examples/router.rs)\n\n### Time Limiter\n\nEnforce timeouts on operations with configurable cancellation:\n\n```rust\nuse tower_resilience::timelimiter::TimeLimiterLayer;\nuse std::time::Duration;\n\nlet layer = TimeLimiterLayer::builder()\n    .timeout_duration(Duration::from_secs(30))\n    .cancel_running_future(true)  // Cancel on timeout\n    .on_timeout(|| {\n        println!(\"Operation timed out!\");\n    })\n    .build();\n\nlet service = layer.layer(my_service);\n```\n\n**Full examples:** [timelimiter.rs](examples/timelimiter.rs) | [timelimiter_example.rs](crates/tower-resilience-timelimiter/examples/timelimiter_example.rs)\n\n## Error Handling\n\nWhen composing multiple resilience layers, each layer has its own error type (e.g., `CircuitBreakerError`, `BulkheadError`). The `ResilienceError\u003cE\u003e` type unifies these into a single error type, eliminating boilerplate.\n\n### The Problem\n\nWithout a unified error type, you'd need `From` implementations for every layer combination:\n\n```rust\n// Without ResilienceError: ~80 lines of boilerplate for 4 layers\nimpl From\u003cBulkheadError\u003e for ServiceError { /* ... */ }\nimpl From\u003cCircuitBreakerError\u003e for ServiceError { /* ... */ }\nimpl From\u003cRateLimiterError\u003e for ServiceError { /* ... */ }\nimpl From\u003cTimeLimiterError\u003e for ServiceError { /* ... */ }\n```\n\n### The Solution\n\nUse `ResilienceError\u003cE\u003e` as your service error type - all layer errors automatically convert:\n\n```rust\nuse tower_resilience::core::ResilienceError;\n\n// Your application error\n#[derive(Debug, Clone)]\nenum AppError {\n    DatabaseDown,\n    InvalidRequest,\n}\n\n// That's it! Zero From implementations needed\ntype ServiceError = ResilienceError\u003cAppError\u003e;\n```\n\n### Pattern Matching\n\nHandle different failure modes explicitly:\n\n```rust\nuse tower_resilience::core::ResilienceError;\n\nfn handle_error\u003cE: std::fmt::Display\u003e(error: ResilienceError\u003cE\u003e) {\n    match error {\n        ResilienceError::Timeout { layer } =\u003e {\n            eprintln!(\"Timeout in {}\", layer);\n        }\n        ResilienceError::CircuitOpen { name } =\u003e {\n            eprintln!(\"Circuit breaker {:?} is open - fail fast\", name);\n        }\n        ResilienceError::BulkheadFull { concurrent_calls, max_concurrent } =\u003e {\n            eprintln!(\"Bulkhead full: {}/{} - try again later\", concurrent_calls, max_concurrent);\n        }\n        ResilienceError::RateLimited { retry_after } =\u003e {\n            if let Some(duration) = retry_after {\n                eprintln!(\"Rate limited, retry after {:?}\", duration);\n            }\n        }\n        ResilienceError::Application(app_err) =\u003e {\n            eprintln!(\"Application error: {}\", app_err);\n        }\n    }\n}\n```\n\n### Helper Methods\n\nQuickly check error categories:\n\n```rust\nif err.is_timeout() {\n    // Handle timeout from any layer (TimeLimiter or Bulkhead)\n}\n\nif err.is_circuit_open() {\n    // Circuit breaker is protecting the system\n}\n\nif err.is_rate_limited() {\n    // Backpressure - slow down\n}\n\nif err.is_application() {\n    // Get the underlying application error\n    if let Some(app_err) = err.application_error() {\n        // Handle app-specific error\n    }\n}\n```\n\n### When to Use\n\n**Use `ResilienceError\u003cE\u003e` when:**\n- Building new services with multiple resilience layers\n- You want zero boilerplate error handling\n- Standard error categorization is sufficient\n\n**Use manual `From` implementations when:**\n- You need very specific error semantics\n- Integrating with legacy error types\n- You need specialized error logging per layer\n\nSee the [`tower_resilience_core::error`](https://docs.rs/tower-resilience-core/latest/tower_resilience_core/error/) module for full documentation.\n\n## Pattern Composition\n\nStack multiple patterns for comprehensive resilience:\n\n```rust\nuse tower::ServiceBuilder;\n\n// Client-side: timeout -\u003e circuit breaker -\u003e retry\nlet client = ServiceBuilder::new()\n    .layer(timeout_layer)\n    .layer(circuit_breaker_layer)\n    .layer(retry_layer)\n    .service(http_client);\n\n// Server-side: rate limit -\u003e bulkhead -\u003e timeout\nlet server = ServiceBuilder::new()\n    .layer(rate_limiter_layer)\n    .layer(bulkhead_layer)\n    .layer(timeout_layer)\n    .service(handler);\n```\n\nFor comprehensive guidance on composing patterns effectively, see:\n\n- **[Composition Guide](https://docs.rs/tower-resilience/latest/tower_resilience/composition/)** - Pattern selection, recommended stacks, layer ordering, and anti-patterns\n- **[Composition Tests](tests/composition_stacks/)** - Working examples of all documented stacks that verify correct compilation\n\n## Benchmarks\n\nHappy path overhead (no failures triggered):\n\n| Pattern | Overhead |\n|---------|----------|\n| Retry (no retries) | ~80-100 ns |\n| Time Limiter | ~107 ns |\n| Rate Limiter | ~124 ns |\n| Bulkhead | ~162 ns |\n| Cache (hit) | ~250 ns |\n| Circuit Breaker (closed) | ~298 ns |\n\n```bash\ncargo bench --bench happy_path_overhead\n```\n\n## Examples\n\n```bash\ncargo run --example circuitbreaker\ncargo run --example bulkhead\ncargo run --example retry\n```\n\nSee [examples/](examples/) for more.\n\n## Stress Tests\n\n```bash\ncargo test --test stress -- --ignored\n```\n\n## MSRV\n\n1.64.0 (matches Tower)\n\n## License\n\nLicensed under either of:\n\n- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)\n- MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)\n\nat your option.\n\n## Contributing\n\nContributions are welcome! Please see the [contributing guidelines](CONTRIBUTING.md) for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoshrotenberg%2Ftower-resilience","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoshrotenberg%2Ftower-resilience","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoshrotenberg%2Ftower-resilience/lists"}