{"id":28604027,"url":"https://github.com/uber/concurrency-loadbalancer","last_synced_at":"2025-06-11T17:40:20.619Z","repository":{"id":52915137,"uuid":"317371339","full_name":"uber/concurrency-loadbalancer","owner":"uber","description":null,"archived":false,"fork":false,"pushed_at":"2024-03-13T19:05:53.000Z","size":90,"stargazers_count":12,"open_issues_count":1,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-05-09T07:59:51.851Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uber.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-11-30T23:25:02.000Z","updated_at":"2024-03-29T15:20:51.000Z","dependencies_parsed_at":"2024-03-12T06:34:44.396Z","dependency_job_id":null,"html_url":"https://github.com/uber/concurrency-loadbalancer","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/uber/concurrency-loadbalancer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fconcurrency-loadbalancer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fconcurrency-loadbalancer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fconcurrency-loadbalancer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fconcurrency-loadbalancer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uber","download_url":"https://codeload.github.com/uber/concurrency-loadbalancer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fconcurrency-loadbalancer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259308163,"owners_count":22837974,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-11T17:40:11.783Z","updated_at":"2025-06-11T17:40:20.612Z","avatar_url":"https://github.com/uber.png","language":"Java","funding_links":[],"categories":["容错组件"],"sub_categories":[],"readme":"# Disclaimer\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fuber%2Fconcurrency-loadbalancer.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fuber%2Fconcurrency-loadbalancer?ref=badge_shield)\n\nThis project is stable and being incubated for long-term support\n\n# Overview\nGenerally, load balancer is a piece of software or hardware that distributes workload among multiple partitions. \nTake microservice domain as an example, a load balancer distributes requests among a cluster of servers. \nThere are many load balancing algorithms, such as round-robin, random, hashing, least work, least latency etc.\nHere we are introducing a generic load balancer library that aims to improve both throughput and latency of application when system is under load,\nprevent cascading failure caused by service degradation. \n\n# Background\nConcurrency is the number of requests a system serves at a given time.\nFor a given system, concurrency is bounded by resources, such as memory, CPU,  thread, connection etc.\nAccording to the little’s law, for a system in steadyC state, concurrency is the product of the average service time and the average service rate (L=λ*W).\nSo, for given concurrency, request throughput is inversely proportional to request latency.\n\nSystems with requests under or over concurrency limits behave differently. \nA system with concurrency below concurrency limit usually has stable request latency, and its request throughput increases linearly with concurrency.\nOn the contrary, A system with concurrency above concurrency limit indicates there are queues forming, and request latency increases linearly with the number of requests.\n\nPractically, system degradation happens. Generally, there are two types of degradation, high request latency and high failure rate.\nWhen latency increases, throughput decreases. With downstream system throughput drops below upstream throughput, request queue build up and it could cause high request latency propagated to upstream service.\nSimilarly, when failure happens frequently in downstream service, those failures may propagate back to upstream service and cause upstream service degradation as well.\n\nAn efficient load balancing algorithm should be able to reduce cascading failure by preventing saturation of partial downstream services propagated to upstream service.\nNote that, if saturation happens on all downstream services, it should not be the responsibility of the load balancer to prevent cascading failure, instead It's more about load shedding than load balancing in that case. \n\n# Solution\nTo solve the problem we invented this loadbalancer \n- Inspired by the Little’s law, The load balancer keeps track of 3 factors (concurrency, latency, and throughput) of each partition.\n- An efficient load balancer should suppress queues from building up, and concurrency is the factor that can reflect the fact of queuing before a request complete.\nso load balancer first compares concurrency of each partition, and distributes requests to the least concurrency partition. \nWhen concurrency is the same across partitions,  we are in a situation where either no queue or queue can’t be prevented. There are following strategies to handle this case:\n    \n    - 3.1 Least request - Choose the partition with least completed requests. Similar to round-robin strategy, this sub-strategy values fairness more than efficiency\n    \n    - 3.2 Least request time - Choose the partition with least aggregated request latency. When latency is the same across partitions, this strategy effectively equals to the least request strategy. \n    However, when there is a degraded partition, request latency of the degraded partition increases, thus, its throughput drops. Conclusively, this sub-strategy values efficiency more than fairness.\n\n- To optimize request success rate, define effective latency for failures, and reflect failure in the term of latency. \nConcretely, when request failed, we keep the concurrency sustained by one for an extra amount of time, and the extra sustain period equals to \n                   \n                   effectiveLatency - observedLatency\n\n  It works as a passive health check mechanism to prevent failure propagating from degraded partition to upstream service.\n\n\n# Example\n\nThere are two concrete implementations of the algorithm described above.\nArrayConcurrencyLoadbalancer is implemented in Array, which has constant computational complexity through partition grouping.\nHeapConcurrencyLoadbalancer is implemented in hashed priority queue, which has Log(N) computational complexity.\n\nExample to create an ArrayConcurrencyLoadbalancer, and enables LeastTime sub-strategy\n```java\nArrayList\u003cString\u003e entries = new ArrayList\u003cString\u003e() {{add(\"url1\"); add(\"url2\"); add(\"url3\");}};\n\nArrayConcurrencyLoadBalancer\u003cString\u003e loadBalancer = ArrayConcurrencyLoadBalancer.newBuilder(String.class)\n     .withTasks(entries)\n     .withSubStrategy(SubStrategy.LeastTime)\n     .build();\n```\n\nExample to create an HeapConcurrencyLoadbalancer\nand when request failed, request latency will be treated effectively same as 30 seconds.\n```java\nArrayList\u003cString\u003e entries = new ArrayList\u003cString\u003e() {{add(\"url1\"); add(\"url2\");}};\n\nHeapConcurrencyLoadBalancer\u003cString\u003e loadBalancer = HeapConcurrencyLoadBalancer.newBuilder(String.class)\n     .withTasks(entries)\n     .withFailureEffectiveLatency(Duration.ofSeconds(30))\n     .build();\n```\n\n# Integration\nTODO\nGRPC example\n\n## License\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fuber%2Fconcurrency-loadbalancer.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fuber%2Fconcurrency-loadbalancer?ref=badge_large)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber%2Fconcurrency-loadbalancer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuber%2Fconcurrency-loadbalancer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber%2Fconcurrency-loadbalancer/lists"}