{"id":28603996,"url":"https://github.com/uber/uforwarder","last_synced_at":"2025-06-11T17:40:10.847Z","repository":{"id":222915646,"uuid":"448046539","full_name":"uber/uForwarder","owner":"uber","description":"Apache Kafka is an open-source distributed event streaming platform used by thousands of companies. uForwarder aims to address several pain points while using Apache Kafka for pub-sub message queueing at scale, including partition scalability and head-of-line blocking.","archived":false,"fork":false,"pushed_at":"2025-03-17T03:24:01.000Z","size":1597,"stargazers_count":38,"open_issues_count":4,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-17T04:27:55.429Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uber.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-14T17:09:35.000Z","updated_at":"2025-03-17T03:24:04.000Z","dependencies_parsed_at":"2024-05-20T18:12:00.015Z","dependency_job_id":null,"html_url":"https://github.com/uber/uForwarder","commit_stats":null,"previous_names":["uber/uforwarder"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/uber/uForwarder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2FuForwarder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2FuForwarder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2FuForwarder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2FuForwarder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uber","download_url":"https://codeload.github.com/uber/uForwarder/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2FuForwarder/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259308162,"owners_count":22837974,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-11T17:39:56.574Z","updated_at":"2025-06-11T17:40:10.843Z","avatar_url":"https://github.com/uber.png","language":"Java","readme":"# UForwarder\n\n## Overview\n\nUforwarder is a proxy that transfers data from kafka to message consumer through RPC protocol.\nUforwarder aims to achieve reliable message delivery in realtime at scale through simple RPC protocol between Kafka and consumers.\n\nProperties and Traits:\n\n- Out of order Message delivery\n- At least once message delivery\n- Parallel message processing of each kafka topic partition\n- Independent scaling of kafka topic partition and consumer instances\n- Retry queue and dead letter queue for error handling\n- Flow control / throttling\n- Head of line blocking detection and auto recovery (TODO: add link to more details)\n\nRPC protocol supported\n\n- GRPC\n\n## Overfiew\n\nFor an overview of the project, please read the Uber Engineering blog [Enabling Seamless Kafka Async Queuing with Consumer Proxy](https://www.uber.com/blog/kafka-async-queuing-with-consumer-proxy/)\n\n## Table of Contents\n\n1. [Build](#build)\n2. [Contributing](#contributing)\n3. [License](#license)\n4. [Acknowledgments](#acknowledgments)\n5. [Contact](#contact)\n\n## Build\n\n### Prerequisite\n\n- Java 11 and above\n- Docker 24.0 and above\n\n### Steps\n\n1. Build binary\n\n  ```\n  ./gradlew jar\n  ```\n\n2. Run unit test\n\n  ```\n  ./gradlew test\n  ```\n\n3. Run integration test\n\n  ```\n  ./gradlew integrationTest\n  ```\n\n4. Run code coverage verification\n\n  ```\n  ./gradlew check\n  ```\n\n4. Build uforwarer Docker image\n\n  ```\n  ./gradlew jar\n  cd uforwarder-image/build/docker\n  docker build -t uforwarder:latest .\n  ```\n\n## Usage\n\n### Prerequisite\n\n- Kafka 2.8 and above\n- Zookeeper 3.4 and above\n\n### Steps\n\n1. Create a new docker network\n\n```\ndocker network create docker-network\n```\n\n2. Start Zookeeper and Kafka broker\n\n```\ndocker run --env ALLOW_ANONYMOUS_LOGIN=yes --network docker-network -p 2181:2181 --name zookeeper zookeeper:3.8.0\ndocker run --env KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 --env KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 --env KAFKA_LISTENERS=DOCKER://0.0.0.0:9092,HOSTER://0.0.0.0:9093 --env KAFKA_ADVERTISED_LISTENERS=DOCKER://kafka:9092,HOSTER://localhost:9093 -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=DOCKER:PLAINTEXT,HOSTER:PLAINTEXT -e KAFKA_INTER_BROKER_LISTENER_NAME=DOCKER --network docker-network -p 9093:9093 --name kafka confluentinc/cp-kafka:5.2.1\n```\n\n3. Run Uforwarder controller and worker\n\n```\ndocker run --env UFORWARDER_PROFILE=uforwarder-controller --env UFORWARDER_KAFKA_CONNECT=kafka:9092 --env UFORWARDER_ZOOKEEPER_CONNECT=zookeeper:2181/uforwarder --network docker-network --name controller -p 8087:8087 uforwarder:latest\ndocker run --env UFORWARDER_PROFILE=uforwarder-worker --env UFORWARDER_KAFKA_CONNECT=kafka:9092 --env UFORWARDER_CONTROLLER_CONNECT=controller:8087 --network docker-network --name worker uforwarder:latest\n```\n\n4. Run Sample Consumer\n\n```\n./gradlew uforwarder-sample-consumer:bootRun\n```\n\n## Contributing\n\nWe accept contributions through Pull Requests (PR's).  Please sign our [Contributing License Agreement (CLA)](cla-assistant.io) after you submit the PR, so that it can be accepted.\n\n## License\n\nApache License Version 2.0\n\n## Contact\n\nuforwarder-committers@googlegroups.com\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber%2Fuforwarder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuber%2Fuforwarder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber%2Fuforwarder/lists"}