{"id":19019351,"url":"https://github.com/ditsuke/kafka-pubsub","last_synced_at":"2026-06-21T15:31:56.186Z","repository":{"id":53254627,"uuid":"520404193","full_name":"ditsuke/kafka-pubsub","owner":"ditsuke","description":"Apache Kafka pub/sub clients and benchmarks","archived":false,"fork":false,"pushed_at":"2022-08-10T07:39:38.000Z","size":79,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-21T18:15:17.450Z","etag":null,"topics":["concurrency","kafka","pubsub"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ditsuke.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-02T07:55:07.000Z","updated_at":"2022-08-06T07:39:53.000Z","dependencies_parsed_at":"2022-08-27T06:22:08.777Z","dependency_job_id":null,"html_url":"https://github.com/ditsuke/kafka-pubsub","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ditsuke/kafka-pubsub","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ditsuke%2Fkafka-pubsub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ditsuke%2Fkafka-pubsub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ditsuke%2Fkafka-pubsub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ditsuke%2Fkafka-pubsub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ditsuke","download_url":"https://codeload.github.com/ditsuke/kafka-pubsub/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ditsuke%2Fkafka-pubsub/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34616509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-21T02:00:05.568Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","kafka","pubsub"],"created_at":"2024-11-08T20:12:03.735Z","updated_at":"2026-06-21T15:31:56.165Z","avatar_url":"https://github.com/ditsuke.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Publish-Subscribe\n\n## Introduction\nThis project is a simple Kafka publish-subscribe system in Go with an added set of requirements it tries to meet covered\nin later sections.\nThe publisher and subscriber are packaged into their own self-contained binaries, `publish` and `consume`, respectively.\nA Zookeeper-backed Kafka cluster is dockerized in with `docker-compose` for convenient and portable reproducibility \nof the test environment. To activate:\n```bash\nmake up\n```\n\nTo bring down the docker-compose project:\n```bash\nmake down\n```\n\n## Components\nThe publisher and subscriber are standalone binaries. To build:\n```bash\nmake build\n```\nThe binaries will be available in the `build` directory.\n\n### Publisher\nThe `publish` binary is a Kafka producer that writer messages to a Kafka topic. Parameters, including the number of\nand the batch size of messages to write (used to optimise throughput) can be configured with command line flags\npassed to the binary.\n\nThe publisher meets the following requirements:\n- Writes, by default, 10,00,000 messages (0-9,999,999) to a test topic divided into 3 partitions in the Kafka cluster.\n  Exactly-once semantics are achieved with the idempotency guarantee. This strategy, while not as strong as achievable\n  with Kafka's transactional API, is enough to ensure exactly-once processing in a single producer session considering\n  the following cases:\n  - Lack of acknowledgement from the Kafka broker -- the producer waits for broker acknowledgement (acks) before producing more.\n  - Recoverable errors stemming from network jitters -- the producer uses idempotent writes to prevent duplicates.\n  To further improve the exactly-once semantics, a transactional producer should be used instead.\n- Writes to the 3 partitions are balanced with a custom simple hashing strategy (since we're only writing serial numbers),\n  and messages within each partition are guaranteed to be written in order.\n\n#### Usage\n```\n$ ./build/publish -h\nUsage:\n  -batch-size int\n        number of events to write in a batch (default 100)\n  -events int\n        number of events to write (default 10000000)\n  -kafka-host string\n        kafka host (default \"localhost\")\n  -kafka-port int\n        kafka port (default 9092)\n  -no-logs\n        no logging to stdout\n  -partitions int\n        number of partitions (default 3)\n  -topic string\n        topic to write to (default \"test-numbers\")\n```\n\n### Consumer\nThe `consume` binary implements a Kafka consumer that uses a consumer group (configure with the `--group` flag) to consume\nmessages from a kafka topic. Each partition in the topic is processed concurrently by partition-specific consumers.\n\nThe consumer meets the following requirements:\n- Load numbers from all partitions in the Kafka topic.\n- Guarantee exactly-once semantics by committing records only once they're processed, and keeping track of consumed\n  offsets to prevent re-consumption on failure to commit.\n\n#### Usage\n```bash\n$ ./build/consume -h\nUsage:\n  -events-to-consume int\n        number of events to consume. \u003e=0 for unlimited. (default 10000000)\n  -group string\n        consumer group to use for consumption (default \"test-group\")\n  -kafka-host string\n        kafka host (default \"localhost\")\n  -kafka-port int\n        kafka port (default 9092)\n  -no-logs\n        no logging to stdout\n  -topic string\n        topic to consume (default \"test-numbers\")\n```\n\n## Benchmarking\nThe Makefile defines 2 benchmarks:\n- `make bench` is a specialised benchmark testing the publisher and subscriber in isolation, with customized throughput metrics\n  for both in `reads/sec` and `writes/sec`.\n- `make bench-full` is a complete benchmark for the project spec -- it tests the publisher and consumer in parallel, with\n   10,000,000 reads and writes in each iteration of the benchmark and outputs the statistics around how much time\n   the operation takes across multiple iterations.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fditsuke%2Fkafka-pubsub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fditsuke%2Fkafka-pubsub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fditsuke%2Fkafka-pubsub/lists"}