{"id":15998596,"url":"https://github.com/d7561985/kafka-ab","last_synced_at":"2025-07-26T22:34:09.456Z","repository":{"id":57596034,"uuid":"334643439","full_name":"d7561985/kafka-ab","owner":"d7561985","description":"Tool for loading kafka via multiple consumers/producers","archived":false,"fork":false,"pushed_at":"2021-08-20T12:57:11.000Z","size":44,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-23T09:15:02.523Z","etag":null,"topics":["kafka","kafka-ab","kafka-cli","kafka-testing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/d7561985.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-01-31T11:53:40.000Z","updated_at":"2021-08-20T12:57:13.000Z","dependencies_parsed_at":"2022-09-26T20:01:29.667Z","dependency_job_id":null,"html_url":"https://github.com/d7561985/kafka-ab","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/d7561985/kafka-ab","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d7561985%2Fkafka-ab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d7561985%2Fkafka-ab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d7561985%2Fkafka-ab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d7561985%2Fkafka-ab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/d7561985","download_url":"https://codeload.github.com/d7561985/kafka-ab/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d7561985%2Fkafka-ab/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267253072,"owners_count":24060135,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-26T02:00:08.937Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-ab","kafka-cli","kafka-testing"],"created_at":"2024-10-08T08:20:27.937Z","updated_at":"2025-07-26T22:34:09.439Z","avatar_url":"https://github.com/d7561985.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Docker Image CI](https://github.com/d7561985/kafka-ab/actions/workflows/docker-image.yml/badge.svg)](https://github.com/d7561985/kafka-ab/actions/workflows/docker-image.yml)\n# Kafka-AB\n`Producer` helps produce a big number of system messages simultaneously to all partitions. Every message waits response with successful emitting to broker.\n\n`Consumer` - read provided topic with unit group per thread with auto-commit option.\n\n## usage\napplication follows the best CLI practises, so it contains help information about environment, supported, etc.\n\n## install\n```bash\n  go install github.com/d7561985/kafka-ab@latest\n```\n### Consumer\n```bash\nOPTIONS:\n   --topic value                 (default: \"my-topic\") [$TOPIC]\n   --kafka-server value          (default: \"PLAINTEXT://127.0.0.1:9094\") [$KAFKA_SERVER]\n   --concurrency value           (default: 10) [$CONCURRENCY]\n   --timelimit value, -t value   Stop all tasks instantly. In case of desired request not reach will exist with status 1  (default: Seconds to max. to spend on benchmarking.) [$TIME_LIMIT]\n   --requests value, -n value    require duration postfix: s - seconds, h - hours and etc (default: Number of requests to perform/consume) [$REQUESTS]\n   --verbosity value, -v value   (default: How much troubleshooting info to print) [$VERBOSITY]\n   --force-name value, -f value  (default: all consumer's groups have the same group name which pass throughout) [$FORCE_NAME]\n   --static, -s                  (default: all consumer's groups hase unique but static name (group-1, group-2 and group-...) [$STATIC]\n   --auto, -a                    (default: auto-commit option) [$AUTO_COMMIT]\n   --earliest, -e                (default: read from the very beginning of log) [$EARLIEST]\n   --help, -h                    show help (default: false)\n```\n\n### Producer\n```bash\nOPTIONS:\n   --topic value                      (default: \"my-topic\") [$TOPIC]\n   --kafka-server value, --srv value  (default: \"PLAINTEXT://127.0.0.1:9094\") [$KAFKA_SERVER]\n   --windowsize value, -b value       Size of message send/receive buffer, in bytes (default: 1024) [$WINDOW_SIZE]\n   --concurrency value, -c value      (default: Number of multiple requests to make/read at a time) [$CONCURRENCY]\n   --requests value, -n value         (default: Number of requests to perform/consume) [$REQUESTS]\n   --timelimit value, -t value        Stop all tasks instantly. In case of desired request not reach will exist with status 1  (default: Seconds to max. to spend on benchmarking.) [$TIME_LIMIT]\n   --timeout value, -s value          require duration postfix: s - seconds, h - hours and etc (default: Seconds to max. wait for each response) [$TIME_OUT]\n   --verbosity value, -v value        (default: How much troubleshooting info to print) [$VERBOSITY]\n   --help, -h                         show help (default: false)\n```\n### go run / build / install\n```bash\n$ go run main.go c --topic my-topic --kafka-server 127.0.0.1:9094 --threads 10\n```\n\n### docker image \n```bash\n$ docker run --rm -it d7561985/kafka-ab:v1.0.0 -kafka-server PLAINTEXT://host.docker.internal:9094 consumer\n\n$ docker run --rm -it -e KAFKA_SERVER=PLAINTEXT://host.docker.internal:9094 d7561985/kafka-ab:v1.0.0  producer\n```\n\n#### K8s\n```bash\n$ kubectl run producer --image d7561985/kafka-ab:v1.0.2 --env KAFKA_SERVER=my-cluster-kafka-external-bootstrap:9094 --restart=Never -it --rm -- p -h\n\n$ kubectl run producer --image d7561985/kafka-ab:v1.0.2 --env KAFKA_SERVER=my-cluster-kafka-external-bootstrap:9094 --restart=Never -it --rm -- c -h\n```\n\n### from what to start\nassume that we have topic with 50 partitions and some replicas\n```yaml\napiVersion: kafka.strimzi.io/v1beta1\nkind: KafkaTopic\nmetadata:\n  name: my-topic\n  labels:\n    strimzi.io/cluster: my-cluster\nspec:\n  partitions: 50\n  replicas: 2\n  config:\n    retention.ms: 3600000\n    segment.bytes: 1073741824\n```\nfor test topic try to use small retention time as log is filled very significantly\n\n\n### Use cases \nApplication exits with status 1 when it can't meet some target. \n\nFor example: consumer don't read any message or consumer not reads desired requests number which helps to perform some CI tests ;) \n#### Consumer\n* `--timelimit` if any of consumer not read any message at the and of limit will exit with status 1\n* `--requests` exit with status 0 when request meat desired number, but if timelimit comes - status will be 1  \n* Concurrent (`--concurrency`) consumer with the force name (`--force-name`) - one group reads from different  partitions\n* `--static` every run consumers will have the same name. This helps to check auto-commit option for example\n* ` --earliest` option allow to deside do you need read from the start of just new income. Set false to read new income only. \n\n\n```bash\nkubectl port-forward service/grafana -n global 3000:3000\nkubectl -n global port-forward svc/kafka-cluster-strimzi-helm-kafka-external-bootstrap 9094:9094\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd7561985%2Fkafka-ab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fd7561985%2Fkafka-ab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd7561985%2Fkafka-ab/lists"}