{"id":34658462,"url":"https://github.com/kafkesc/kafka-dev-cluster","last_synced_at":"2025-12-24T18:37:18.734Z","repository":{"id":160083503,"uuid":"545617867","full_name":"kafkesc/kafka-dev-cluster","owner":"kafkesc","description":"A battery-included BUT development-only pairing of Makefile \u0026 docker-compose.yml, designed to quickly spin-up a Kafka cluster for development purposes.","archived":false,"fork":false,"pushed_at":"2024-04-05T14:28:48.000Z","size":38,"stargazers_count":4,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-05T15:39:20.718Z","etag":null,"topics":["cluster","developer-tools","development","docker-compose","kafka","makefile","zookeeper"],"latest_commit_sha":null,"homepage":"","language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kafkesc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-10-04T17:31:37.000Z","updated_at":"2023-05-11T15:21:11.000Z","dependencies_parsed_at":"2024-01-07T22:38:21.980Z","dependency_job_id":null,"html_url":"https://github.com/kafkesc/kafka-dev-cluster","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kafkesc/kafka-dev-cluster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kafkesc%2Fkafka-dev-cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kafkesc%2Fkafka-dev-cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kafkesc%2Fkafka-dev-cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kafkesc%2Fkafka-dev-cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kafkesc","download_url":"https://codeload.github.com/kafkesc/kafka-dev-cluster/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kafkesc%2Fkafka-dev-cluster/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28006369,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-24T02:00:07.193Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cluster","developer-tools","development","docker-compose","kafka","makefile","zookeeper"],"created_at":"2025-12-24T18:37:16.356Z","updated_at":"2025-12-24T18:37:18.717Z","avatar_url":"https://github.com/kafkesc.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafkesc's `kafka-dev-cluster`: Kafka Development Cluster\n\nA **_battery-included_** BUT **_development-only_** pairing of `Makefile` \u0026 `docker-compose.yml`,\ndesigned to quickly spin-up a [Kafka](https://kafka.apache.org/) cluster for development purposes.\n\nThe `docker-compose-infra.yml` launches:\n\n* a Kafka cluster made of 3 Brokers\n* a ZooKeeper ensemble, made of a single server\n\nThis should be plenty for local development of services and tools, designed\nto interact and operate against Kafka. Even when you are [offline](#using-offline).\n\nFor the `docker-compose-work.yml`, refer to the [Workload Generation](#workload-generation) section below.\n\n## Dependencies\n\n* [`kcat`](https://github.com/edenhill/kcat)\n* [`docker` + `docker-compose`](https://docs.docker.com/get-docker/) \n* [`jq`](https://stedolan.github.io/jq/) \n* `md5sum`, `head` ([GNU Core Utilities](https://en.wikipedia.org/wiki/List_of_GNU_Core_Utilities_commands))\n\n## Getting started\n\nThe project provides a `Makefile` with super-simple functionalities.\nIt provides single _mnemonic devices_ to manage the cluster lifecycle and \ndo basic operations against Kafka topics.\n\nA good `Makefile` autocompletion will make things even easier.\n\n|            Command | Arguments                                                                                                | Description                                                                                  |\n|-------------------:|----------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|\n|             `init` |                                                                                                          | Prepares localhost to launch the cluster                                                     |\n|            `start` | `timeout=SEC`                                                                                            | Launches the cluster                                                                         |\n|             `stop` | `timeout=SEC`                                                                                            | Shuts down the cluster: remove every resource for the project                                |\n|          `restart` | `timeout=SEC`                                                                                            | Restarts the cluster                                                                         |\n|             `kill` | `timeout=SEC`                                                                                            | Forcefully shuts down the cluster (i.e. `SIGKILL`)                                           |\n|             `logs` | `?service=(zookeeper,kafka-0[1-3])`                                                                      | Tail-follow logs of the running services (default: all services).                            |\n|               `ps` |                                                                                                          | Docker status of the running services                                                        |\n|             `pull` |                                                                                                          | Pull all the Docker images necessary to run the cluster                                      |\n|          `consume` | `topic=TOPIC`,\u003cbr/\u003e `?offset=(beginning, end, stored, OFFSET, -OFFSET, s@TS, e@TS)`,\u003cbr/\u003e `?group=GROUP` | Consume from a topic, from a given offset and using a given `group.id`; use `CTRL+C` to stop |\n|          `produce` | `topic=TOPIC`,\u003cbr/\u003e `?key=KEY`,\u003cbr/\u003e `?value=VALUE`                                                      | Produce to a topic, using a given key/value pair                                             |\n|     `topic.create` | `topic=TOPIC`,\u003cbr/\u003e `?partitions=PC`,\u003cbr/\u003e `?repfac=RF`                                                  | Create a new topic                                                                           |\n|       `topic.read` | `topic=TOPIC`                                                                                            | Describe a topic                                                                             |\n|     `topic.delete` | `topic=TOPIC`                                                                                            | Delete a topic                                                                               |\n|     `meta.brokers` |                                                                                                          | Lists cluster brokers                                                                        |\n|      `meta.topics` |                                                                                                          | Lists clusters topics                                                                        |\n|      `meta.groups` |                                                                                                          | Lists clusters consumer groups                                                               |\n|   `workload.setup` |                                                                                                          | Creates necessary topics for the producers/consumers to use                                  |\n|   `workload.start` |                                                                                                          | Start `docker-compose-work.yml`: launch producers ([ksunami]) and consumers ([kcat])         |\n|    `workload.stop` |                                                                                                          | Stop `docker-compose-work.yml`: remove every resource for the project                        |\n| `workload.restart` |                                                                                                          | Restart `docker-compose-work.yml`                                                            |\n|    `workload.kill` |                                                                                                          | Forcefully shuts down `docker-compose-work.yml` (i.e. `SIGKILL`)                             | \n|      `workload.ps` |                                                                                                          | Docker status of the running services                                                        |\n|    `workload.logs` | `?service=(prod-0[1-3],cons-0[1-3][abc]?)`                                                               | Tail-follow logs of the running services (default: all services).                            |\n|    `workload.pull` |                                                                                                          | Pull all the Docker images necessary to run `docker-compose-work.yml`                        |\n\n**NOTE:**\n\n* `?ARG`: the argument is optional\n* `SEC`: amount of seconds\n* `group` defaults to `kafkesc-devcluster-group-id`\n* `offset` defaults to `end` - see [`kcat`](https://github.com/edenhill/kcat) for more details\n  * `s@TS`: timestamp in milliseconds to start at\n  * `e@TS`: timestamp in milliseconds to end at (not included)\n  * `OFFSET`: integer of the _absolute_ offset of a record\n  * `-OFFSET`: integer of the _relative_ offset of a record from the end\n* `key` defaults to a random alphanumeric string of 12 characters (ex. `K-a21d38311c`)\n* `value` defaults to a random alphanumeric string of 22 characters (ex. `V-6bbeba0cf4d0d5c2de36`)\n* `partitions` defaults to `3`\n  * `PC`: partitions count for the `topic`\n* `repfac` defaults to `3`\n  * `RF`: replication factor for the `topic`\n\n## Connecting\n\nOnce `start`ed, you can connect to the services using:\n\n|                         | Configuration strings                             |\n|-------------------------|---------------------------------------------------|\n| Kafka bootstrap brokers | `localhost:19091,localhost:19092,localhost:19093` |\n| ZooKeeper server        | `localhost:2181`                                  |\n\n## Supported environment variables\n\n`docker-compose` accepts environment variables that will be applied in the configuration, following the \nrules documented [here](https://docs.docker.com/compose/environment-variables/).\n\nThe default values are defined in [`.env`](./.env).\n\n## Network structure\n\nThe `docker-compose-infra.yml` is in charge of creating a single, default `bridge` network: `kafkesc-devcluster-network`.\n\nThe `docker-compose-work.yml` depends on it: it sets as its own `default` network, the one created by `docker-compose-infra.yml`,\nso that producers and consumers can connect to the Kafka brokers.\n\nThe containers launched by the `-infra` docker compose are reachable from your `localhost`. The ports are mapped like this:\n\n|     Service | `localhost` client port | `kafka-devcluster_default` client port |\n|------------:|:-----------------------:|:--------------------------------------:|\n| `zookeeper` |         `2181`          |                 `2181`                 |\n|  `kafka-01` |         `19091`         |                 `9091`                 |\n|  `kafka-02` |         `19092`         |                 `9092`                 |\n|  `kafka-03` |         `19093`         |                 `9093`                 |\n\nThe Kafka brokers will communicate with each other using the `kafka-0[1-3]:909[1-3]`.\nInstead, to communicate with ZooKeeper, brokers will use `zookeeper:2181`.\n\n### Running commands _inside_ a container\n\nThis is probably obvious from section above, but it's good to explicitly call out:\nwhen `docker exec`-uting from inside one of the docker containers, it's possible to\naddress the sibling containers using the _Service_ name as target hostname.\n\n## Workload Generation\n\nIn addition to the \"infrastructure\", this project can also spawn a _workload_.\nMaybe it's not realistic to use this for benchmarking, but having a producers/consumers in place, that are moving\nrecords on the cluster, can be a time saver during development.\n\nThis is realised by the `docker-compose-work.yml` setup. But of course, everything is controlled via `Makefile`, so\nyou should care about these details only if you want to make changes (or if something is broken).\n\n### Producers included in the Workload\n\nInstances of [ksunami] are set up for a specific producer behaviour:\n\n|  Produced Topic Name | Container name | Ksunami Behaviour                                                              |\n|---------------------:|---------------:|--------------------------------------------------------------------------------|\n|         `workload01` |      `prod-01` | spikes once a day; 100x traffic at spike                                       |\n|         `workload02` |      `prod-02` | no traffic for most; massive spike with 10000x traffic for 30s every 5 minutes |\n|         `workload03` |      `prod-03` | pretty stable; `min` and `max` phase almost identical                          |\n\n### Consumers included in the Workload\n\nInstances of [kcat] are set up for a specific grouping:\n\n| Consumed Topic | Consumer Group | Container name |\n|---------------:|---------------:|:---------------|\n|   `workload01` |      `cons-01` | `cons-01a`     |\n|   `workload01` |      `cons-01` | `cons-01b`     |\n|   `workload02` |      `cons-02` | `cons-02`      |\n|   `workload03` |      `cons-03` | `cons-03a`     |\n|   `workload03` |      `cons-03` | `cons-03b`     |\n|   `workload03` |      `cons-03` | `cons-03c`     |\n\nPlease see [docker-compose-workload.yml](./docker-compose-work.yml) for details.\n\n## Storage and Persistence\n\n**TODO**\n\nAt current stage, at shutdown all data is lost.\n\nMaybe this is a chance for _Your_ contribution? :wink: :innocent:\n\n## Using offline\n\nIt's possible that you might need to use `kafka-dev-cluster` while offline, or when you simply don't\nhave enough bandwidth to pull hundreds of megabytes of docker images.\n\nFor those cases, just use the `pull` commands provided to pre-fetch all the docker images, before\nyou get going.\n\n## License\n\n[Apache License 2.0](./LICENSE)\n\n[ksunami]: https://crates.io/crates/ksunami\n[kcat]: https://github.com/edenhill/kcat","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkafkesc%2Fkafka-dev-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkafkesc%2Fkafka-dev-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkafkesc%2Fkafka-dev-cluster/lists"}