{"id":19297444,"url":"https://github.com/factorhouse/kafka-local","last_synced_at":"2025-10-19T15:02:27.535Z","repository":{"id":144558698,"uuid":"373007229","full_name":"factorhouse/kafka-local","owner":"factorhouse","description":"Run Local Kafka with Docker Compose","archived":false,"fork":false,"pushed_at":"2024-09-12T12:32:17.000Z","size":111,"stargazers_count":17,"open_issues_count":0,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-09-12T23:57:17.039Z","etag":null,"topics":["docker","docker-compose","kafka","kafka-sasl","kafka-ui"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/factorhouse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-02T01:41:38.000Z","updated_at":"2024-09-12T12:32:21.000Z","dependencies_parsed_at":"2024-02-13T12:40:48.205Z","dependency_job_id":"47c1a42e-3fb6-4eb9-b442-ab9c6f70a428","html_url":"https://github.com/factorhouse/kafka-local","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorhouse%2Fkafka-local","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorhouse%2Fkafka-local/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorhouse%2Fkafka-local/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorhouse%2Fkafka-local/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/factorhouse","download_url":"https://codeload.github.com/factorhouse/kafka-local/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223893164,"owners_count":17220834,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","kafka","kafka-sasl","kafka-ui"],"created_at":"2024-11-09T23:04:55.562Z","updated_at":"2025-10-19T15:02:22.516Z","avatar_url":"https://github.com/factorhouse.png","language":null,"funding_links":[],"categories":["Testing \u0026 Development"],"sub_categories":["Local Development"],"readme":"# Run local Kafka with Docker Compose\n\n### Versions\n\n| Confluent Container Version   | Kafka Equivalent               |\n|-------------------------------|--------------------------------|\n| `confluentinc/cp-kafka:7.7.0` | `org.apache.kafka/kafka:3.7.0` |\n\n### Guide\n\n* [Introduction](#introduction)\n* [Prequisites](#prerequisites)\n  * [Install Docker](#install-docker)\n  * [Clone this repository](#clone-this-repository)\n  * [Change into the repository directory](#change-into-the-repository-directory)\n* [Run Kpow Community Edition (optional)](#run-kpow-community-edition-optional)\n* [Run a simple Kafka cluster (no authentication)](#run-a-simple-kafka-cluster-no-authentication)\n  * [Start the Kafka cluster](#start-the-kafka-cluster)\n  * [Stop the Kafka cluster](#stop-the-kafka-cluster)\n  * [Access the Kafka cluster](#access-the-kafka-cluster)\n    * [Localhost bootstrap](#localhost-bootstrap)\n    * [Docker host bootstrap](#docker-host-bootstrap)\n    * [host.docker.internal bootstrap](#hostdockerinternal-bootstrap)\n* [Run a SASL Kafka cluster (with authentication)](#run-a-sasl-kafka-cluster-with-authentication)\n  * [Client authentication](#client-authentication)\n* [License](#license) \n\n## Introduction\n\nThis repository contains docker compose configuration to run a local Kafka cluster with Docker Compose.\n\nWe use similar configuration for local development of our Kafka UI and API product, [Kpow for Apache Kafka](https://factorhouse.io/kpow).\n\nTwo types of Kafka Cluster are supported, simple (no authentication) and SASL authenticated. \n\nSee [kpow-local](https://github.com/factorhouse/kpow-local) for a more complex local configuration consisting of Kpow, Kafka, Schema, Connect, and ksqlDB.\n\n## Prerequisites\n\n### Install Docker\n\nThe local cluster runs with Docker Compose, so you will need to [install Docker](https://www.docker.com/).\n\nOnce Docker is installed, clone this repository and run the following commands from the base path.\n\n### Clone this repository\n\n```\ngit clone git@github.com:factorhouse/kafka-local.git\n```\n\n### Change into the repository directory\n\n```\ncd kafka-local\n```\n\n## Run Kpow Community Edition (Optional)\n\nThe community edition of Kpow for Apache Kafka is free to use by individuals and organisations.\n\nYou can check it out if you like, but the configuration in this repository doesn't require Kpow in any way.\n\n![Kpow UI](/resources/img/kpow-overview.png)\n\nStart a local Kafka cluster with the configuration in this repository then:\n\n* Get a [free Kpow Community license](https://factorhouse.io/kpow/community/)\n* Enter the license details into [resoources/kpow/no-auth.env](resources/kpow/no-auth.env) or [resoources/kpow/sasl-auth.env](resources/kpow/sasl-auth.env)\n* Start Kpow Community Edition:\n\n**Start Kpow Community Edition with No Auth Kafka Cluster**\n\n```\ndocker run --network=kafka-local_default -p 3000:3000 -m2G --env-file ./resources/kpow/no-auth.env factorhouse/kpow-ce:latest\n```\n\n**Start Kpow Community Edition with SASL Auth Kafka Cluster**\n\n```\ndocker run --network=kafka-local_default -p 3000:3000 -m2G --env-file ./resources/kpow/sasl-auth.env factorhouse/kpow-ce:latest\n```\n\n* Navigate to http://localhost:3000 (the UI might look empty until you start creating topics and writing data)\n\n## Run a simple Kafka cluster (no authentication)\n\n### Start the Kafka cluster\n\nThis command starts a Kafka Cluster that does not require clients to authenticate.\n\n```bash\ndocker compose -f docker-compose-no-auth.yml up\n```\n\n```\n[+] Running 5/5\n ✔ Network kafka-local_default      Created0.0s\n ✔ Container zookeeper              Created0.0s\n ✔ Container kafka-local-kafka-3-1  Created0.0s\n ✔ Container kafka-local-kafka-1-1  Created0.0s\n ✔ Container kafka-local-kafka-2-1  Created0.0s\nAttaching to kafka-1-1, kafka-2-1, kafka-3-1, zookeeper\nzookeeper  | ===\u003e User\nzookeeper  | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)\nzookeeper  | ===\u003e Configuring ...\n```\n\n### Stop the Kafka cluster\n\nFirst, hit ctrl-c in the terminal running the Docker Compose process.\n\n```bash\n^C\nGracefully stopping... (press Ctrl+C again to force)\n[+] Stopping 4/4\n ✔ Container kafka-local-kafka-1-1  Stopped5.8s\n ✔ Container kafka-local-kafka-2-1  Stopped0.7s\n ✔ Container kafka-local-kafka-3-1  Stopped0.7s\n ✔ Container zookeeper              Stopped0.5s\ncanceled\n```\n\nThen stop/clear the Docker Compose resources\n\n```\ndocker compose -f docker-compose-no-auth.yml down\n```\n\n```\n[+] Running 5/0\n ✔ Container kafka-local-kafka-1-1  Removed                                                                                                                                                                    0.0s\n ✔ Container kafka-local-kafka-2-1  Removed                                                                                                                                                                    0.0s\n ✔ Container kafka-local-kafka-3-1  Removed                                                                                                                                                                    0.0s\n ✔ Container zookeeper              Removed                                                                                                                                                                    0.0s\n ✔ Network kafka-local_default      Removed\n```\n \n### Access the Kafka cluster\n\nTo access this cluster you can:\n\n1. Connect to the bootstrap on localhost / 127.0.0.1 (most likely non-docker applications)\n2. Connect to the bootstrap on the Docker defined hosts (kakfa-1, kafka-2, kafka-3)\n3. Connect to the bootstrap using `host.docker.internal` which is similar to (1)\n\n#### Localhost bootstrap\n\nApplications that are external to Docker can access the Kafka cluster via the Localhost bootstrap.\n\n```\nbootstrap: 127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094\n```\n\n#### Docker host bootstrap\n\nContainerized applications can connect to the Kafka cluster via the Docker Host bootstrap.\n\nThese docker hosts (kakfa-1, kafka-2, kafka-3) are defined within the comoose.yml.\n\nWhen starting your Docker container, specify that it should share the `kafka-local_default` network.\n \n```\ndocker run --network=kafka-local_default ...\n```\n\nThen connect to the hosts that are running on that network\n\n```\nbootstrap: kafka-1:19092,kafka-2:19093,kafka-3:19094 \n```\n\n#### host.docker.internal bootstrap\n\nThis is a good trick for running a docker container that connects back to a port open on the host machine.\n\n`host.docker.internal` effective routes back to localhost.\n\n```\nbootstrap: host.docker.internal:9092,host.docker.internal:9093,host.docker.internal:9094 \n```\n\n## Run a SASL Kafka cluster (with authentication)\n\nUse the `docker-compose-sasl-auth.yml` configuration to run a SASL authenticated cluster:\n\n```\ndocker compose -f docker-compose-sasl-auth.yml up\n```\n\nBootstrap configuration is the same as the simple cluster however clients must authenticate to connect. \n\nAuthentication configuration is specified in [resources/docker/kafka_jaas.conf](resources/docker/kafka_jaas.conf).\n\n### Client authentication\n\nTo connect a client to this client use the following connection settings:\n\n```\nsecurity.protocol: SASL_PLAINTEXT\nsasl.mechanism:    PLAIN\nsasl.jaas.config:  org.apache.kafka.common.security.plain.PlainLoginModule required username=\"client\" password=\"client-secret\";\n```\n\n## License\n\nThis repository is released under the Apache 2.0 License.\n\nCopyright © Factor House.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffactorhouse%2Fkafka-local","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffactorhouse%2Fkafka-local","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffactorhouse%2Fkafka-local/lists"}