{"id":19297456,"url":"https://github.com/factorhouse/kpow-local","last_synced_at":"2025-04-22T09:31:13.910Z","repository":{"id":52620566,"uuid":"218170098","full_name":"factorhouse/kpow-local","owner":"factorhouse","description":"Run Local Kafka and Kpow with Docker Compose","archived":false,"fork":false,"pushed_at":"2024-03-26T02:56:24.000Z","size":1681,"stargazers_count":13,"open_issues_count":0,"forks_count":5,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-03-27T00:40:17.647Z","etag":null,"topics":["database","developer-tools","docker","docker-compose","kafka","kafka-cluster","kpow","management","monitoring","streams","ui"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/factorhouse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-10-29T00:21:51.000Z","updated_at":"2024-03-27T00:40:17.648Z","dependencies_parsed_at":"2024-03-26T00:46:57.776Z","dependency_job_id":null,"html_url":"https://github.com/factorhouse/kpow-local","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorhouse%2Fkpow-local","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorhouse%2Fkpow-local/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorhouse%2Fkpow-local/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/factorhouse%2Fkpow-local/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/factorhouse","download_url":"https://codeload.github.com/factorhouse/kpow-local/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223893164,"owners_count":17220834,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","developer-tools","docker","docker-compose","kafka","kafka-cluster","kpow","management","monitoring","streams","ui"],"created_at":"2024-11-09T23:04:59.412Z","updated_at":"2025-04-22T09:31:13.904Z","avatar_url":"https://github.com/factorhouse.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Run Local Kafka and Kpow with Docker Compose\n\n[![Release to DockerHub](https://github.com/factorhouse/kpow/actions/workflows/release.yml/badge.svg?branch=main)](https://github.com/factorhouse/kpow/actions/workflows/release.yml)\n![Docker Pulls](https://img.shields.io/docker/pulls/operatr/kpow)\n\n### Versions\n\n| Kpow Version                 | Confluent Container Version   | Kafka Equivalent               |\n|------------------------------|-------------------------------|--------------------------------|\n| `factorhouse/kpow-ce:latest` | `confluentinc/cp-kafka:7.8.0` | `org.apache.kafka/kafka:3.8.0` |\n\n### Guide\n\n* [Introduction](#introduction)\n* [Prequisites](#prerequisites)\n  * [Install Docker](#install-docker)\n  * [Clone this repository](#clone-this-repository)\n  * [Change into the repository directory](#change-into-the-repository-directory)\n* [Run Kafka resources with Kpow Community](#run-kafka-resources-with-kpow-community)\n  * [Configure a Kpow Community License](#configure-a-kpow-community-license)\n  * [Start resouces](#start-resources)\n  * [Stop resources](#stop-resources)\n  * [Access resources](#access-resources)\n    * [Kpow Community Edition](#kpow-community-edition)\n    * [Kafka Cluster](#kafka-cluster)\n    * [Kafka Connect](#kafka-connect)\n    * [Schema Registry](#schema-registry)\n  * [Add Kafka Connect Connectors](#add-kafka-connect-connectors)\n    * [Custom connectors](#custom-connectors)\n* [Run Kafka resources with Kpow Enterprise](#run-kafka-resources-with-kpow-enterprise)\n  * [Configure a Kpow Enterprise Trial License](#configure-a-kpow-enterprise-trial-license)\n  * [Use the Kpow Enterprise Trial Docker Compose](#use-the-kpow-enterprise-trial-docker-compose)\n* [Support](#support)\n* [License](#license)\n\n## Introduction\n\nThis repository contains a Docker Compose environment that will start:\n\n- A 3-node Kafka cluster\n- Kafka Connect \n- Schema Registry \n- Kpow for Apache Kafka, either:\n  - Kpow Community Edition (free for individuals or organisations), or; \n  - Kpow Enterprise Trial (supports authentication, RBAC, etc)\n\nAll container images used support `linux/amd64` and `linux/arm64` platforms.\n\nSee [kafka-local](https://github.com/factorhouse/kafka-local) for a simple local configuration consisting of only Kafka.\n\n## Prerequisites\n\n### Install Docker\n\nThe local cluster runs with Docker Compose, so you will need to [install Docker](https://www.docker.com/).\n\nOnce Docker is installed, clone this repository and run the following commands from the base path.\n\n### Clone this repository\n\n```\ngit clone git@github.com:factorhouse/kpow-local.git\n```\n\n### Change into the repository directory\n\n```\ncd kpow-local\n```\n\n## Run Kafka resources with Kpow Community\n\nFollow these instructions to run Kafka, Connect, and Schema resources with Kpow Community Edition.\n\n### Configure a Kpow Community License\n\n* Get a [free Kpow Community license](https://factorhouse.io/kpow/community/)\n* Enter the license details into [resources/kpow/local-community.env](resources/kpow/local-community.env)\n\n```\nBOOTSTRAP=kafka-1:19092,kafka-2:19093,kafka-3:19094\nCONNECT_REST_URL=http://connect:8083\nSCHEMA_REGISTRY_URL=http://schema:8081\nSCHEMA_REGISTRY_AUTH=USER_INFO\nSCHEMA_REGISTRY_USER=admin\nSCHEMA_REGISTRY_PASSWORD=admin\n\n### Your License Details\nLICENSE_ID=\u003clicense-id\u003e\nLICENSE_CODE=\u003clicense-code\u003e\nLICENSEE=\u003clicensee\u003e\nLICENSE_EXPIRY=\u003clicense-expiry\u003e\nLICENSE_SIGNATURE=\u003clicense-signature\u003e\n```\n\n### Start resources\n\n```\ndocker compose -f docker-compose-community.yml up\n```\n\n```\n[+] Running 13/13\n ✔ kpow Pulled                                                                                                                                                                                                                                            2.7s\n ✔ schema 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled\n ...\n ...\n```\n\n### Stop resources\n\nFirst, hit ctrl-c in the terminal running the Docker Compose process.\n\n```bash\n^C\nGracefully stopping... (press Ctrl+C again to force)\n[+] Stopping 1/3\n ✔ Container kpow-local-kpow-1    Stopped                                                                                                                                                                                                                 0.0s\n[+] Stopping 2/3w-local-schema-1  Stopping                                                                                                                                                                                                                0.4s\n ✔ Container kpow-local-kpow-1    Stopped                                                                                                                                                                                                                 0.0s\n[+] Stopping 3/3w-local-schema-1  Stopping                                                                                                                                                                                                                0.5s\n ✔ Container kpow-local-kpow-1     Stopped                                                                                                                                                                                                                0.0s\n ✔ Container kpow-local-schema-1   Stopped\n ...\n```\n\nThen stop/clear the Docker Compose resources\n\n```\ndocker compose -f docker-compose-community.yml down\n```\n\n```\n[+] Running 8/7\n ✔ Container kpow-local-kpow-1       Removed                                                                                                                                                                                                              0.0s\n ✔ Container kpow-local-schema-1     Removed                                                                                                                                                                                                              0.0s\n ✔ Container connect                 Removed                                                                                                                                                                                                              0.0s\n ✔ Container kpow-local-kafka-2-1    Removed                                                                                                                                                                                                              0.0s\n ✔ Container kpow-local-kafka-3-1    Removed                                                                                                                                                                                                              0.0s\n ✔ Container kpow-local-kafka-1-1    Removed                                                                                                                                                                                                              0.0s\n ✔ Container kpow-local-zookeeper-1  Removed                                                                                                                                                                                                              0.5s\n ✔ Network kpow-local_default        Removed\n```\n\n## Access resources\n\n### Kpow Community Edition\n\nThe community edition of Kpow for Apache Kafka is free to use by individuals and organisations.\n\nKpow's UI will be available at http://localhost:3000 once all Kafka resources are running.\n\n![Kpow UI](/resources/img/kpow-overview.png)\n\n### Kafka Cluster\n\nTo access the cluster you can:\n\n1. Connect to the bootstrap on localhost / 127.0.0.1 (most likely non-docker applications)\n2. Connect to the bootstrap on the Docker defined hosts (kakfa-1, kafka-2, kafka-3)\n3. Connect to the bootstrap using `host.docker.internal` which is similar to (1)\n\n#### Localhost bootstrap\n\nApplications that are external to Docker can access the Kafka cluster via the Localhost bootstrap.\n\n```\nbootstrap: 127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094\n```\n\n#### Docker host bootstrap\n\nContainerized applications can connect to the Kafka cluster via the Docker Host bootstrap.\n\nThese docker hosts (kakfa-1, kafka-2, kafka-3) are defined within the comoose.yml.\n\nWhen starting your Docker container, specify that it should share the `kpow-local_default` network.\n\n```\ndocker run --network=kpow-local_default ...\n```\n\nThen connect to the hosts that are running on that network\n\n```\nbootstrap: kafka-1:19092,kafka-2:19093,kafka-3:19094 \n```\n\n#### host.docker.internal bootstrap\n\nThis is a good trick for running a docker container that connects back to a port open on the host machine.\n\n`host.docker.internal` effective routes back to localhost.\n\n```\nbootstrap: host.docker.internal:9092,host.docker.internal:9093,host.docker.internal:9094 \n```\n\n### Kafka Connect\n\nSimilarly to Kafka cluster configuration you can access the unauthenticated Kafka Connect cluster via:\n\n* `http://localhost:8083` (plain localhost for non-docker applications)\n* `http://connect:8083` (when setting the docker network and connecting to the docker host)\n* `http://host.docker.internal:8083` (when connecting from docker back to the host localhost)\n\n### Schema Registry\n\n* `http://localhost:8081` (plain localhost for non-docker applications)\n* `http://schema:8081` (when setting the docker network and connecting to the docker host)\n* `http://host.docker.internal:8081` (when connecting from docker back to the host localhost)\n\nEach schema client requires the basic authentication credentials of `admin/admin`.\n\n## Add Kafka Connect Connectors\n\n### Custom Connectors\n\nTo add custom Kafka Connect connectors create a `resources/connect` directory and add all JARs there.\n\nFor example, to add the [Debezium PostgresSQL connectors](https://debezium.io/documentation/reference/stable/connectors/postgresql.html):\n\n```\nmkdir -p ./resources/connect\n```\n\n```\ncurl -L -o ./resources/connect/debezium-connector-postgres-1.9.6.Final-plugin.tar.gz https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.9.6.Final/debezium-connector-postgres-1.9.6.Final-plugin.tar.gz\n```\n\n```\ncd resources/connect\n```\n\n``` \ntar –xvzf debezium-connector-postgres-1.9.6.Final-plugin.tar.gz\n```\n\nOnce installed, the connector will be visible within Kpow and available to create.\n\n## Run Kafka resources with Kpow Enterprise\n\n![Kpow Login](/resources/img/kpow-login.png)\n\nFollow these instructions to run Kafka, Connect, and Schema resources with Kpow Enterprise Edition.\n\nA trial of Kpow allows you access to all features, including Authentication, Authorisation, RBAC, Audit Log, Prometheus egress, and more.\n\nFor example purposes we configure a simple [file-based authentication](/resources/jaas/hash-realm.properties) with an example [RBAC configuration](/resources/rbac/hash-rbac.yml).\n\nKpow [supports a number of authentication providers](https://docs.factorhouse.io/kpow-ee/authentication/overview/) including: \n\n* Okta\n* OpenID\n* OAuth2\n* Ldap\n* AWS SSO\n* Azure AD\n* Keycloak\n\n### Configure a Kpow Enterprise Trial License\n\n* Get a [free Kpow Trial license](https://factorhouse.io/kpow/get-started/)\n* Enter the license details into [resources/kpow/local-trial.env](resources/kpow/local-trial.env)\n\n```\nJAVA_TOOL_OPTIONS=-Djava.security.auth.login.config=/etc/kpow/jaas/hash-jaas.conf\nAUTH_PROVIDER_TYPE=jetty\nRBAC_CONFIGURATION_FILE=/etc/kpow/rbac/hash-rbac.yml\n\nBOOTSTRAP=kafka-1:19092,kafka-2:19093,kafka-3:19094\nCONNECT_REST_URL=http://connect:8083\nSCHEMA_REGISTRY_URL=http://schema:8081\nSCHEMA_REGISTRY_AUTH=USER_INFO\nSCHEMA_REGISTRY_USER=admin\nSCHEMA_REGISTRY_PASSWORD=admin\n\n### Your License Details\nLICENSE_ID=\u003clicense-id\u003e\nLICENSE_CODE=\u003clicense-code\u003e\nLICENSEE=\u003clicensee\u003e\nLICENSE_EXPIRY=\u003clicense-expiry\u003e\nLICENSE_SIGNATURE=\u003clicense-signature\u003e\n```\n\n### Use the Kpow Enterprise Trial Docker Compose\n\nFollow the same start/stop/access method as for Kpow Community with the trial compose configuration, e.g.\n\n```\ndocker compose -f docker-compose-trial.yml up\n```\n\n## Support\n\nAny issues? Contact [support](https://factorhouse.io/support/) or view our [docs](https://docs.factorhouse.io/kpow-ce/).\n\n## License\n\nThis repository is released under the Apache 2.0 License.\n\nCopyright © Factor House.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffactorhouse%2Fkpow-local","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffactorhouse%2Fkpow-local","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffactorhouse%2Fkpow-local/lists"}