{"id":16800015,"url":"https://github.com/dajudge/kafkaproxy","last_synced_at":"2025-07-15T20:47:38.652Z","repository":{"id":36170533,"uuid":"220828831","full_name":"dajudge/kafkaproxy","owner":"dajudge","description":"kafkaproxy is a reverse proxy for the wire protocol of Apache Kafka.","archived":false,"fork":false,"pushed_at":"2023-06-13T23:00:15.000Z","size":467,"stargazers_count":78,"open_issues_count":10,"forks_count":12,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-04T02:34:21.310Z","etag":null,"topics":["kafka","kafka-protocol"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dajudge.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-10T18:03:07.000Z","updated_at":"2025-03-25T09:37:31.000Z","dependencies_parsed_at":"2024-10-28T12:19:58.273Z","dependency_job_id":"4889c3c3-5ef6-421e-bb28-f7760971bbff","html_url":"https://github.com/dajudge/kafkaproxy","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"purl":"pkg:github/dajudge/kafkaproxy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dajudge%2Fkafkaproxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dajudge%2Fkafkaproxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dajudge%2Fkafkaproxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dajudge%2Fkafkaproxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dajudge","download_url":"https://codeload.github.com/dajudge/kafkaproxy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dajudge%2Fkafkaproxy/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265458559,"owners_count":23769175,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-protocol"],"created_at":"2024-10-13T09:30:31.298Z","updated_at":"2025-07-15T20:47:38.596Z","avatar_url":"https://github.com/dajudge.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CI](https://github.com/dajudge/kafkaproxy/actions/workflows/build.yaml/badge.svg)](https://github.com/dajudge/kafkaproxy/actions/workflows/build.yaml)\n[![Codacy Badge](https://app.codacy.com/project/badge/Grade/2842f83708864fd68a65bf0a82f32bb0)](https://www.codacy.com/gh/dajudge/kafkaproxy/dashboard?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=dajudge/kafkaproxy\u0026amp;utm_campaign=Badge_Grade)\n\n# kafkaproxy\nkafkaproxy is a reverse proxy for the wire protocol of Apache Kafka. \n\nSince the Kafka wire protocol publishes the set of available broker endpoints from the brokers to the clients during\nclient bootstrapping, the brokers must be configured to publish endpoints that are reachable from the client side. This\ncan be a cumbersome restriction in several different situations, such as:\n*  Network topologies preventing direct access to the broker nodes\n*  Multiple networks from which broker nodes should be reachable\n*  DNS resolution restrictions when accessing TLS secured broker nodes\n*  Using the sidecar pattern for TLS termination in Kubernetes \n\nThis is where kafkaproxy comes into play and allows for transparent relaying of the Kafka wire protocol by rewriting\nthe relevant parts of the communication where the brokers publish the endpoint names - with user-configurable endpoints\nwhere the the proxy instances can be reached.\n\n# Run kafkaproxy in Docker\nkafkaproxy is built to be run in a container. The released versions are available at [Docker Hub](https://hub.docker.com/r/dajudge/kafkaproxy).\n## Command line\nThe following configuration parameters are mandatory:\n* `KAFKAPROXY_HOSTNAME`: the hostname at which the proxy can be reached from the clients.\n* `KAFKAPROXY_BASE_PORT`: the first port to be used by kafkaproxy.\n* `KAFKAPROXY_BOOTSTRAP_SERVERS`: the comma separated list of initially mapped endpoints. This is usually the list of bootstrap brokers or a load balancer in front of the kafka brokers.\n\nFor example:\n```\ndocker run \\\n    --net host \\\n    -e KAFKAPROXY_HOSTNAME=localhost \\\n    -e KAFKAPROXY_BASE_PORT=4000 \\\n    -e KAFKAPROXY_BOOTSTRAP_SERVERS=kafka:9092 \\\n    -d dajudge/kafkaproxy:0.0.18\n``` \n*Note:* You will have to make the proxy ports (starting with `KAFKAPROXY_BASE_PORT` and incrementing from there) available from outside the container with `-p PORT:PORT` if you're not using `--net host`.\n\n## Demonstration setup with `docker-compose`\nIf you have `docker-compose` installed you can try out kafkaproxy by using the demonstration setup provided in the\n`example` directory. So clone the [kafkaproxy repo](https://github.com/dajudge/kafkaproxy) and run the following commands:\n\n**Step 1:** Start kafka, zookeeper and kafkaproxy.\n```\ndocker-compose -f example/docker-compose.yml up -d\n```\nKafka will take a couple of seconds to fully start and become available.\n\n**Step 2:** Create `my-test-topic`.\n```\ndocker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-topics --create --topic my-test-topic --bootstrap-server localhost:4000 --partitions 1 --replication-factor 1\n```\n\n**Step 3:** Publish a message to `my-test-topic`.\n```\necho \"Hello, kafkaproxy\" | docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-console-producer --broker-list localhost:4000 --topic my-test-topic\n```\n\n**Step 4:** Consume to produced message from `my-test-topic`.\n```\ndocker run --rm --net host -it confluentinc/cp-zookeeper:5.2.1 kafka-console-consumer --bootstrap-server localhost:4000 --topic my-test-topic --from-beginning --max-messages 1\n```\n\n**Cleanup:** Stop and remove the demonstration containers.\n```\ndocker-compose -f example/docker-compose.yml rm -sf\n```\n\n**Explanation:** The `docker-compose.yml` file starts up a kafka broker (along with it's required zookeeper) that is\nonly available from within the docker network as `kafka1:9092`. The kafkaproxy is configured to\nproxy this kafka instance as `localhost:4000` which is also mapped from outside the docker network.\n\n# Configuration\nkafkaproxy is configured using mostly environment variables and a broker map file in YAML format. The following\nsection describe the configuration options in detail.\n\n## General configuration\nkafkaproxy requires some general information to start. \n\n| Name                               | Default value | Destription\n| ---------------------------------- | ------------- | -----------\n| `KAFKAPROXY_HOSTNAME`              |               | The hostname of the proxy as seen by the clients.\n| `KAFKAPROXY_BASE_PORT`             |               | The base of the ports to be used by the proxy. Each new required port is created by incrementing on top of the base port.\n| `KAFKAPROXY_BIND_ADDRESS`          | `0.0.0.0`     | The address server sockets will bind to (both proxy ports and HTTP).\n| `KAFKAPROXY_BOOTSTRAP_SERVERS`     |               | The comma separated list of initially mapped endpoints. This is usually the list of bootstrap brokers or a load balancer in front of the kafka brokers.\n| `KAFKAPROXY_LOG_LEVEL`             | `INFO`        | The log level of the root logger. This must be a valid log level for [logback](http://logback.qos.ch/manual/configuration.html).\n| `KAFKAPROXY_ENABLE_JSON_LOGGING`   | `false`       | Enable/disable json logging feature.\n| `KAFKAPROXY_HTTP_PORT`             | `8080`        | The HTTP port the metrics REST endpoint will be exposed on. \n \n## Client SSL configuration\nThe client SSL configuration determines how the Kafka clients have to connect to kafkaproxy.\nConfiguration can be provided using the following environment variables:\n\n| Name                                                 | Default value | Destription\n| ---------------------------------------------------- | ------------- | -----------\n| `KAFKAPROXY_CLIENT_SSL_ENABLED`                      | `false`       | Enables SSL encrypted communication between clients and kafkaproxy. \n| `KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_LOCATION`          |               | The filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.\n| `KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_REFRESH_SECS`      | 300           | The minimum amount if time between checks for updates of the trust store in seconds.\n| `KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_PASSWORD`          |               | The password to access the trust store. Provide no value if the trust store is not password protected. Ignored when `KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_PASSWORD_LOCATION` is set.\n| `KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_PASSWORD_LOCATION` |               | The filesystem location of the password to access the trust store. Overrides `KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_PASSWORD`.\n| `KAFKAPROXY_CLIENT_SSL_TRUSTSTORE_TYPE`              | `jks`         | The type of the trust store.\n| `KAFKAPROXY_CLIENT_SSL_KEYSTORE_LOCATION`            |               | The filesystem location of the proxy's server key store. If no value is provided the JRE's default key store will be used.\n| `KAFKAPROXY_CLIENT_SSL_KEYSTORE_REFRESH_SECS`        | 300           | The minimum amount if time between checks for updates of the proxy's server key store in seconds.\n| `KAFKAPROXY_CLIENT_SSL_KEYSTORE_PASSWORD`            |               | The password to access the proxy's server key store. Provide no value if the key store is not password protected. Ignored when `KAFKAPROXY_CLIENT_SSL_KEYSTORE_PASSWORD_LOCATION` is set.\n| `KAFKAPROXY_CLIENT_SSL_KEYSTORE_PASSWORD_LOCATION`   |               | The filesystem location of the password to access the proxy's server key store. Overrides `KAFKAPROXY_CLIENT_SSL_KEYSTORE_PASSWORD`.\n| `KAFKAPROXY_CLIENT_SSL_KEY_PASSWORD`                 |               | The password to access the proxy's server key. Provide no value if the key is not password protected.\n| `KAFKAPROXY_CLIENT_SSL_KEY_TYPE`                     | `jks`         | The type of the key store.\n| `KAFKAPROXY_CLIENT_SSL_AUTH_REQUIRED`                | `false`       | Require a valid client certificate from clients connecting to the proxy.\n\n## Kafka SSL configuration\nThe Kafka SSL configuration determines how kafkaproxy connects to the Kafka broker instances.\nConfiguration can be provided using the following environment variables:\n\n| Name                                                | Default value | Destription\n| --------------------------------------------------- | ------------- | -----------\n| `KAFKAPROXY_KAFKA_SSL_ENABLED`                      | `false`       | Enables SSL encrypted communication kafkaproxy and the Kafka brokers.\n| `KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_LOCATION`          |               | The filesystem location of the trust store to use. If no value is provided the JRE's default trust store will be used.\n| `KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_REFRESH_SECS`      | 300           | The minimum amount if time between checks for updates of the trust store store in seconds.\n| `KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_PASSWORD`          |               | The password to access the trust store. Provide no value if the trust store is not password protected. Ignored when `KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_PASSWORD_LOCATION` is set.\n| `KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_PASSWORD_LOCATION` |               | The filesystem location of the password to access the trust store. Overrides `KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_PASSWORD`. \n| `KAFKAPROXY_KAFKA_SSL_TRUSTSTORE_TYPE`              | `jks`         | The type of the trust store.\n| `KAFKAPROXY_KAFKA_SSL_VERIFY_HOSTNAME`              | `true`        | Indicates if the hostnames of the Kafka brokers are validated against the SSL certificates they provide when connecting.\n| `KAFKAPROXY_KAFKA_SSL_KEYSTORE_LOCATION`            |               | The filesystem location of the proxy's client key store. Required only when `KAFKAPROXY_KAFKA_SSL_CLIENT_CERT_STRATEGY` is set to `KEYSTORE`.\n| `KAFKAPROXY_KAFKA_SSL_KEYSTORE_REFRESH_SECS`        | 300           | The minimum amount if time between checks for updates of the proxy client's key store in seconds.\n| `KAFKAPROXY_KAFKA_SSL_KEYSTORE_PASSWORD`            |               | The password to access the proxy's client key store. Provide no value if the key store is not password protected. Ignored when `KAFKAPROXY_KAFKA_SSL_KEYSTORE_PASSWORD_LOCATION` is set.\n| `KAFKAPROXY_KAFKA_SSL_KEYSTORE_PASSWORD_LOCATION`   |               | The filesystem location of the password to access the proxy's client key store. Overrides `KAFKAPROXY_KAFKA_SSL_KEYSTORE_PASSWORD`.\n| `KAFKAPROXY_KAFKA_SSL_KEYSTORE_TYPE`                | `jks`         | The type of the key store.\n| `KAFKAPROXY_KAFKA_SSL_KEY_PASSWORD`                 |               | The password to access the proxy's client key. Provide no value if the key is not password protected. Ignored when `KAFKAPROXY_KAFKA_SSL_KEY_PASSWORD_LOCATION` is set.\n| `KAFKAPROXY_KAFKA_SSL_KEY_PASSWORD_LOCATION`        |               | The filesytem location of the password to access the proxy's client key. Overrides `KAFKAPROXY_KAFKA_SSL_KEY_PASSWORD`.\n\n# Further Reading\n*  [A Guide To The Kafka Protocol](https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol)\n*  [Kafka protocol guide](http://kafka.apache.org/protocol.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdajudge%2Fkafkaproxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdajudge%2Fkafkaproxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdajudge%2Fkafkaproxy/lists"}