{"id":13807201,"url":"https://github.com/wurstmeister/kafka-docker","last_synced_at":"2025-05-13T23:11:00.156Z","repository":{"id":12732684,"uuid":"15405607","full_name":"wurstmeister/kafka-docker","owner":"wurstmeister","description":"Dockerfile for Apache Kafka","archived":false,"fork":false,"pushed_at":"2024-05-08T12:11:08.000Z","size":336,"stargazers_count":6943,"open_issues_count":129,"forks_count":2723,"subscribers_count":160,"default_branch":"master","last_synced_at":"2025-04-10T12:40:44.818Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://wurstmeister.github.io/kafka-docker/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wurstmeister.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-12-23T22:01:52.000Z","updated_at":"2025-04-08T15:55:40.000Z","dependencies_parsed_at":"2024-01-22T19:33:47.326Z","dependency_job_id":"e77ccd38-80fe-4a31-ac36-4995ee1e6646","html_url":"https://github.com/wurstmeister/kafka-docker","commit_stats":{"total_commits":203,"total_committers":78,"mean_commits":"2.6025641025641026","dds":0.7093596059113301,"last_synced_commit":"901c084811fa9395f00af3c51e0ac6c32c697034"},"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wurstmeister%2Fkafka-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wurstmeister%2Fkafka-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wurstmeister%2Fkafka-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wurstmeister%2Fkafka-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wurstmeister","download_url":"https://codeload.github.com/wurstmeister/kafka-docker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254042339,"owners_count":22004901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T01:01:22.438Z","updated_at":"2025-05-13T23:10:55.109Z","avatar_url":"https://github.com/wurstmeister.png","language":"Shell","funding_links":[],"categories":["Shell","Data Ingestion"],"sub_categories":[],"readme":"[![Docker Pulls](https://img.shields.io/docker/pulls/wurstmeister/kafka.svg)](https://hub.docker.com/r/wurstmeister/kafka/)\n[![Docker Stars](https://img.shields.io/docker/stars/wurstmeister/kafka.svg)](https://hub.docker.com/r/wurstmeister/kafka/)\n[![](https://images.microbadger.com/badges/version/wurstmeister/kafka.svg)](https://microbadger.com/images/wurstmeister/kafka \"Get your own version badge on microbadger.com\")\n[![](https://images.microbadger.com/badges/image/wurstmeister/kafka.svg)](https://microbadger.com/images/wurstmeister/kafka \"Get your own image badge on microbadger.com\")\n[![Build Status](https://app.travis-ci.com/wurstmeister/kafka-docker.svg?branch=master)](https://app.travis-ci.com/wurstmeister/kafka-docker)\n\n\nkafka-docker\n============\n\nDockerfile for [Apache Kafka](http://kafka.apache.org/)\n\nThe image is available directly from [Docker Hub](https://hub.docker.com/r/wurstmeister/kafka/)\n\nTags and releases\n-----------------\n\nAll versions of the image are built from the same set of scripts with only minor variations (i.e. certain features are not supported on older versions). The version format mirrors the Kafka format, `\u003cscala version\u003e-\u003ckafka version\u003e`. Initially, all images are built with the recommended version of scala documented on [http://kafka.apache.org/downloads](http://kafka.apache.org/downloads). To list all available tags:\n\n```\ncurl -s https://registry.hub.docker.com/v2/repositories/wurstmeister/kafka/tags\\?page_size\\=1024 | jq -r '.results[].name' | sort -u | egrep '\\d.\\d{2}-.*'\n```\n\nEverytime the image is updated, all tags will be pushed with the latest updates. This should allow for greater consistency across tags, as well as any security updates that have been made to the base image.\n\n---\n\n## Announcements\n\n* **04-Jun-2019** - Update base image to openjdk 212 ([Release notes](https://www.oracle.com/technetwork/java/javase/8u212-relnotes-5292913.html). Please force pull to get these latest updates - including security patches etc.\n\n---\n\n## Pre-Requisites\n\n- install docker-compose [https://docs.docker.com/compose/install/](https://docs.docker.com/compose/install/)\n- modify the ```KAFKA_ADVERTISED_HOST_NAME``` in [docker-compose.yml](https://raw.githubusercontent.com/wurstmeister/kafka-docker/master/docker-compose.yml) to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host ip if you want to run multiple brokers.)\n- if you want to customize any Kafka parameters, simply add them as environment variables in ```docker-compose.yml```, e.g. in order to increase the ```message.max.bytes``` parameter set the environment to ```KAFKA_MESSAGE_MAX_BYTES: 2000000```. To turn off automatic topic creation set ```KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'```\n- Kafka's log4j usage can be customized by adding environment variables prefixed with ```LOG4J_```. These will be mapped to ```log4j.properties```. For example: ```LOG4J_LOGGER_KAFKA_AUTHORIZER_LOGGER=DEBUG, authorizerAppender```\n\n**NOTE:** There are several 'gotchas' with configuring networking. If you are not sure about what the requirements are, please check out the [Connectivity Guide](https://github.com/wurstmeister/kafka-docker/wiki/Connectivity) in the [Wiki](https://github.com/wurstmeister/kafka-docker/wiki)\n\n## Usage\n\nStart a cluster:\n\n- ```docker-compose up -d ```\n\nAdd more brokers:\n\n- ```docker-compose scale kafka=3```\n\nDestroy a cluster:\n\n- ```docker-compose stop```\n\n## Note\n\nThe default ```docker-compose.yml``` should be seen as a starting point. By default each broker will get a new port number and broker id on restart. Depending on your use case this might not be desirable. If you need to use specific ports and broker ids, modify the docker-compose configuration accordingly, e.g. [docker-compose-single-broker.yml](https://github.com/wurstmeister/kafka-docker/blob/master/docker-compose-single-broker.yml):\n\n- ```docker-compose -f docker-compose-single-broker.yml up```\n\n## Broker IDs\n\nYou can configure the broker id in different ways\n\n1. explicitly, using ```KAFKA_BROKER_ID```\n2. via a command, using ```BROKER_ID_COMMAND```, e.g. ```BROKER_ID_COMMAND: \"hostname | awk -F'-' '{print $$2}'\"```\n\nIf you don't specify a broker id in your docker-compose file, it will automatically be generated (see [https://issues.apache.org/jira/browse/KAFKA-1070](https://issues.apache.org/jira/browse/KAFKA-1070). This allows scaling up and down. In this case it is recommended to use the ```--no-recreate``` option of docker-compose to ensure that containers are not re-created and thus keep their names and ids.\n\n\n## Automatically create topics\n\nIf you want to have kafka-docker automatically create topics in Kafka during\ncreation, a ```KAFKA_CREATE_TOPICS``` environment variable can be\nadded in ```docker-compose.yml```.\n\nHere is an example snippet from ```docker-compose.yml```:\n\n        environment:\n          KAFKA_CREATE_TOPICS: \"Topic1:1:3,Topic2:1:1:compact\"\n\n```Topic 1``` will have 1 partition and 3 replicas, ```Topic 2``` will have 1 partition, 1 replica and a `cleanup.policy` set to `compact`. Also, see FAQ: [Topic compaction does not work](https://github.com/wurstmeister/kafka-docker/wiki#topic-compaction-does-not-work)\n\nIf you wish to use multi-line YAML or some other delimiter between your topic definitions, override the default `,` separator by specifying the `KAFKA_CREATE_TOPICS_SEPARATOR` environment variable.\n\nFor example, `KAFKA_CREATE_TOPICS_SEPARATOR: \"$$'\\n'\"` would use a newline to split the topic definitions. Syntax has to follow docker-compose escaping rules, and [ANSI-C](https://www.gnu.org/software/bash/manual/html_node/ANSI_002dC-Quoting.html) quoting.\n\n## Advertised hostname\n\nYou can configure the advertised hostname in different ways\n\n1. explicitly, using ```KAFKA_ADVERTISED_HOST_NAME```\n2. via a command, using ```HOSTNAME_COMMAND```, e.g. ```HOSTNAME_COMMAND: \"route -n | awk '/UG[ \\t]/{print $$2}'\"```\n\nWhen using commands, make sure you review the \"Variable Substitution\" section in [https://docs.docker.com/compose/compose-file/](https://docs.docker.com/compose/compose-file/#variable-substitution)\n\nIf ```KAFKA_ADVERTISED_HOST_NAME``` is specified, it takes precedence over ```HOSTNAME_COMMAND```\n\nFor AWS deployment, you can use the Metadata service to get the container host's IP:\n```\nHOSTNAME_COMMAND=wget -t3 -T2 -qO-  http://169.254.169.254/latest/meta-data/local-ipv4\n```\nReference: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html\n\n### Injecting HOSTNAME_COMMAND into configuration\n\nIf you require the value of `HOSTNAME_COMMAND` in any of your other `KAFKA_XXX` variables, use the `_{HOSTNAME_COMMAND}` string in your variable value, i.e.\n\n```\nKAFKA_ADVERTISED_LISTENERS=SSL://_{HOSTNAME_COMMAND}:9093,PLAINTEXT://9092\n```\n\n## Advertised port\n\nIf the required advertised port is not static, it may be necessary to determine this programatically. This can be done with the `PORT_COMMAND` environment variable.\n\n```\nPORT_COMMAND: \"docker port $$(hostname) 9092/tcp | cut -d: -f2\"\n```\n\nThis can be then interpolated in any other `KAFKA_XXX` config using the `_{PORT_COMMAND}` string, i.e.\n\n```\nKAFKA_ADVERTISED_LISTENERS: PLAINTEXT://1.2.3.4:_{PORT_COMMAND}\n```\n\n## Listener Configuration\n\nIt may be useful to have the [Kafka Documentation](https://kafka.apache.org/documentation/) open, to understand the various broker listener configuration options.\n\nSince 0.9.0, Kafka has supported [multiple listener configurations](https://issues.apache.org/jira/browse/KAFKA-1809) for brokers to help support different protocols and discriminate between internal and external traffic. Later versions of Kafka have deprecated ```advertised.host.name``` and ```advertised.port```.\n\n**NOTE:** ```advertised.host.name``` and ```advertised.port``` still work as expected, but should not be used if configuring the listeners.\n\n### Example\n\nThe example environment below:\n\n```\nHOSTNAME_COMMAND: curl http://169.254.169.254/latest/meta-data/public-hostname\nKAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094\nKAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094\nKAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT\nKAFKA_INTER_BROKER_LISTENER_NAME: INSIDE\n```\n\nWill result in the following broker config:\n\n```\nadvertised.listeners = OUTSIDE://ec2-xx-xx-xxx-xx.us-west-2.compute.amazonaws.com:9094,INSIDE://:9092\nlisteners = OUTSIDE://:9094,INSIDE://:9092\ninter.broker.listener.name = INSIDE\n```\n\n### Rules\n\n* No listeners may share a port number.\n* An advertised.listener must be present by protocol name and port number in the list of listeners.\n\n## Broker Rack\n\nYou can configure the broker rack affinity in different ways\n\n1. explicitly, using ```KAFKA_BROKER_RACK```\n2. via a command, using ```RACK_COMMAND```, e.g. ```RACK_COMMAND: \"curl http://169.254.169.254/latest/meta-data/placement/availability-zone\"```\n\nIn the above example the AWS metadata service is used to put the instance's availability zone in the ```broker.rack``` property.\n\n## JMX\n\nFor monitoring purposes you may wish to configure JMX. Additional to the standard JMX parameters, problems could arise from the underlying RMI protocol used to connect\n\n* java.rmi.server.hostname - interface to bind listening port\n* com.sun.management.jmxremote.rmi.port - The port to service RMI requests\n\nFor example, to connect to a kafka running locally (assumes exposing port 1099)\n\n      KAFKA_JMX_OPTS: \"-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=1099\"\n      JMX_PORT: 1099\n\nJconsole can now connect at ```jconsole 192.168.99.100:1099```\n\n## Docker Swarm Mode\n\nThe listener configuration above is necessary when deploying Kafka in a Docker Swarm using an overlay network. By separating OUTSIDE and INSIDE listeners, a host can communicate with clients outside the overlay network while still benefiting from it from within the swarm.\n\nIn addition to the multiple-listener configuration, additional best practices for operating Kafka in a Docker Swarm include:\n\n* Use \"deploy: global\" in a compose file to launch one and only one Kafka broker per swarm node.\n* Use compose file version '3.2' (minimum Docker version 16.04) and the \"long\" port definition with the port in \"host\" mode instead of the default \"ingress\" load-balanced port binding. This ensures that outside requests are always routed to the correct broker. For example:\n\n```\nports:\n   - target: 9094\n     published: 9094\n     protocol: tcp\n     mode: host\n```\n\nOlder compose files using the short-version of port mapping may encounter Kafka client issues if their connection to individual brokers cannot be guaranteed.\n\nSee the included sample compose file ```docker-compose-swarm.yml```\n\n## Release process\n\nSee the [wiki](https://github.com/wurstmeister/kafka-docker/wiki/ReleaseProcess) for information on adding or updating versions to release to Dockerhub.\n\n## Tutorial\n\n[http://wurstmeister.github.io/kafka-docker/](http://wurstmeister.github.io/kafka-docker/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwurstmeister%2Fkafka-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwurstmeister%2Fkafka-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwurstmeister%2Fkafka-docker/lists"}