{"id":18379178,"url":"https://github.com/ches/docker-kafka","last_synced_at":"2025-08-20T10:31:11.922Z","repository":{"id":20345929,"uuid":"23620761","full_name":"ches/docker-kafka","owner":"ches","description":"Apache Kafka on Docker","archived":false,"fork":false,"pushed_at":"2018-02-19T12:09:42.000Z","size":90,"stargazers_count":146,"open_issues_count":22,"forks_count":137,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-12-17T06:46:24.407Z","etag":null,"topics":["apache-kafka","docker","docker-image","kafka"],"latest_commit_sha":null,"homepage":"https://hub.docker.com/r/ches/kafka/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ches.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-09-03T13:14:11.000Z","updated_at":"2024-05-21T13:58:00.000Z","dependencies_parsed_at":"2022-08-28T19:01:48.325Z","dependency_job_id":null,"html_url":"https://github.com/ches/docker-kafka","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ches%2Fdocker-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ches%2Fdocker-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ches%2Fdocker-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ches%2Fdocker-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ches","download_url":"https://codeload.github.com/ches/docker-kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230415317,"owners_count":18222158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","docker","docker-image","kafka"],"created_at":"2024-11-06T00:37:14.268Z","updated_at":"2024-12-19T10:08:40.839Z","avatar_url":"https://github.com/ches.png","language":"Shell","readme":"Apache Kafka on Docker\n======================\n\nThis repository holds a build definition and supporting files for building a\n[Docker] image to run [Kafka] in containers. It is published as an Automated\nBuild [on Docker Hub], as `ches/kafka`.\n\nThis build intends to provide an operator-friendly Kafka deployment suitable for\nusage in a production Docker environment:\n\n  - It runs one service, no bundled ZooKeeper (for more convenient development,\n    use [Docker Compose]!).\n  - Configuration is parameterized, enabling a Kafka cluster to be run from\n    multiple container instances.\n  - Kafka data and logs can be handled outside the container(s) using volumes.\n  - JMX is exposed, for Kafka and JVM metrics visibility.\n\nIf you find any shortcomings with the build regarding operability, pull requests\nor feedback via GitHub issues are welcomed.\n\n[Docker Compose]: https://docs.docker.com/compose/\n\nUsage Quick Start\n-----------------\n\nHere is a minimal-configuration example running the Kafka broker service, then\nusing the container as a client to run the basic producer and consumer example\nfrom [the Kafka Quick Start]:\n\n```\n# A non-default bridge network enables convenient name-to-hostname discovery\n$ docker network create kafka-net\n\n$ docker run -d --name zookeeper --network kafka-net zookeeper:3.4\n$ docker run -d --name kafka --network kafka-net --env ZOOKEEPER_IP=zookeeper ches/kafka\n\n$ docker run --rm --network kafka-net ches/kafka \\\n\u003e   kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181\nCreated topic \"test\".\n\n# In separate terminals:\n$ docker run --rm --interactive --network kafka-net ches/kafka \\\n\u003e   kafka-console-producer.sh --topic test --broker-list kafka:9092\n\u003ctype some messages followed by newline\u003e\n\n$ docker run --rm --network kafka-net ches/kafka \\\n\u003e   kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server kafka:9092\n```\n\n### Volumes\n\nThe container exposes two volumes that you may wish to bind-mount, or process\nelsewhere with `--volumes-from`:\n\n- `/data`: Path where Kafka's data is stored (`log.dirs` in Kafka configuration)\n- `/logs`: Path where Kafka's logs (`INFO` level) will be written, via log4j\n\n### Ports and Linking\n\nThe container publishes two ports:\n\n- `9092`: Kafka's standard broker communication\n- `7203`: JMX publishing, for e.g. jconsole or VisualVM connection\n\nKafka requires Apache ZooKeeper. You can satisfy the dependency by simply\nlinking another container that exposes ZooKeeper on its standard port of 2181,\nas shown in the above example, **ensuring** that you link using an alias of\n`zookeeper`.\n\nAlternatively, you may configure a specific address for Kafka to find ZK. See\nthe Configuration section below.\n\n### A more complex local development setup\n\nThis example shows more configuration options and assumes that you wish to run a\ndevelopment environment with Kafka ports mapped directly to localhost, for\ninstance if you're writing a producer or consumer and want to avoid rebuilding a\ncontainer for it to run in as you iterate. This requires that localhost is your\nDocker host, i.e. your workstation runs Linux. If you're using something like\nboot2docker, substitute the value of `boot2docker ip` below.\n\n```bash\n$ mkdir -p kafka-ex/{data,logs} \u0026\u0026 cd kafka-ex\n$ docker run -d --name zookeeper --publish 2181:2181 zookeeper:3.4\n$ docker run -d \\\n    --hostname localhost \\\n    --name kafka \\\n    --volume ./data:/data --volume ./logs:/logs \\\n    --publish 9092:9092 --publish 7203:7203 \\\n    --env KAFKA_ADVERTISED_HOST_NAME=127.0.0.1 --env ZOOKEEPER_IP=127.0.0.1 \\\n    ches/kafka\n```\n\nConfiguration\n-------------\n\nSome parameters of Kafka configuration can be set through environment variables\nwhen running the container (`docker run -e VAR=value`). These are shown here\nwith their default values, if any:\n\n- `KAFKA_BROKER_ID=0`\n\n  Maps to Kafka's `broker.id` setting. Must be a unique integer for each broker\n  in a cluster.\n- `KAFKA_PORT=9092`\n\n  Maps to Kafka's `port` setting. The port that the broker service listens on.\n  You will need to explicitly publish a new port from container instances if you\n  change this.\n- `KAFKA_ADVERTISED_HOST_NAME=\u003ccontainer's IP within docker0's subnet\u003e`\n\n  Maps to Kafka's `advertised.host.name` setting. Kafka brokers gossip the list\n  of brokers in the cluster to relieve producers from depending on a ZooKeeper\n  library. This setting should reflect the address at which producers can reach\n  the broker on the network, i.e. if you build a cluster consisting of multiple\n  physical Docker hosts, you will need to set this to the hostname of the Docker\n  *host's* interface where you forward the container `KAFKA_PORT`.\n- `KAFKA_ADVERTISED_PORT=9092`\n\n  As above, for the port part of the advertised address. Maps to Kafka's\n  `advertised.port` setting. If you run multiple broker containers on a single\n  Docker host and need them to be accessible externally, this should be set to\n  the port that you forward to on the Docker host.\n- `KAFKA_DEFAULT_REPLICATION_FACTOR=1`\n\n  Maps to Kafka's `default.replication.factor` setting. The default replication\n  factor for automatically created topics.\n- `KAFKA_NUM_PARTITIONS=1`\n\n  Maps to Kafka's `num.partitions` setting. The default number of log partitions\n  per topic.\n- `KAFKA_AUTO_CREATE_TOPICS_ENABLE=true`\n\n  Maps to Kafka's `auto.create.topics.enable`.\n- `KAFKA_INTER_BROKER_PROTOCOL_VERSION`\n\n  Maps to Kafka's `inter.broker.protocol.version`. If you have a cluster that\n  runs brokers with different Kafka versions make sure they communicate with\n  the same protocol version.\n- `KAFKA_LOG_MESSAGE_FORMAT_VERSION`\n\n  Maps to Kafka's `log.message.format.version`. Specifies the protocol version\n  with which your cluster communicates with its consumers.\n- `KAFKA_LOG_RETENTION_HOURS=168`\n\n  Maps to Kafka's `log.retention.hours`. The number of hours to keep a log file\n  before deleting it.\n\n- `KAFKA_MESSAGE_MAX_BYTES`\n\n  Maps to Kafka's `message.max.bytes`. The maximum size of message that the\n  server can receive. Default: 1000012\n\n- `KAFKA_REPLICA_FETCH_MAX_BYTES`\n\n  Maps to Kafka's `replica.fetch.max.bytes`. The number of bytes of messages to \n  attempt to fetch for each partition. This is not an absolute maximum, if \n  the first message in the first non-empty partition of the fetch is larger \n  than this value, the message will still be returned to ensure that progress \n  can be made. The maximum message size accepted by the broker is defined via \n  message.max.bytes (broker config) or max.message.bytes (topic config). \n  Default: 1048576\n\n- `JAVA_RMI_SERVER_HOSTNAME=$KAFKA_ADVERTISED_HOST_NAME`\n\n  Maps to the `java.rmi.server.hostname` JVM property, which is used to bind the\n  interface that will accept remote JMX connections. Like\n  `KAFKA_ADVERTISED_HOST_NAME`, it may be necessary to set this to a reachable\n  address of *the Docker host* if you wish to connect a JMX client from outside\n  of Docker.\n- `ZOOKEEPER_IP=\u003ctaken from linked \"zookeeper\" container, if available\u003e`\n\n  **Required** if no container is linked with the alias \"zookeeper\" and\n  publishing port 2181, or not using `ZOOKEEPER_CONNECTION_STRING` instead. Used\n  in constructing Kafka's `zookeeper.connect` setting.\n- `ZOOKEEPER_PORT=2181`\n\n  Used in constructing Kafka's `zookeeper.connect` setting.\n- `ZOOKEEPER_CONNECTION_STRING=\u003ccomma separated string of host:port pairs\u003e`\n\n  Set a string with host:port pairs for connecting to a ZooKeeper Cluster. This\n  setting overrides `ZOOKEEPER_IP` and `ZOOKEEPER_PORT`.\n- `ZOOKEEPER_CHROOT`, ex: `/v0_8_1`\n\n  ZooKeeper root path used in constructing Kafka's `zookeeper.connect` setting.\n  This is blank by default, which means Kafka will use the ZK `/`. You should\n  set this if the ZK instance/cluster is shared by other services, or to\n  accommodate Kafka upgrades that change schema. Starting in Kafka 0.8.2, it\n  will create the path in ZK automatically; with earlier versions, you must\n  ensure it is created before starting brokers.\n\nJMX\n---\n\nRemote JMX access can be a bit of a pain to set up. The start script for this\ncontainer tries to make it as painless as possible, but it's important to\nunderstand that if you want to connect a client like VisualVM from outside other\nDocker containers (e.g. directly from your host OS in development), then you'll\nneed to configure RMI to be addressed *as the Docker host IP or hostname*. If\nyou have set `KAFKA_ADVERTISED_HOST_NAME`, that value will be used and is\nprobably what you want. If not (you're only using other containers to talk to\nKafka brokers) or you need to override it for some reason, then you can instead\nset `JAVA_RMI_SERVER_HOSTNAME`.\n\nFor example in practice, if your Docker host is VirtualBox run by Docker\nMachine, a `run` command like this should allow you to connect VisualVM from\nyour host OS to `$(docker-machine ip docker-vm):7203`:\n\n    $ docker run -d --name kafka -p 7203:7203 \\\n        --link zookeeper:zookeeper \\\n        --env JAVA_RMI_SERVER_HOSTNAME=$(docker-machine ip docker-vm) \\\n        ches/kafka\n\nNote that it is fussy about port as well—it may not work if the same port\nnumber is not used within the container and on the host (any advice for\nworkarounds is welcome).\n\nFinally, please note that by default remote JMX has authentication and SSL\nturned off (these settings are taken from Kafka's own default start scripts). If\nyou expose the JMX hostname/port from the Docker host in a production\nenvironment, you should make make certain that access is locked down\nappropriately with firewall rules or similar. A more advisable setup in a Docker\nsetting would be to run a metrics collector in another container, and link it to\nthe Kafka container(s).\n\nIf you need finer-grained configuration, you can totally control the relevant\nJava system properties by setting `KAFKA_JMX_OPTS` yourself—see `start.sh`.\n\nFork Legacy\n-----------\n\nThis image/repo was originally forked from [relateiq/kafka]. My original\nmotivations for forking were:\n\n- Change the Kafka binary source to an official Apache artifact. RelateIQ's was\n  on a private S3 bucket, and this opaqueness is not suitable for a\n  publicly-shared image for reasons of trust.\n- Changes described in [this pull request](https://github.com/relateiq/docker-kafka/pull/4).\n\nAfter a period of unresponsiveness from upstream on pull requests and my repo\ntallying far more downloads on Docker Hub, I have made further updates and\nchanges with the expectation of maintaining independently from here on. This\nproject's changelog file describes these in detail.\n\n\n[Docker]: http://www.docker.io\n[Kafka]: http://kafka.apache.org\n[on Docker Hub]: https://hub.docker.com/r/ches/kafka/\n[relateiq/kafka]: https://github.com/relateiq/docker-kafka\n[the Kafka Quick Start]: http://kafka.apache.org/documentation.html#quickstart\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fches%2Fdocker-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fches%2Fdocker-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fches%2Fdocker-kafka/lists"}