{"id":22379161,"url":"https://github.com/sky-uk/kafka-configurator","last_synced_at":"2025-07-31T01:32:00.657Z","repository":{"id":46594035,"uuid":"93863467","full_name":"sky-uk/kafka-configurator","owner":"sky-uk","description":"Kafka Topic Configurator","archived":false,"fork":false,"pushed_at":"2023-01-14T17:44:02.000Z","size":113,"stargazers_count":33,"open_issues_count":13,"forks_count":9,"subscribers_count":19,"default_branch":"master","last_synced_at":"2023-07-07T11:01:56.401Z","etag":null,"topics":["cli","configuration","kafka"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sky-uk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-06-09T13:51:46.000Z","updated_at":"2023-07-07T11:01:56.402Z","dependencies_parsed_at":"2023-02-09T20:15:33.834Z","dependency_job_id":null,"html_url":"https://github.com/sky-uk/kafka-configurator","commit_stats":null,"previous_names":[],"tags_count":15,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fkafka-configurator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fkafka-configurator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fkafka-configurator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fkafka-configurator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sky-uk","download_url":"https://codeload.github.com/sky-uk/kafka-configurator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228204604,"owners_count":17884711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","configuration","kafka"],"created_at":"2024-12-04T23:09:09.600Z","updated_at":"2024-12-04T23:09:10.116Z","avatar_url":"https://github.com/sky-uk.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Configurator\n\n[![Build Status](https://travis-ci.org/sky-uk/kafka-configurator.svg?branch=master)](https://travis-ci.org/sky-uk/kafka-configurator)\n[![Download](https://api.bintray.com/packages/sky-uk/oss-maven/kafka-configurator/images/download.svg)](https://bintray.com/sky-uk/oss-maven/kafka-configurator/_latestVersion)\n\nCommand line tool to create and update Kafka topics based on the provided configuration.\n\nThis software can be used in one of two ways:\n\n - As a standalone tool for automatically creating topics and updating their parameters. It reads a YAML description of the desired setup, compares it with the current state and alters the topics that are different.\n \n - As a dependency that can be added to your code-base as a library that allows you to manage kafka-topics within your application.\n\n\n## Usage as a standalone program\n### Download\n\n#### Binary archive\n\nReleased artifacts are published to [Bintray](https://bintray.com/sky-uk/oss-maven/kafka-configurator/_latestVersion#files) as zip or tgz archives.\n\nIt does not require an installation process: just extract the archive into any directory and execute `bin/kafka-configurator` (or `bin\\kafka-configurator.bat` on Windows) to see the usage instructions.\n\n#### Docker Image\n\nThe Docker Image is available from Docker Hub at [skyuk/kafka-configurator](https://hub.docker.com/r/skyuk/kafka-configurator)\n\n\n```\nUsage: kafka-configurator [options]\n\n  -f, --files \u003cfile1\u003e,\u003cfile2\u003e...\n                           Topic configuration files\n  --bootstrap-servers \u003cvalue\u003e\n                           Kafka brokers URLs for bootstrap (comma-separated)\n  --properties \u003cvalue\u003e     Kafka admin client config as comma-separated pairs\n```\n\nThe topic configuration file has the following format:\n```yaml\ntopic1:\n  partitions: 10\n  replication: 3\n  config:\n    cleanup.policy: compact\n    delete.retention.ms: 86400000\n    min.compaction.lag.ms: 21600000\n    retention.ms: 0\n    min.insync.replicas: 3\n\ntopic2:\n  partitions: 5\n  replication: 2\n  config:\n    cleanup.policy: delete\n    delete.retention.ms: 0\n    retention.ms: 604800000\n    min.insync.replicas: 2\n```\n\nThe root items are topic names to be created or updated, and contain their configuration parameters: `partitions` and `replication` are integers, while the `config` block accepts any valid [topic-level configuration](https://kafka.apache.org/documentation/#topicconfigs).\nWe let Kafka validate these configurations for us so we don't have to explicitly support each topic-level configuration.\n[Yaml anchors](https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/) can also be used to reduce duplication in the topic configuration (see example [here](src/test/resources/topic-configuration-anchors.yml)).\n\n### Demo\n\nCreate a `test-topics.yml` file with the contents of the example configuration above.\n\n##### Using the extracted binary\n\nStart Kafka and Zookeeper using two separate shells in the Kafka root directory:\n```\n1$ bin/zookeeper-server-start.sh config/zookeeper.properties\n2$ bin/kafka-server-start.sh config/server.properties\n```\n\nExecute the Kafka Configurator:\n```\n$ bin/kafka-configurator -f test-topics.yml --zookeeper localhost:2181\n```\n\nQuery the topics using the CLI tool bundled with Kafka:\n```\n$ bin/kafka-topics.sh --zookeeper localhost:2181 --describe\nTopic:topic1    PartitionCount:10       ReplicationFactor:1     Configs:retention.ms=0,delete.retention.ms=86400000,min.insync.replicas=2,cleanup.policy=compact\n        ...\nTopic:topic2    PartitionCount:5        ReplicationFactor:1     Configs:retention.ms=86400000,delete.retention.ms=0,min.insync.replicas=2,cleanup.policy=delete\n        ...\n```\n\nAny changes to the `test-topics.yml` file will be applied to the existing topics at each subsequent run.\n\n##### Using the Docker image\n\nAssuming you know the `\u003czookeeper_address\u003e` and have placed your config file named `test-topics.yml` inside the `\u003cconfig_dir_on_host\u003e` directory on the Docker host, an example of how to run the image is:\n\n`docker run -it -v \u003cconfig_dir_on_host\u003e:/etc/kafka-configurator skyuk/kafka-configurator -f=/etc/kafka-configurator/test-topics.yml --zookeeper=\u003czookeeper_address\u003e`\n\nAlternatively you could extend the `skyuk/kafka-configurator` image and `COPY` your configuration file directly into your extended image.\n\n##### Injecting Kafka Admin client config\n\nAny [Kafka Admin client config](http://kafka.apache.org/documentation/#adminclientconfigs) is supported, both when running the binary directly or via Docker. \n\nThese can be passed in the `--properties` command line option as key=value comma separated pairs:\n\n`kafka-configurator -f test-topics.yml --zookeeper localhost:2181 --properties client.id=foo,ssl.key.password=bar`\n\nOr they can be injected as environment variables prefixed with `KAFKA_`:\n\n`docker run -it -v \u003cconfig_dir_on_host\u003e:/etc/kafka-configurator -e KAFKA_CLIENT_ID=foo -e KAFKA_SSL_KEY_PASSWORD=bar skyuk/kafka-configurator -f=/etc/kafka-configurator/test-topics.yml --zookeeper=\u003czookeeper_address\u003e`\n\n## Usage for adding as a dependency\n\n##### SBT\n```sbtshell\nuseJCenter := true\n\nlibraryDependencies += \"com.sky\" %% \"kafka-configurator\" % \"VERSION\"\n\n```\n\n##### Gradle\n```groovy\nrepositories {\n    jcenter()\n}\n\ncompile 'com.sky:kafka-configurator_2.12:VERSION'\n```\n\n##### Maven\n```xml\n\u003crepositories\u003e\n    \u003crepository\u003e\n        \u003cid\u003ejcenter\u003c/id\u003e\n        \u003cname\u003ejcenter\u003c/name\u003e\n        \u003curl\u003ehttp://jcenter.bintray.com\u003c/url\u003e\n    \u003c/repository\u003e\n\u003c/repositories\u003e\n\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.sky\u003c/groupId\u003e\n  \u003cartifactId\u003ekafka-configurator_2.12\u003c/artifactId\u003e\n  \u003cversion\u003eVERSION\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n### Scala Example:\n```scala\nval config = AppConfig(files = Seq(new File(\"topics.yml\")), bootstrapServers = \"examplekafka.com:9092\")\nval result = KafkaConfiguratorApp.reader(config).configureTopicsFrom(config.files.toList)\n\nresult match {\n  case Success((Nil, logs)) =\u003e logger.info(s\"Topics successfully configured. $logs\")\n  case Success((errors, logs)) =\u003e logger.error(s\"Configurator errors: $errors, $logs\")\n  case Failure(e) =\u003e logger.error(\"Kafka Configurator has failed\", e)\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsky-uk%2Fkafka-configurator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsky-uk%2Fkafka-configurator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsky-uk%2Fkafka-configurator/lists"}