{"id":14109113,"url":"https://github.com/sleighzy/ansible-kafka","last_synced_at":"2026-01-07T11:05:27.131Z","repository":{"id":43280074,"uuid":"83594975","full_name":"sleighzy/ansible-kafka","owner":"sleighzy","description":"Ansible role for installing and configuring Apache Kafka on RedHat and Debian platforms.","archived":false,"fork":false,"pushed_at":"2024-11-14T22:45:51.000Z","size":96,"stargazers_count":129,"open_issues_count":3,"forks_count":86,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-28T05:13:27.271Z","etag":null,"topics":["ansible","ansible-galaxy","ansible-role","centos","debian","kafka","molecule","redhat","rhel"],"latest_commit_sha":null,"homepage":"","language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sleighzy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-01T19:52:53.000Z","updated_at":"2025-02-21T14:00:41.000Z","dependencies_parsed_at":"2023-11-28T22:24:14.769Z","dependency_job_id":"a70318dc-d88a-403a-b01b-8b97fb39b1a8","html_url":"https://github.com/sleighzy/ansible-kafka","commit_stats":{"total_commits":63,"total_committers":11,"mean_commits":"5.7272727272727275","dds":0.6190476190476191,"last_synced_commit":"b6da2851e8c1161255eea45be995de307cd065a4"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sleighzy%2Fansible-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sleighzy%2Fansible-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sleighzy%2Fansible-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sleighzy%2Fansible-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sleighzy","download_url":"https://codeload.github.com/sleighzy/ansible-kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247128753,"owners_count":20888235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","ansible-galaxy","ansible-role","centos","debian","kafka","molecule","redhat","rhel"],"created_at":"2024-08-14T10:02:04.852Z","updated_at":"2026-01-07T11:05:27.103Z","avatar_url":"https://github.com/sleighzy.png","language":"Jinja","funding_links":[],"categories":["Jinja"],"sub_categories":[],"readme":"# Apache Kafka\n\n![Lint Code Base] ![Molecule]\n\nAnsible role to install and configure [Apache Kafka] 3.8\n\n[Apache Kafka] is a distributed event streaming platform using publish-subscribe\ntopics. Applications and streaming components can produce and consume messages\nby subscribing to these topics. Kafka is extremely fast, handling megabytes of\nreads and writes per second from thousands of clients. Messages are persisted\nand replicated to prevent data loss. Data streams are partitioned and can be\nelastically scaled with no downtime.\n\n## WARNING\n\nThis Ansible role does not handle the migration process of upgrading from older\nversions of Kafka. Please ensure that you read the upgrade documentation and\nupdate the relevant configuration files before running this role.\n\n\u003chttps://kafka.apache.org/35/documentation.html#upgrade\u003e\n\nFor example, depending on how you upgrade, the `server.properties` file may need\nthe following properties added to reflect your current version prior to running\nthis Ansible playbook:\n\n- `inter.broker.protocol.version`\n- `log.message.format.version`\n\n## Supported Platforms\n\n- RedHat 6\n- RedHat 7\n- RedHat 8\n- Debian 10.x\n- Ubuntu 18.04.x\n- Ubuntu 20.04.x\n\n## Requirements\n\n- [Apache ZooKeeper]\n- Java 8 (deprecated) / 11 / 17\n\nThe below Apache ZooKeeper role from Ansible Galaxy can be used if one is\nneeded.\n\n```sh\nansible-galaxy install sleighzy.zookeeper\n```\n\nAnsible 2.9.16 or 2.10.4 are the minimum required versions to workaround an\nissue with certain kernels that have broken the `systemd` status check. The\nerror message \"`Service is in unknown state`\" will be output when attempting to\nstart the service via the Ansible role and the task will fail. The service will\nstart as expected if the `systemctl start` command is run on the physical host.\nSee \u003chttps://github.com/ansible/ansible/issues/71528\u003e for more information.\n\n## Role Variables\n\n| Variable                                       | Default                              | Comments                                                         |\n| ---------------------------------------------- | ------------------------------------ | ---------------------------------------------------------------- |\n| kafka_download_base_url                        | \u003chttps://downloads.apache.org/kafka\u003e |                                                                  |\n| kafka_download_validate_certs                  | yes                                  |                                                                  |\n| kafka_version                                  | 3.8.1                                |                                                                  |\n| kafka_scala_version                            | 2.13                                 |                                                                  |\n| kafka_create_user_group                        | true                                 |                                                                  |\n| kafka_user                                     | kafka                                |                                                                  |\n| kafka_group                                    | kafka                                |                                                                  |\n| kafka_root_dir                                 | /opt                                 |                                                                  |\n| kafka_dir                                      | {{ kafka_root_dir }}/kafka           |                                                                  |\n| kafka_start                                    | yes                                  |                                                                  |\n| kafka_restart                                  | yes                                  |                                                                  |\n| kafka_log_dir                                  | /var/log/kafka                       |                                                                  |\n| kafka_broker_id                                | 0                                    |                                                                  |\n| kafka_java_heap                                | -Xms1G -Xmx1G                        |                                                                  |\n| kafka_background_threads                       | 10                                   |                                                                  |\n| kafka_listeners                                | PLAINTEXT://:9092                    |                                                                  |\n| kafka_num_network_threads                      | 3                                    |                                                                  |\n| kafka_num_io_threads                           | 8                                    |                                                                  |\n| kafka_num_replica_fetchers                     | 1                                    |                                                                  |\n| kafka_socket_send_buffer_bytes                 | 102400                               |                                                                  |\n| kafka_socket_receive_buffer_bytes              | 102400                               |                                                                  |\n| kafka_socket_request_max_bytes                 | 104857600                            |                                                                  |\n| kafka_replica_socket_receive_buffer_bytes      | 65536                                |                                                                  |\n| kafka_data_log_dirs                            | /var/lib/kafka/logs                  |                                                                  |\n| kafka_num_partitions                           | 1                                    |                                                                  |\n| kafka_num_recovery_threads_per_data_dir        | 1                                    |                                                                  |\n| kafka_log_cleaner_threads                      | 1                                    |                                                                  |\n| kafka_offsets_topic_replication_factor         | 1                                    |                                                                  |\n| kafka_transaction_state_log_replication_factor | 1                                    |                                                                  |\n| kafka_transaction_state_log_min_isr            | 1                                    |                                                                  |\n| kafka_log_retention_hours                      | 168                                  |                                                                  |\n| kafka_log_segment_bytes                        | 1073741824                           |                                                                  |\n| kafka_log_retention_check_interval_ms          | 300000                               |                                                                  |\n| kafka_auto_create_topics_enable                | false                                |                                                                  |\n| kafka_delete_topic_enable                      | true                                 |                                                                  |\n| kafka_default_replication_factor               | 1                                    |                                                                  |\n| kafka_group_initial_rebalance_delay_ms         | 0                                    |                                                                  |\n| kafka_zookeeper_connect                        | localhost:2181                       |                                                                  |\n| kafka_zookeeper_connection_timeout             | 6000                                 |                                                                  |\n| kafka_bootstrap_servers                        | localhost:9092                       |                                                                  |\n| kafka_consumer_group_id                        | kafka-consumer-group                 |                                                                  |\n| kafka_server_config_params                     |                                      | General dictionary that will be templated into server.properties |\n\nSee [log4j.yml](./defaults/main/002-log4j.yml) for detailed  \nlog4j-related available variables.\n\n## Starting and Stopping Kafka services using systemd\n\n- The Kafka service can be started via: `systemctl start kafka`\n- The Kafka service can be stopped via: `systemctl stop kafka`\n\n## Starting and Stopping Kafka services using initd\n\n- The Kafka service can be started via: `service kafka start`\n- The Kafka service can be stopped via: `service kafka stop`\n\n## Default Properties\n\n| Property                       | Value                |\n| ------------------------------ | -------------------- |\n| ZooKeeper connection           | localhost:2181       |\n| Kafka bootstrap servers        | localhost:9092       |\n| Kafka consumer group ID        | kafka-consumer-group |\n| Kafka broker ID                | 0                    |\n| Number of partitions           | 1                    |\n| Data log file retention period | 168 hours            |\n| Enable auto topic creation     | false                |\n| Enable topic deletion          | true                 |\n\n### Ports\n\n| Port | Description         |\n| ---- | ------------------- |\n| 9092 | Kafka listener port |\n\n### Directories and Files\n\n| Directory / File                                             |                                         |\n| ------------------------------------------------------------ | --------------------------------------- |\n| Kafka installation directory (symlink to installed version)  | `/opt/kafka`                            |\n| Kafka configuration directory (symlink to /opt/kafka/config) | `/etc/kafka`                            |\n| Directory to store data files                                | `/var/lib/kafka/logs`                   |\n| Directory to store logs files                                | `/var/log/kafka`                        |\n| Kafka service                                                | `/usr/lib/systemd/system/kafka.service` |\n\n## Example Playbook\n\nAdd the below to a playbook to run those role against hosts belonging to the\n`kafka-nodes` group.\n\n```yaml\n- hosts: kafka-nodes\n  roles:\n    - sleighzy.kafka\n```\n\n## Linting\n\nLinting should be done using [ansible-lint].\n\n```sh\npip3 install ansible-lint --user\n\nansible-lint -c ./.ansible-lint .\n```\n\n## Testing\n\nThis module uses the [Ansible Molecule] testing framework. This test suite\ncreates a Kafka and ZooKeeper cluster consisting of three nodes running within\nDocker containers. Each container runs a different OS to test the supported\nplatforms for this Ansible role.\n\nAs per the [Molecule Installation guide] this should be done using a virtual\nenvironment. The commands below will create a Python virtual environment and\ninstall Molecule including the Docker driver.\n\n```sh\n$ python3 -m venv molecule-venv\n$ source molecule-venv/bin/activate\n(molecule-venv) $ pip3 install ansible docker \"molecule-plugins[docker]\"\n```\n\nRun playbook and tests. Linting errors need to be corrected before Molecule will\nexecute any tests. This will run all tests and then destroy the Docker\ncontainers.\n\n```sh\nmolecule test\n```\n\nThe below command can be used to run the playbook without the tests. This can be\nrun multiple times when making changes to the role, and ensuring that operations\nare idempotent.\n\n```sh\nmolecule converge\n```\n\nThe below commands can be used to just run the tests without tearing everything\ndown. The command `molecule verify` can be repeated for each test run.\n\n```sh\nmolecule create\nmolecule converge\nmolecule verify\n```\n\nTear down Molecule tests and Docker containers.\n\n```sh\nmolecule destroy\n```\n\n## License\n\n![MIT license]\n\n[ansible-lint]: https://docs.ansible.com/ansible-lint/\n[ansible molecule]: https://molecule.readthedocs.io/\n[apache kafka]: http://kafka.apache.org/\n[apache zookeeper]: https://zookeeper.apache.org/\n[lint code base]:\n  https://github.com/sleighzy/ansible-kafka/workflows/Lint%20Code%20Base/badge.svg\n[molecule]:\n  https://github.com/sleighzy/ansible-kafka/workflows/Molecule/badge.svg\n[molecule installation guide]:\n  https://molecule.readthedocs.io/en/stable/installation.html\n[mit license]: https://img.shields.io/badge/License-MIT-blue.svg\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsleighzy%2Fansible-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsleighzy%2Fansible-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsleighzy%2Fansible-kafka/lists"}