{"id":22845503,"url":"https://github.com/idealista/kafka_role","last_synced_at":"2025-10-10T20:35:19.119Z","repository":{"id":41811499,"uuid":"83357752","full_name":"idealista/kafka_role","owner":"idealista","description":"Ansible role to install Apache Kafka","archived":false,"fork":false,"pushed_at":"2025-09-03T13:17:15.000Z","size":340,"stargazers_count":18,"open_issues_count":3,"forks_count":20,"subscribers_count":26,"default_branch":"main","last_synced_at":"2025-09-03T13:26:42.491Z","etag":null,"topics":["ansible","ansible-role","apache-kafka","debian","kafka"],"latest_commit_sha":null,"homepage":"","language":"YAML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/idealista.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-02-27T21:16:29.000Z","updated_at":"2024-09-09T04:41:23.000Z","dependencies_parsed_at":"2025-04-28T16:46:46.266Z","dependency_job_id":"6c027dc7-f078-46b5-a406-8d2686e7a316","html_url":"https://github.com/idealista/kafka_role","commit_stats":null,"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"purl":"pkg:github/idealista/kafka_role","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idealista%2Fkafka_role","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idealista%2Fkafka_role/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idealista%2Fkafka_role/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idealista%2Fkafka_role/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/idealista","download_url":"https://codeload.github.com/idealista/kafka_role/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idealista%2Fkafka_role/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279005272,"owners_count":26083863,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","ansible-role","apache-kafka","debian","kafka"],"created_at":"2024-12-13T03:17:28.003Z","updated_at":"2025-10-10T20:35:19.113Z","avatar_url":"https://github.com/idealista.png","language":"YAML","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Logo](https://raw.githubusercontent.com/idealista/kafka_role/master/logo.gif)\n\n[![Build Status](https://travis-ci.com/idealista/kafka_role.svg?branch=develop)](https://travis-ci.com/idealista/kafka_role)\n[![Ansible Galaxy](https://img.shields.io/badge/galaxy-idealista.kafka--role-B62682.svg)](https://galaxy.ansible.com/idealista/kafka_role)\n\n# Kafka Ansible role\n\nThis Ansible role installs Apache Kafka in a Debian Environment. The server is installed using the sources.\n\n- [Getting Started](#getting-started)\n  - [Prerequisities](#prerequisities)\n  - [Installing](#installing)\n- [Usage](#usage)\n- [Testing](#testing)\n- [Built With](#built-with)\n- [Versioning](#versioning)\n- [Authors](#authors)\n- [License](#license)\n- [Contributing](#contributing)\n\n## Getting Started\n\nThese instructions will get you a copy of the role for your Ansible Playbook. Once launched, it will install a [Apache Kafka](https://kafka.apache.org/) distributed streaming platform in a Debian Environment.\n\n\u003e [!IMPORTANT]\n\u003e From 3.0.0 Role version, Kafka v4.0.0 is installed in KRaft mode (without Zookeeper) and requires JDK 11 or higher, additionally the Log4j 2.x is used, see the [usage](#usage) section for more details.\n\n### Prerequisities\n\nAnsible \u003e= 2.9 version installed.\n\nMolecule \u003e= 3.x.x version for testing purposes.\n\nFor testing purposes, [Molecule](https://molecule.readthedocs.io/) with [Docker](https://www.docker.com/) as driver and [Goss](https://github.com/aelsabbahy/goss) as verifier.\n\n### Installing\n\nCreate or add to your roles dependency file (e.g requirements.yml):\n\n```\n- src: idealista.kafka_role\n  version: 3.0.0\n  name: kafka_role\n```\n\nInstall the role with ansible-galaxy command:\n\n```\nansible-galaxy install -p roles -r requirements.yml -f\n```\n\nUse in a playbook:\n\n```\n---\n- hosts: someserver\n  roles:\n    - role: kafka_role\n```\n\n## Usage\n\nLook to the [defaults](defaults/main.yml) properties file to see the possible configuration properties.\n\nThis role supports Kafka in KRaft mode now and we made it the default mode, so if you were using this role before, please set the neccesary variables to keep your previous configuration working.\n\nTo maintian compatibility with previous versions of the role, if you set `kafka_use_kraft: false` the role will install Kafka in Zookeeper mode as before, and use the old templates for the configuration files.\n\nThis are an example of the minimum variables you may need to set get a working Kafka cluster in KRaft mode using the new template using the `kafka_cfg`, `broker_cfg` and `controller_cfg` variables (Check the [kafka-cfg](defaults/main/kafka-cfg.yml) for default cfg):\n\nAt broker level or host_vars level:\n\n```yml\n# Unique identifier for each Kafka node\nkafka_node_id: 1\n\n# We use the kafka_node_id to generate a unique uuid for each node, you can set the value of your choice\nkafka_node_uuid: \"{{ kafka_node_id | to_uuid | uuid_to_base64 }}\".\n\n# Controller URI for this node\nkafka_controller_uri: \"{{ kafka_node_id }}@{{ ansible_host }}:{{ kafka_controller_port }}\"\n\n# Initial controller for this node if the node acts as controller\nkafka_initial_controller: \"{{ kafka_controller_uri }}:{{ kafka_node_uuid }}\"\n```\n\nAt a general level or group_vars level:\n\n```yml\n# The use purpose in general for all the nodes, instead you can set it at host_vars level\nkafka_process_roles: \"broker,controller\"  # broker, controller, or both\n\n# A unique identifier for the Kafka cluster, you can set it to any value but it must be the same for all the nodes in the cluster\nkafka_cluster_uuid: \"{{ 'kafka_role' | to_uuid | uuid_to_base64 }}\"\n\n# List of all the controllers in the cluster with their node id and host/ip\n# An example to generate the controller quorum voters value could be:\nkafka_controller_quorum_voters: \"{{  groups['brokers'] | map('extract', hostvars, 'kafka_controller_uri') | join(',') }}\"\n\n# List of all the listeners for the cluster\n# An example to generate the listeners value could be:\nkafka_listeners: \u003e-\n  {%- for listener in kafka_listeners_list -%}\n    {{ listener.name }}://{{ listener.host }}\n  {%- if not loop.last -%},{%- endif -%}\n  {%- endfor -%}\n\n# Where kafka_listeners_list looks like:\nkafka_listeners_list:\n  - name: BROKERS\n    host: \"0.0.0.0:{{ kafka_brokers_port }}\"\n    advertised_host: \"{{ kafka_host_name }}:{{ kafka_brokers_port }}\"\n    protocol: PLAINTEXT\n  - name: CLIENTS\n    host: \"0.0.0.0:{{ kafka_clients_port }}\"\n    advertised_host: \"{{ kafka_host_name }}:{{ kafka_clients_port }}\"\n    protocol: PLAINTEXT\n  - name: CONTROLLER\n    host: \"0.0.0.0:{{ kafka_controller_port }}\"\n    advertised_host: \"{{ kafka_host_name }}:{{ kafka_controller_port }}\"\n    protocol: PLAINTEXT\n\n# Kafka inter broker and controller listener names\nkafka_inter_broker_listener_name: BROKERS\nkafka_controller_listener_names: CONTROLLER\n\n# Kafka listeners using the kafka_listeners_list variable\nkafka_advertised_listeners: \u003e-\n  {%- for listener in kafka_listeners_list -%}\n    {{ listener.name }}://{{ listener.advertised_host }}\n  {%- if not loop.last -%},{%- endif -%}\n  {%- endfor -%}\n\n# Map of listener name to security protocol using the kafka_listeners_list variable\nkafka_security_protocol_map: \u003e-\n  {%- for listener in kafka_listeners_list -%}\n    {{ listener.name }}:{{ listener.protocol }}\n  {%- if not loop.last -%},{%- endif -%}\n  {%- endfor -%},SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL\n\n# kafka controller quorum bootstrap servers using the inventory \"brokers\" group\nkafka_controller_quorum_bootstrap_servers: \"{{ groups['brokers'] | map('extract', hostvars, 'ansible_host') | product([':' + kafka_controller_port | string]) | map('join') | join(',') }}\"\n\n# If you want to customize the log4j configuration or if you are using kafka \u003c 4.0.0 to set the log4j 1.x configuration\nkafka_log4j_template_path: log4j2.yml.j2\nkafka_log4j_file_name: log4j2.yml\n```\n\n\u003e [!CAUTION]\n\u003e These were examples and should be adapted to your specific needs.\n\nAdditionally but not necessary, we recommend to set the following variables too:\n\n```yml\nkafka_xmx: \"to_your_value\"\nkafka_xms: \"to_your_value\"\n```\n\nKafka topics could be configured through the role. Just set the topics like:\n\n```yml\nkafka_topics:\n  - name: 'test'\n    partitions: '3'\n    replicas: '3'\n  - name: 'test2'\n    partitions: '3'\n    replicas: '1'\n```\n\nEnable delete topic var if you want to remove topics from the cluster.\n\nThe number of partitions can be modified but not the replicas. Please have this in mind when create the topics.\n\nAlso notice that you can't decrease the number of partitions of a created topic.\n\n\u003e [!NOTE]\n\u003e Ansible does not support generating base64 encoded UUIDs, so for this role we developed a custom filter plugin to do so. For this role it is included but, If you want to use this feature outside, you must copy the `filter_plugins` folder in this repo to your Ansible project.\n\n## Testing\n\n### Install dependencies\n\n```sh\n$ pipenv sync\n```\n\nFor more information read the [pipenv docs](ipenv-fork.readthedocs.io/en/latest/).\n\n### Testing\n\n```sh\n$ pipenv run molecule test\n```\n\n## Built With\n\n![Ansible](https://img.shields.io/badge/ansible-2.7.5.0-green.svg)\n\n## Versioning\n\nFor the versions available, see the [tags on this repository](https://github.com/idealista/kafka_role/tags).\n\nAdditionaly you can see what change in each version in the [CHANGELOG.md](CHANGELOG.md) file.\n\n## Authors\n\n- **Idealista** - *Work with* - [idealista](https://github.com/idealista)\n\nSee also the list of [contributors](https://github.com/idealista/kafka_role/contributors) who participated in this project.\n\n## License\n\n![Apache 2.0 License](https://img.shields.io/hexpm/l/plug.svg)\n\nThis project is licensed under the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license - see the [LICENSE](LICENSE) file for details.\n\n## Contributing\n\nPlease read [CONTRIBUTING.md](.github/CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidealista%2Fkafka_role","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidealista%2Fkafka_role","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidealista%2Fkafka_role/lists"}