{"id":17227319,"url":"https://github.com/stephensorriaux/ansible-kafka-admin","last_synced_at":"2025-04-04T09:08:33.316Z","repository":{"id":31716485,"uuid":"127802722","full_name":"StephenSorriaux/ansible-kafka-admin","owner":"StephenSorriaux","description":"Manage your topic's configuration (partitions, replication factor, parameters), ACLs, quotas, users and get stats, without any effort with this library. It does not use the Kafka scripts and does not require ssh connection to the remote broker.","archived":false,"fork":false,"pushed_at":"2025-02-25T14:11:03.000Z","size":579,"stargazers_count":151,"open_issues_count":11,"forks_count":46,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-03-28T08:07:22.174Z","etag":null,"topics":["ansible","consumer-groups","kafka","kafka-acl","kafka-client","kafka-quotas","kafka-topic","kafka-topics","kafka-user","library","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/StephenSorriaux.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-02T19:30:15.000Z","updated_at":"2025-02-25T14:07:27.000Z","dependencies_parsed_at":"2023-02-14T02:30:30.445Z","dependency_job_id":"cf40ffe3-2840-422b-9a8a-4c8288e43f12","html_url":"https://github.com/StephenSorriaux/ansible-kafka-admin","commit_stats":{"total_commits":140,"total_committers":15,"mean_commits":9.333333333333334,"dds":0.5071428571428571,"last_synced_commit":"edf73b08ffebaa28d3127373947fcb2974b28060"},"previous_names":[],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephenSorriaux%2Fansible-kafka-admin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephenSorriaux%2Fansible-kafka-admin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephenSorriaux%2Fansible-kafka-admin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephenSorriaux%2Fansible-kafka-admin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/StephenSorriaux","download_url":"https://codeload.github.com/StephenSorriaux/ansible-kafka-admin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247149501,"owners_count":20891954,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","consumer-groups","kafka","kafka-acl","kafka-client","kafka-quotas","kafka-topic","kafka-topics","kafka-user","library","python"],"created_at":"2024-10-15T04:18:57.551Z","updated_at":"2025-04-04T09:08:33.297Z","avatar_url":"https://github.com/StephenSorriaux.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ansible-kafka-admin ![Build Status](https://github.com/StephenSorriaux/ansible-kafka-admin/workflows/Python%20package/badge.svg?branch=master) [![Plant Tree](https://img.shields.io/badge/dynamic/json?color=brightgreen\u0026label=Plant%20Tree\u0026query=%24.total\u0026url=https%3A%2F%2Fpublic.offset.earth%2Fusers%2Ftreeware%2Ftrees)](https://plant.treeware.earth/StephenSorriaux/ansible-kafka-admin)\nA low level ansible library to manage Kafka configuration. It does not use the Kafka scripts and directly connect to Kafka and Zookeeper (if needed) to ensure resource creation. No ssh connection is needed to the remote host.\n\nIf you want to increase partitions, replication factor, change your topic's parameters, manage your ACLs and quotas without any effort, this library would be perfect for you.\n## Available Modules\n* kafka_lib (deprecated)\n* [kafka_topic](library/kafka_topic.py): Manage kafka topic\n* [kafka_topics](library/kafka_topics.py): Manage more than one topic in bulk mode\n* [kafka_acl](library/kafka_acl.py): Manage kafka acl\n* [kafka_acls](library/kafka_acls.py): Manage more than one acl in bulk mode\n* [kafka_quotas](library/kafka_quotas.py): Manage quotas on user or client-id\n* [kafka_info](library/kafka_info.py): Get infos on kafka resources\n* [kafka_stat_lag](library/kafka_stat_lag.py): get lag info on topics / consumer groups\n* [kafka_consumer_group](library/kafka_consumer_group.py): interact with kafka consumer groups\n* [kafka_user](library/kafka_user.py): Manage kafka user\n* [kafka_users](library/kafka_users.py): Manage more than one user in bulk mode\n## Requirements\nThis library uses [kafka-python](https://github.com/dpkp/kafka-python), [kazoo](https://github.com/python-zk/kazoo) and [pure-sasl](https://github.com/thobbs/pure-sasl) libraries. Install them using pip:\n```bash\npip install -r requirements.txt\n```\n**Please use only those versions as some updates might break the library.**\n\nFor now, this library is compatible with **Kafka in version 0.11.0 and more**. It is actively tested will all major versions of Kafka (0.X, 1.X, 2.X, 3.X).\n\nIt can be used with Kafka configured in PLAINTEXT, SASL_PLAINTEXT, SSL and SASL_SSL.\n\n**Concerning Zookeeper**, it is not compatible with Kerberos authentication yet, only with SSL, SASL and DIGEST authentication.\n## Need some help?\nCome see us in our Slack workspace using this [invitation link](https://join.slack.com/t/ansible-kafka-admin/shared_invite/zt-rbnfiidg-84VbMa7US66ukqkBPTlADw).\n\n## Installation\n### As a role\nAdd the following requirement in your playbook's **requirements.yml**:\n```yaml\n---\n# from GitHub, overriding the name and specifying a specific tag\n- src: https://github.com/StephenSorriaux/ansible-kafka-admin\n  name: kafka_lib\n```\n### As an ansible collection\nAdd the following requirement in your playbook's **requirements.yml**:\n```yaml\n---\n# See avalable releases for VERSION\ncollections:\n- name: https://github.com/StephenSorriaux/ansible-kafka-admin/releases/download/VERSION/StephenSorriaux-ansible_kafka_admin-VERSION.tar.gz\n```\nThen you can use it adding `StephenSorriaux.ansible_kafka_admin.` namespace prefix before any of defined modules.\n### Using Ansible Galaxy\nThe lib is available under [https://galaxy.ansible.com/StephenSorriaux/kafka-admin](https://galaxy.ansible.com/StephenSorriaux/kafka-admin).\n## Usage\n### Creating, updating, deleting topics and ACLs\n**Note**:\nZookeeper is only needed when:\n* replication-factor changed with Kafka \u003c 2.4.0\n* using kafka_quotas with Kafka \u003c 2.6.0\n* Kafka \u003c= 0.11.0\n\nSee the [`examples`](examples) folder for real-life examples.\n\nHere some examples on how to use this library:\n```yaml\n# creates two topics 'test' \u0026 'test2' with provided configuation for plaintext configured Kafka and Zookeeper\n- name: create topics\n  kafka_topics:\n    api_version: \"1.0.1\"\n    zookeeper: \"{{ hostvars['zookeeper']['ansible_eth0']['ipv4']['address'] }}:2181\"\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n    topics:\n    - name: \"test\"\n      partitions: 2\n      replica_factor: 1\n      options:\n        retention.ms: 574930\n        flush.ms: 12345\n      state: \"present\"\n    - name: \"test2\"\n      partitions: 2\n      replica_factor: 1\n      options:\n        retention.ms: 574930\n        flush.ms: 12345\n      state: \"present\"\n\n# Force reassign on topics\n- name: force reassign\n  kafka_topics:\n    api_version: \"1.0.1\"\n    zookeeper: \"{{ hostvars['zookeeper']['ansible_eth0']['ipv4']['address'] }}:2181\"\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n    topics:\n    - name: \"test\"\n      partitions: 2\n      replica_factor: 1\n      options:\n        retention.ms: 574930\n        flush.ms: 12345\n      state: \"present\"\n      force_reassign: True\n      preserve_leader: True\n    - name: \"test2\"\n      partitions: 2\n      replica_factor: 1\n      options:\n        retention.ms: 574930\n        flush.ms: 12345\n      state: \"present\"\n      force_reassign: True\n      preserve_leader: True\n\n# creates a topic 'test' with provided configuation for plaintext configured Kafka and Zookeeper\n- name: create topic\n  kafka_topic:\n    api_version: \"1.0.1\"\n    name: \"test\"\n    partitions: 2\n    replica_factor: 1\n    options:\n      retention.ms: 574930\n      flush.ms: 12345\n    state: \"present\"\n    zookeeper: \"{{ hostvars['zookeeper']['ansible_eth0']['ipv4']['address'] }}:2181\"\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n\n# same as before but re-using default value for \"flush.ms\" (thus removing the specific config for topic)\n- name: create topic\n  kafka_topic:\n    api_version: \"1.0.1\"\n    name: \"test\"\n    partitions: 2\n    replica_factor: 1\n    options:\n      retention.ms: 574930\n    state: \"present\"\n    zookeeper: \"{{ hostvars['zookeeper']['ansible_eth0']['ipv4']['address'] }}:2181\"\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n\n# from previous topic, update the number of partitions and the number of replicas. Be aware that this action can take some times to happen on Kafka\n# so be sure to set the `zookeeper_max_retries` and `zookeeper_sleep_time` parameters to avoid hitting the timeout.\n- name: update topic\n  kafka_topic:\n    api_version: \"1.0.1\"\n    name: \"test\"\n    partitions: 4\n    replica_factor: 2\n    options:\n      retention.ms: 574930\n    state: \"present\"\n    zookeeper: \"{{ hostvars['zookeeper']['ansible_eth0']['ipv4']['address'] }}:2181\"\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n\n# creates a topic for a sasl_ssl configured Kafka and plaintext Zookeeper\n- name: create topic\n  kafka_topic:\n    api_version: \"1.0.1\"\n    name: 'test'\n    partitions: 2\n    replica_factor: 1\n    options:\n      retention.ms: 574930\n      flush.ms: 12345\n    state: 'present'\n    zookeeper: \"{{ hostvars['zookeeper']['ansible_eth0']['ipv4']['address'] }}:2181\"\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n    security_protocol: 'SASL_SSL'\n    sasl_plain_username: 'username'\n    sasl_plain_password: 'password'\n    ssl_cafile: '{{ content_of_ca_cert_file_or_path_to_ca_cert_file }}'\n\n# creates a topic for a plaintext configured Kafka and a digest authentication Zookeeper\n- name: create topic\n  kafka_topic:\n    api_version: \"1.0.1\"\n    name: 'test'\n    partitions: 2\n    replica_factor: 1\n    options:\n      retention.ms: 574930\n      flush.ms: 12345\n    state: 'present'\n    zookeeper: \"{{ hostvars['zookeeper']['ansible_eth0']['ipv4']['address'] }}:2181\"\n    zookeeper_auth_scheme: \"digest\"\n    zookeeper_auth_value: \"username:password\"\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n\n# deletes topics\n- name: delete topic\n  kafka_topics:\n    api_version: \"1.0.1\"\n    zookeeper: \"{{ hostvars['zookeeper']['ansible_eth0']['ipv4']['address'] }}:2181\"\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n    topics:\n    - name: 'test'\n      state: 'absent'\n    - name: 'test2'\n      state: 'absent'\n\n# deletes a topic\n- name: delete topic\n  kafka_topic:\n    api_version: \"1.0.1\"\n    name: 'test'\n    state: 'absent'\n    zookeeper: \"{{ hostvars['zookeeper']['ansible_eth0']['ipv4']['address'] }}:2181\"\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n\n# deletes a topic using automatic api_version discovery\n- name: delete topic\n  kafka_topic:\n    name: 'test'\n    state: 'absent'\n    zookeeper: \"{{ hostvars['zookeeper']['ansible_eth0']['ipv4']['address'] }}:2181\"\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n\n# create multiple users\n- name: create users\n  kafka_users:\n    users:\n      - name: user1\n        password: changeit\n        state: present\n\n# create a single user\n- name: create user\n  kafka_user:\n    name: user1\n    password: changeit\n    state: present\n\n\n# create ACLs for all topics\n- name: create acls\n  kafka_acls:\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n    acls:\n    - acl_resource_type: 'topic'\n      name: '*'\n      acl_principal: 'User:Alice'\n      acl_operation: 'write'\n      acl_permission: 'allow'\n      # Only with kafka api \u003e= 2.0.0\n      acl_pattern_type: 'literal'\n      state: 'present'\n    - acl_resource_type: 'topic'\n      name: '*'\n      acl_principal: 'User:Bob'\n      acl_operation: 'write'\n      acl_permission: 'allow'\n      # Only with kafka api \u003e= 2.0.0\n      acl_pattern_type: 'literal'\n      state: 'present'\n\n# create an ACL for all topics\n- name: create acl\n  kafka_acl:\n    acl_resource_type: 'topic'\n    name: '*'\n    acl_principal: 'User:Alice'\n    acl_operation: 'write'\n    acl_permission: 'allow'\n    # Only with kafka api \u003e= 2.0.0\n    acl_pattern_type: 'literal'\n    state: 'present'\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n\n# delete ACLs for a single topic `test`\n- name: delete acls\n  kafka_acls:\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n    acls:\n    - acl_resource_type: 'topic'\n      name: 'test'\n      acl_principal: 'User:Bob'\n      acl_operation: 'write'\n      acl_permission: 'allow'\n      state: 'absent'\n    - acl_resource_type: 'topic'\n      name: 'test'\n      acl_principal: 'User:Alice'\n      acl_operation: 'write'\n      acl_permission: 'allow'\n      state: 'absent'\n\n# delete an ACL for a single topic `test`\n- name: delete acl\n  kafka_acl:\n    acl_resource_type: 'topic'\n    name: 'test'\n    acl_principal: 'User:Bob'\n    acl_operation: 'write'\n    acl_permission: 'allow'\n    state: 'absent'\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n\n# Ensure Quota present for users `test` and `test2`\n- name: Ensure quota for user test and test2\n  kafka_quotas:\n    entries:\n    # Applied on default-user\n    - entity:\n        user: '\u003cdefault\u003e'\n      quotas:\n        producer_byte_rate: 1044050\n        consumer_byte_rate: 1044050\n        request_percentage: 65\n    # Applied on default-client\n    - entity:\n        client: '\u003cdefault\u003e'\n      quotas:\n        producer_byte_rate: 1044050\n        consumer_byte_rate: 1044050\n        request_percentage: 65\n    - entity:\n        user: test\n      quotas:\n        producer_byte_rate: 104405\n        consumer_byte_rate: 104405\n        request_percentage: 55\n    - entity:\n        user: test2\n      quotas:\n        producer_byte_rate: 1044050\n        consumer_byte_rate: 1044050\n        request_percentage: 65\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n\n\n```\n### Getting lag statistics\nYou might want to perform an updates on your Kafka server only if no (or not much) lag is present for a specific consumer group. The following configuration will allow you to do that:\n```yaml\n# Get kafka consumers LAG statistics\n- name: Get kafka consumers LAG stats\n  kafka_stat_lag:\n    consummer_group: \"{{ consummer_group | default('pra-mirror')}}\"\n    bootstrap_servers: \"{{ ansible_ssh_host }}:9094\"\n    api_version: \"{{ kafka_api_version }}\"\n    sasl_mechanism: \"PLAIN\"\n    security_protocol: \"SASL_SSL\"\n    sasl_plain_username: \"admin\"\n    sasl_plain_password: \"{{ kafka_admin_password }}\"\n    ssl_check_hostname: False\n    ssl_cafile: \"{{ kafka_cacert | default('/etc/ssl/certs/cacert.crt') }}\"\n  register: result\n  until:  (result.msg | from_json).global_lag_count == 0\n  retries: 60\n  delay: 2\n```\nThe available statistics are:\n```json\n{\n  // a JSON object for each topic the consumer group subscribed to\n  \"topic_A\": {\n    // a JSON object for each partition\n    \"0\": {\n      \"current_offset\": 1234,\n      \"last_offset\": 1235,\n      \"lag\": 1\n    },\n    // more partitions\n  },\n  // more topics\n  // all topics lag for the consumer group\n  \"global_lag_count\": 1\n}\n```\n### Getting information about topics, brokers, consumer groups\nYou can use the `kafka_info` lib to retrieve some infornation about topics, brokers, consumer groups and ACLs.\n\nDepending on Ansible version, results will be saved under the `results` or `ansible_module_results` key (named `\u003cresults_key\u003e` in below examples).\n#### Brokers\nPlaybook:\n```yaml\n- name: get brokers information\n  kafka_info:\n    resource: \"broker\"\n    bootstrap_servers: \"{{ ansible_ssh_host }}\"\n    api_version: \"{{ kafka_api_version }}\"\n  changed_when: false\n  register: brokers\n```\n`brokers` will be:\n```json\n{\n    \"\u003cresults_key\u003e\": {\n        \"1001\": {\n            \"host\": \"172.17.0.9\",\n            \"nodeId\": 1001,\n            \"port\": 9092,\n            \"rack\": null\n        },\n        \"1002\": {\n            \"host\": \"172.17.0.10\",\n            \"nodeId\": 1002,\n            \"port\": 9092,\n            \"rack\": null\n        }\n    }\n}\n```\n#### Topics\nPlaybook:\n```yaml\n- name: get topics\n  kafka_info:\n    resource: \"topic\"\n    bootstrap_servers: \"{{ ansible_ssh_host }}\"\n    api_version: \"{{ kafka_api_version }}\"\n  register: topics\n```\n`topics` will be:\n```json\n{\n    \"\u003cresults_key\u003e\": {\n        \"test_1600292339\": {\n            \"0\": {\n                \"isr\": [\n                    1002\n                ],\n                \"leader\": 1002,\n                \"replicas\": [\n                    1002\n                ],\n                \"earliest_offset\": 0,\n                \"latest_offset\": 42,\n                \"at_min_isr\": true,\n                \"under_replicated\": false,\n                \"under_min_isr\": false,\n                \"unavailable_partition\": false\n            }\n        }\n    }\n}\n```\n#### Topics-Configuration\nPlaybook:\n```yaml\n- name: get topics configuration\n  kafka_info:\n    resource: \"topic-config\"\n    bootstrap_servers: \"{{ ansible_ssh_host }}\"\n    api_version: \"{{ kafka_api_version }}\"\n  register: topics\n```\n`topics` will be:\n```json\n{\n    \"\u003cresults_key\u003e\": {\n        \"test_1600292339\": {\n            \"retention.ms\": \"66574936\"\n        }\n    }\n}\n```\n#### Consumer groups\nPlaybook:\n```yaml\n- name: get consumer groups\n  kafka_info:\n    resource: \"consumer_group\"\n    bootstrap_servers: \"{{ ansible_ssh_host }}\"\n    api_version: \"{{ kafka_api_version }}\"\n  register: consumer_groups\n```\n`consumer_groups` will be:\n```json\n{\n    \"\u003cresults_key\u003e\": {\n        \"AWESOME_consumer_group_1607465801\": {\n            \"coordinator\": {\n                \"host\": \"172.17.0.9\",\n                \"nodeId\": 1001,\n                \"port\": 9092,\n                \"rack\": null\n            },\n            \"error_code\": 0,\n            \"group_state\": \"Empty\",\n            \"members\": {},\n            \"protocol\": \"\",\n            \"protocol_type\": \"consumer\"\n        },\n        \"AWESOME_consumer_group_1607466258\": {\n            \"coordinator\": {\n                \"host\": \"172.17.0.10\",\n                \"nodeId\": 1002,\n                \"port\": 9092,\n                \"rack\": null\n            },\n            \"error_code\": 0,\n            \"group_state\": \"Stable\",\n            \"members\": {\n                \"kafka-python-2.0.1-e5500fee-8df9-4f37-bcd7-788522a1c382\": {\n                    \"client_host\": \"/172.17.0.1\",\n                    \"client_id\": \"kafka-python-2.0.1\",\n                    \"member_assignment\": {\n                        \"assignment\": {\n                            \"test_1607465755\": [\n                                0\n                            ]\n                        },\n                        \"user_data\": \"\",\n                        \"version\": 0\n                    },\n                    \"member_metadata\": {\n                        \"subscription\": [\n                            \"test_1607465755\"\n                        ],\n                        \"user_data\": \"\",\n                        \"version\": 0\n                    }\n                }\n            },\n            \"protocol\": \"range\",\n            \"protocol_type\": \"consumer\"\n        }\n    }\n}\n```\n#### ACLs\nPlaybook:\n```yaml\n- name: get ACLs\n  kafka_info:\n    resource: \"acl\"\n    bootstrap_servers: \"{{ ansible_ssh_host }}\"\n    api_version: \"{{ kafka_api_version }}\"\n  register: acls\n```\n`acls` will be:\n```json\n{\n  \"\u003cresults_key\u003e\": {\n      \"topic\": {\n          \"*\": [\n              {\n                  \"host\": \"*\",\n                  \"operation\": \"write\",\n                  \"pattern_type\": \"literal\",\n                  \"permission_type\": \"allow\",\n                  \"principal\": \"User:Alice\",\n                  \"resource_name\": \"*\",\n                  \"resource_type\": \"topic\"\n              }\n          ],\n          \"toto\": [\n              {\n                  \"host\": \"*\",\n                  \"operation\": \"write\",\n                  \"pattern_type\": \"literal\",\n                  \"permission_type\": \"allow\",\n                  \"principal\": \"User:Alice\",\n                  \"resource_name\": \"toto\",\n                  \"resource_type\": \"topic\"\n              }\n          ]\n      }\n  },\n}\n```\n\n## Interact with kafka consumer groups\nWhen a consumer is no longer subscribed to a topic, it remains present in the consumer group and a lag is noted\nWe can use this feature to delete it effectivelly from the consumer group\n```yaml\n- name:  Delete offset for consumer group\nkafka_consumer_group:\n    consumer_group: \"{{ consumer_group | default('test_consumer')}}\"\n    topics:\n    - name: test # mandatory\n      partitions: [0, 1, 2] # Optional\n    action: delete\n    bootstrap_servers: \"{{ ansible_ssh_host }}:9094\"\n    api_version: \"{{ kafka_api_version }}\"\n    sasl_mechanism: \"PLAIN\"\n    security_protocol: \"SASL_SSL\"\n    sasl_plain_username: \"admin\"\n    sasl_plain_password: \"{{ kafka_admin_password }}\"\n    ssl_check_hostname: False\n    ssl_cafile: \"{{ kafka_cacert | default('/etc/ssl/certs/cacert.crt') }}\"\n```\n## Change kafka client configuration\nWhen using bulks modules you can have sometimes timeout.\nYou may need to tune this values:\n```yaml\n  request_timeout_ms:\n    description:\n      - 'timeout for kafka client requests'\n    default: 30000\n  connections_max_idle_ms:\n    description:\n      - 'close idle connections after'\n    default: 540000\n```\n## Using SSL\nSince SSL is requiring SSLcontext from Python, you need to use **Python 2.7.9 and superior**.\n\nIt is possible to connect to Kafka or Zookeeper using a SSL secured connection by:\n* precising the path to cacert / server cert / server key on the remote host ;\n* directly giving the cacert / server cert / server key (treated as a password) content: a tempfile will be created on the remote host and deleted before ending the module.\n\nUsing path to a cacert file:\n```yaml\ncacert_path: /path/to/my/cacert/file/on/remote/host\n\n# creates a topic for a sasl_ssl configured Kafka and plaintext Zookeeper\n- name: create topic\n  kafka_topic:\n    api_version: \"1.0.1\"\n    name: 'test'\n    partitions: 2\n    replica_factor: 1\n    options:\n      retention.ms: 574930\n      flush.ms: 12345\n    state: 'present'\n    zookeeper: \"{{ hostvars['zookeeper']['ansible_eth0']['ipv4']['address'] }}:2181\"\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n    security_protocol: 'SASL_SSL'\n    sasl_plain_username: 'username'\n    sasl_plain_password: 'password'\n    ssl_cafile: '{{ cacert_path }}'\n```\nUsing cacert file content:\n\n```yaml\ncacert_content: |\n  -----BEGIN CERTIFICATE-----\n  CERT_CONTENT\n  -----END CERTIFICATE-----\n\n# creates a topic for a sasl_ssl configured Kafka and plaintext Zookeeper\n- name: create topic\n  kafka_topic:\n    api_version: \"1.0.1\"\n    name: 'test'\n    partitions: 2\n    replica_factor: 1\n    options:\n      retention.ms: 574930\n      flush.ms: 12345\n    state: 'present'\n    zookeeper: \"{{ hostvars['zookeeper']['ansible_eth0']['ipv4']['address'] }}:2181\"\n    bootstrap_servers: \"{{ hostvars['kafka1']['ansible_eth0']['ipv4']['address'] }}:9092,{{ hostvars['kafka2']['ansible_eth0']['ipv4']['address'] }}:9092\"\n    security_protocol: 'SASL_SSL'\n    sasl_plain_username: 'username'\n    sasl_plain_password: 'password'\n    ssl_cafile: '{{ cacert_content }}'\n```\n## Python compatibility\nThis library is tested with the following versions of Python:\n* Python 2.7\n* Python 3.5\n* Python 3.6\n* Python 3.7\n* Python 3.8\n* Python 3.9\n* Python 3.10\n* Python 3.11\n* Python 3.12\n\n## Ansible compatibility\nTo date, this library is compatible with all known versions of Ansible 2.X (`ansible`/`ansible-core` 2.X).\n\n## Tests\nThis library is tested using [Molecule](https://github.com/ansible/molecule). In order to avoid code duplication, tests are defined in the `default` scenario.\n\nEach test can be run using (see pytest for more options):\n\n```\nmolecule create\nmolecule converge\n```\n\n## Contributing\nYou are very welcomed to contribute to this library, do not hesitate to submit issues and pull-requests.\n\n## License\nApache License 2.0, see [LICENSE file](LICENSE).\n\n## Treeware\n\nYou're free to use this package, but if it makes it to your production environment and would like to give some support, we would appreciate you buying the world a tree.\n\nIt’s now common knowledge that one of the best tools to tackle the climate crisis and keep our temperatures from rising above 1.5C is to \u003ca href=\"https://www.bbc.co.uk/news/science-environment-48870920\"\u003eplant trees\u003c/a\u003e. If you support this package and contribute to the Treeware forest you’ll be creating employment for local families and restoring wildlife habitats.\n\nYou can buy trees here [offset.earth/treeware](https://plant.treeware.earth/StephenSorriaux/ansible-kafka-admin)\n\nRead more about Treeware at [treeware.earth](http://treeware.earth)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephensorriaux%2Fansible-kafka-admin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstephensorriaux%2Fansible-kafka-admin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephensorriaux%2Fansible-kafka-admin/lists"}