{"id":47891058,"url":"https://github.com/osodevops/ansible_kafka_platform","last_synced_at":"2026-04-04T03:06:05.087Z","repository":{"id":337772703,"uuid":"1155124268","full_name":"osodevops/ansible_kafka_platform","owner":"osodevops","description":"Collection of Anisble roles to install KRaft Kafka Clusters","archived":false,"fork":false,"pushed_at":"2026-03-25T18:44:59.000Z","size":150,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-26T19:06:52.020Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://oso.sh/fully-managed-kafka/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/osodevops.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-11T06:32:52.000Z","updated_at":"2026-03-25T18:31:36.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/osodevops/ansible_kafka_platform","commit_stats":null,"previous_names":["osodevops/ansible_kafka_platform"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/osodevops/ansible_kafka_platform","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osodevops%2Fansible_kafka_platform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osodevops%2Fansible_kafka_platform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osodevops%2Fansible_kafka_platform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osodevops%2Fansible_kafka_platform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/osodevops","download_url":"https://codeload.github.com/osodevops/ansible_kafka_platform/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osodevops%2Fansible_kafka_platform/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31385942,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T01:22:39.193Z","status":"online","status_checked_at":"2026-04-04T02:00:07.569Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-04T03:06:00.809Z","updated_at":"2026-04-04T03:06:05.076Z","avatar_url":"https://github.com/osodevops.png","language":"Shell","readme":"# OSO Kafka Platform Collection\n\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)\n\nProduction-ready Ansible collection for deploying Apache Kafka with KRaft consensus (no ZooKeeper), TLS/SASL security, and a full monitoring stack.\n\n## Overview\n\nThis collection automates the complete lifecycle of an Apache Kafka cluster:\n\n- **Apache Kafka** with KRaft mode (controllers + brokers)\n- **TLS encryption** with self-signed CA and per-node certificates\n- **SASL/SCRAM-SHA-512** authentication\n- **Prometheus** metrics collection with Kafka-specific alert rules\n- **Grafana** dashboards for cluster, broker, controller, and JVM monitoring\n- **Kafbat UI** for web-based cluster management\n\n## Roles\n\n| Role | Description |\n|------|-------------|\n| `common` | OS prerequisites: packages, Java 21, storage, kernel tuning, NTP |\n| `ssl` | TLS certificate generation: self-signed CA, keystores, truststores |\n| `kafka_install` | Apache Kafka binary download, extraction, JMX exporter |\n| `kafka_controller` | KRaft controller deployment with quorum management |\n| `kafka_broker` | Kafka broker deployment with SSL and SASL/SCRAM |\n| `kafka_topics` | Declarative Kafka topic creation and management |\n| `kafka_ui` | Kafbat UI deployment for cluster management |\n| `prometheus` | Prometheus with Kafka scrape targets and alert rules |\n| `grafana` | Grafana with pre-built Kafka dashboards |\n| `kafka_analysis` | Read-only topic configuration analysis and reporting |\n\n## Requirements\n\n- **Ansible:** \u003e= 2.16.0\n- **Python:** \u003e= 3.10\n- **Target OS:** Ubuntu 20.04+, Debian 11+, RHEL/Rocky 8+\n- **Java:** OpenJDK 21 (installed by the `common` role)\n\n### Network Ports\n\n| Port | Service | Protocol |\n|------|---------|----------|\n| 9093 | KRaft Controller | TCP |\n| 9092 | Kafka Broker (SSL) | TCP |\n| 9094 | Kafka Broker (SASL_SSL) | TCP |\n| 7071 | JMX Exporter | TCP |\n| 9090 | Prometheus | TCP |\n| 3000 | Grafana | TCP |\n| 8080 | Kafka UI | TCP |\n\n## Installation\n\n```bash\nansible-galaxy collection install osodevops.kafka_platform\n```\n\nOr add to your `requirements.yml`:\n\n```yaml\ncollections:\n  - name: osodevops.kafka_platform\n    version: \"\u003e=1.0.0\"\n```\n\n## Quick Start\n\n1. **Install the collection:**\n\n   ```bash\n   ansible-galaxy collection install osodevops.kafka_platform\n   ```\n\n2. **Copy the example inventory:**\n\n   ```bash\n   cp -r ~/.ansible/collections/ansible_collections/osodevops/kafka_platform/examples/inventory ./inventory\n   ```\n\n3. **Customize your inventory:**\n\n   Edit `inventory/hosts.yml` with your hostnames and IPs, then update the group_vars files:\n   - `group_vars/all/main.yml` — cluster ID, quorum voters, paths\n   - `group_vars/all/security.yml` — TLS and SCRAM passwords (encrypt with `ansible-vault`)\n   - `group_vars/all/topics.yml` — topic definitions\n   - `group_vars/all/monitoring.yml` — Prometheus targets\n\n4. **Deploy the cluster:**\n\n   ```bash\n   ansible-playbook -i inventory/hosts.yml osodevops.kafka_platform.site\n   ```\n\n5. **Verify health:**\n\n   ```bash\n   ansible-playbook -i inventory/hosts.yml osodevops.kafka_platform.health_check\n   ```\n\n## Architecture\n\n```\n                    ┌─────────────────────────────────┐\n                    │       KRaft Controller Quorum    │\n                    │  ┌──────┐ ┌──────┐ ┌──────┐     │\n                    │  │ C-01 │ │ C-02 │ │ C-03 │     │\n                    │  │ :9093│ │ :9093│ │ :9093│     │\n                    │  └──┬───┘ └──┬───┘ └──┬───┘     │\n                    └─────┼────────┼────────┼─────────┘\n                          │  Metadata Replication  │\n                    ┌─────┼────────┼────────┼─────────┐\n                    │  ┌──┴───┐ ┌──┴───┐ ┌──┴───┐     │\n                    │  │ B-01 │ │ B-02 │ │ B-03 │     │\n                    │  │ :9092│ │ :9092│ │ :9092│     │\n                    │  │ :9094│ │ :9094│ │ :9094│     │\n                    │  └──┬───┘ └──┬───┘ └──┬───┘     │\n                    │     Kafka Broker Fleet           │\n                    └─────┬────────┬────────┬─────────┘\n                          │  JMX :7071      │\n                    ┌─────┴────────────────────────────┐\n                    │  ┌────────────┐  ┌────────────┐  │\n                    │  │ Prometheus │  │  Grafana   │  │\n                    │  │   :9090   │──│   :3000    │  │\n                    │  └────────────┘  │  Kafka UI  │  │\n                    │                  │   :8080    │  │\n                    │                  └────────────┘  │\n                    │        Monitoring Stack          │\n                    └──────────────────────────────────┘\n```\n\n- **Controllers** manage cluster metadata via Raft consensus (KRaft)\n- **Brokers** handle data plane operations (produce/consume)\n- **JMX Exporter** exposes Kafka metrics on each node\n- **Prometheus** scrapes metrics from all Kafka nodes\n- **Grafana** provides dashboards and alerting\n\n## Configuration\n\nSee [docs/VARIABLES.md](docs/VARIABLES.md) for the complete variable reference.\n\nKey configuration files in your inventory:\n\n| File | Purpose |\n|------|---------|\n| `group_vars/all/main.yml` | Kafka version, paths, cluster ID, quorum voters |\n| `group_vars/all/kafka.yml` | Replication, partitions, retention, networking |\n| `group_vars/all/security.yml` | TLS certificates, SCRAM users and passwords |\n| `group_vars/all/topics.yml` | Topic definitions with per-topic config |\n| `group_vars/all/monitoring.yml` | Prometheus, Grafana, Kafka UI settings |\n\n## Security\n\n### TLS/SSL\n\nThe `ssl` role generates a self-signed CA and per-node certificates:\n- PKCS12 keystores with SAN (Subject Alternative Name) support\n- Mutual TLS (mTLS) between controllers and brokers\n- Configurable certificate validity periods\n\n### SASL/SCRAM-SHA-512\n\nThe `kafka_broker` role configures SASL/SCRAM authentication:\n- Per-application SCRAM users with individual credentials\n- ACL-based authorization with deny-by-default support\n- Super user configuration for administrative access\n\n**Important:** Encrypt `security.yml` with `ansible-vault` in production.\n\n## Playbooks\n\n| Playbook | Description |\n|----------|-------------|\n| `site.yml` | Full cluster deployment (greenfield/brownfield aware) |\n| `common.yml` | OS prerequisites only |\n| `kafka_controller.yml` | Controller deployment |\n| `kafka_broker.yml` | Broker deployment |\n| `monitoring.yml` | Prometheus + Grafana + Kafka UI |\n| `create_topics.yml` | Topic provisioning |\n| `create_acls.yml` | ACL provisioning |\n| `create_scram_users.yml` | SCRAM user creation |\n| `health_check.yml` | Full cluster health verification |\n| `rolling_restart.yml` | Zero-downtime rolling restart |\n| `upgrade.yml` | Rolling Kafka version upgrade |\n| `analysis.yml` | Topic configuration analysis |\n\n## Day-2 Operations\n\n### Rolling Restart\n\n```bash\nansible-playbook -i inventory/hosts.yml osodevops.kafka_platform.rolling_restart\n```\n\nRestarts controllers first (serial:1), then brokers (serial:1), with health checks between each node.\n\n### Kafka Version Upgrade\n\n```bash\nansible-playbook -i inventory/hosts.yml osodevops.kafka_platform.upgrade \\\n  -e kafka_new_version=4.1.0\n```\n\nDownloads the new version to all nodes, then performs a rolling upgrade with zero-downtime.\n\n### Topic Management\n\n```bash\nansible-playbook -i inventory/hosts.yml osodevops.kafka_platform.create_topics\n```\n\nCreates topics defined in `group_vars/all/topics.yml`. Idempotent — skips existing topics.\n\n## Testing\n\nThis collection includes Molecule test scenarios:\n\n```bash\n# Basic single-node test (PLAINTEXT)\nmake test\n\n# Multi-node TLS + SCRAM test\nmake test-tls\n```\n\nRequirements: Docker, Python 3.10+, molecule, molecule-plugins[docker].\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Run `make lint` and `make test`\n5. Submit a pull request\n\n## License\n\nApache-2.0 — see [LICENSE](LICENSE) for details.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosodevops%2Fansible_kafka_platform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosodevops%2Fansible_kafka_platform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosodevops%2Fansible_kafka_platform/lists"}