{"id":50759315,"url":"https://github.com/datarocks-ag/redpanda-provisioner","last_synced_at":"2026-06-11T08:30:35.599Z","repository":{"id":353706638,"uuid":"1162204266","full_name":"datarocks-ag/redpanda-provisioner","owner":"datarocks-ag","description":"A Go CLI tool that idempotently provisions Redpanda/Kafka resources from a YAML config file. Designed as a Docker Compose init container or Kubernetes Job.","archived":false,"fork":false,"pushed_at":"2026-04-25T05:39:00.000Z","size":72,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"develop","last_synced_at":"2026-04-25T06:25:09.355Z","etag":null,"topics":["docker","go","kafka","privisioner","redpanda"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datarocks-ag.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-20T01:25:47.000Z","updated_at":"2026-04-25T05:06:48.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/datarocks-ag/redpanda-provisioner","commit_stats":null,"previous_names":["datarocks-ag/redpanda-provisioner"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/datarocks-ag/redpanda-provisioner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datarocks-ag%2Fredpanda-provisioner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datarocks-ag%2Fredpanda-provisioner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datarocks-ag%2Fredpanda-provisioner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datarocks-ag%2Fredpanda-provisioner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datarocks-ag","download_url":"https://codeload.github.com/datarocks-ag/redpanda-provisioner/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datarocks-ag%2Fredpanda-provisioner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34190582,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","go","kafka","privisioner","redpanda"],"created_at":"2026-06-11T08:30:34.769Z","updated_at":"2026-06-11T08:30:35.583Z","avatar_url":"https://github.com/datarocks-ag.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# redpanda-provisioner\n\n[![CI](https://github.com/datarocks-ag/redpanda-provisioner/actions/workflows/ci.yaml/badge.svg)](https://github.com/datarocks-ag/redpanda-provisioner/actions/workflows/ci.yaml)\n![coverage](https://raw.githubusercontent.com/datarocks-ag/redpanda-provisioner/badges/.badges/develop/coverage.svg)\n\nA Go CLI tool that idempotently provisions Redpanda/Kafka resources from a YAML config file. Designed as a Docker Compose init container or Kubernetes Job.\n\n## Features\n\n- Idempotent provisioning of topics, schemas, SASL users, and ACLs\n- YAML config with `${VAR}` environment variable expansion\n- Configurable strategy: `update` (default) or `create` (skip existing)\n- Schema Registry support (Avro, Protobuf, JSON Schema)\n- SASL/SCRAM authentication (SCRAM-SHA-256, SCRAM-SHA-512)\n- ACL management with literal and prefixed patterns\n- Exponential backoff retry for broker and schema registry connectivity\n- Structured JSON logging via `log/slog`\n- Never deletes topics, schemas, users, or ACLs not in config\n- Never decreases partition count (warns on mismatch)\n\n## Quick Start\n\n```bash\ndocker compose up\n```\n\nThis starts Redpanda and runs the provisioner with the example config.\n\n## Configuration\n\nConnection details can be set in YAML (`broker:` and `schema_registry:`\nblocks) or via environment variables. **YAML wins when set; env vars fill in\nthe rest** — so existing env-only deployments keep working unchanged. You\ncan also embed env references inside YAML strings using `${VAR}` or\n`${VAR:-default}`.\n\n### Environment Variables\n\n| Variable | Required | Default | Description |\n|---|---|---|---|\n| `REDPANDA_BROKERS` | no | `localhost:9092` | Comma-separated broker addresses |\n| `REDPANDA_SASL_USERNAME` | no | — | SASL username |\n| `REDPANDA_SASL_PASSWORD` | no | — | SASL password |\n| `REDPANDA_SASL_MECHANISM` | no | `SCRAM-SHA-256` | SCRAM-SHA-256 or SCRAM-SHA-512 |\n| `REDPANDA_TLS_ENABLED` | no | `false` | Enable TLS (any value `strconv.ParseBool` understands) |\n| `SCHEMA_REGISTRY_URL` | no | — | Schema Registry URL (required if schemas are configured) |\n| `SCHEMA_REGISTRY_USERNAME` | no | — | HTTP basic-auth user for SR |\n| `SCHEMA_REGISTRY_PASSWORD` | no | — | HTTP basic-auth password for SR |\n| `REDPANDA_CONFIG_PATH` | no | `./config.yaml` | Path to YAML config |\n| `LOG_LEVEL` | no | `info` | Log level (debug/info/warn/error) |\n\nAll auth variables are optional (dev environments often have no auth).\n\n### YAML connection block\n\n```yaml\nbroker:\n  addresses:\n    - redpanda:9092\n  sasl:\n    mechanism: SCRAM-SHA-256\n    username: ${REDPANDA_ADMIN_USERNAME}\n    password: ${REDPANDA_ADMIN_PASSWORD}\n  tls:\n    enabled: false\n\nschema_registry:\n  url: http://redpanda:8081\n  username: ${SR_USERNAME:-}\n  password: ${SR_PASSWORD:-}\n```\n\n## Strategy\n\nControl whether existing resources are updated or skipped using the `strategy` field:\n\n- `update` (default) — create resources if missing, update if they already exist\n- `create` — create resources if missing, skip if they already exist\n\nStrategy can be set globally or per topic. Per-topic strategy overrides the global setting.\n\n```yaml\nstrategy: \"create\"              # global: skip existing resources\n\ntopics:\n  - name: orders\n    strategy: \"update\"          # override: always reconcile this topic\n    partitions: 6\n    replication_factor: 3\n```\n\n**Note:** Users, ACLs, and schemas are always upserted idempotently — strategy applies to topics only.\n\n## Environment Variable Expansion\n\nString values support `${VAR}` and `${VAR:-default}` syntax. The variable\nname must match `[A-Za-z_][A-Za-z0-9_]*` (POSIX env var rules).\n\n- `${VAR}` — replaced with the env var value. If the env var is **unset and\n  no default is provided**, config-load fails with an error pointing at the\n  offending field. This is a deliberate fail-closed design: silently leaving\n  the literal `\"${PASSWORD}\"` in a credential field used to mask itself as\n  an unrelated broker error.\n- `${VAR:-fallback}` — replaced with the env var if set, otherwise the\n  literal `fallback`.\n- `${VAR:-}` — explicit \"may be empty\" escape hatch.\n\n```yaml\nusers:\n  - username: orders-service\n    password: ${ORDERS_SERVICE_PASSWORD}    # required: load fails if unset\n\nschema_registry:\n  password: ${SR_PASSWORD:-}                # optional: empty if unset\n\nstrategy: ${STRATEGY:-update}               # default: update\n```\n\n## Provisioning Order\n\nResources are provisioned in dependency order:\n\n1. **Topics** — looked up by name; created if not found, updated if they exist\n2. **Schemas** — registered with Schema Registry (naturally idempotent); compatibility level set if specified\n3. **Users** — SASL/SCRAM users upserted via `AlterUserSCRAMs`\n4. **ACLs** — created via `CreateACLs` (idempotent in Kafka protocol)\n\n## Config Example\n\nSee [config.example.yaml](config.example.yaml) for a full example.\n\n```yaml\ntopics:\n  - name: orders\n    partitions: 6\n    replication_factor: 3\n    config:\n      retention.ms: \"259200000\"\n      cleanup.policy: delete\n      min.insync.replicas: \"2\"\n      compression.type: zstd\n\nschemas:\n  - subject: orders-value\n    type: avro                  # avro | protobuf | json\n    file: schemas/orders.avsc\n    compatibility: BACKWARD\n\nusers:\n  - username: orders-service\n    password: ${ORDERS_SERVICE_PASSWORD}\n    mechanism: SCRAM-SHA-256    # SCRAM-SHA-256 | SCRAM-SHA-512\n\nacls:\n  - principal: \"User:orders-service\"\n    operations: [write, describe]\n    resource_type: topic        # topic | group | cluster | transactional_id\n    resource_name: orders\n    pattern: literal            # literal | prefixed\n    permission: allow           # allow | deny\n```\n\n## Connection Retry\n\nOn startup, the tool retries connecting to the broker and schema registry with exponential backoff (1s initial, 30s cap, 15 retries, 5min total timeout). This handles Docker Compose startup ordering without requiring `wait-for-it` scripts.\n\n## Development\n\n```bash\nmake build            # Build binary\nmake test             # Run unit tests\nmake test-integration # Run integration tests (requires Docker)\nmake lint             # Run golangci-lint\nmake vet              # Run go vet\nmake docker           # Build Docker image\n```\n\n## Docker Compose Usage\n\n```yaml\nservices:\n  redpanda:\n    image: redpandadata/redpanda:v24.3.1\n    command:\n      - redpanda start\n      - --smp 1\n      - --memory 512M\n      - --overprovisioned\n      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092\n      - --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092\n      - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081\n    ports: [\"19092:19092\", \"18081:18081\"]\n    volumes: [redpandadata:/var/lib/redpanda/data]\n    healthcheck:\n      test: [\"CMD-SHELL\", \"rpk cluster health | grep -q 'Healthy:.*true'\"]\n      interval: 5s\n      timeout: 5s\n      retries: 20\n\n  redpanda-provisioner:\n    image: ghcr.io/datarocks-ag/redpanda-provisioner:latest\n    depends_on:\n      redpanda: { condition: service_healthy }\n    environment:\n      REDPANDA_BROKERS: redpanda:9092\n      SCHEMA_REGISTRY_URL: http://redpanda:8081\n      REDPANDA_CONFIG_PATH: /config.yaml\n    volumes:\n      - ./config.example.yaml:/config.yaml:ro\n\n  app:\n    image: your-app\n    depends_on:\n      redpanda-provisioner:\n        condition: service_completed_successfully\n\nvolumes:\n  redpandadata:\n```\n\n## Container Image\n\n```bash\ndocker pull ghcr.io/datarocks-ag/redpanda-provisioner:latest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatarocks-ag%2Fredpanda-provisioner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatarocks-ag%2Fredpanda-provisioner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatarocks-ag%2Fredpanda-provisioner/lists"}