{"id":51384985,"url":"https://github.com/cozystack/kilo-clustermesh-operator","last_synced_at":"2026-07-03T19:37:44.334Z","repository":{"id":358034392,"uuid":"1238873103","full_name":"cozystack/kilo-clustermesh-operator","owner":"cozystack","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-19T16:23:06.000Z","size":309,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-19T17:37:40.303Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cozystack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-14T14:34:32.000Z","updated_at":"2026-06-19T16:22:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/cozystack/kilo-clustermesh-operator","commit_stats":null,"previous_names":["cozystack/kilo-clustermesh-operator"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/cozystack/kilo-clustermesh-operator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cozystack%2Fkilo-clustermesh-operator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cozystack%2Fkilo-clustermesh-operator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cozystack%2Fkilo-clustermesh-operator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cozystack%2Fkilo-clustermesh-operator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cozystack","download_url":"https://codeload.github.com/cozystack/kilo-clustermesh-operator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cozystack%2Fkilo-clustermesh-operator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35099548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-03T02:00:05.635Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-03T19:37:43.851Z","updated_at":"2026-07-03T19:37:44.326Z","avatar_url":"https://github.com/cozystack.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kilo-clustermesh-operator\n\n\u003e Kubernetes operator that connects two or more clusters into a WireGuard-based mesh network using [Kilo](https://github.com/squat/kilo).\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Features](#features)\n- [Requirements](#requirements)\n- [Quick Start](#quick-start)\n- [How It Works](#how-it-works)\n- [Documentation](#documentation)\n- [Project Status](#project-status)\n- [License](#license)\n\n---\n\n## Overview\n\n`kilo-clustermesh-operator` extends Kilo's single-cluster WireGuard mesh to span multiple Kubernetes clusters. You declare a `ClusterMesh` resource that lists all participating clusters, and the operator reconciles Kilo `Peer` objects so that every node in each remote cluster becomes a peer on the local cluster — enabling direct pod-to-pod and service connectivity across clusters without a shared control plane.\n\nThe operator runs on a single cluster and reaches remote clusters via kubeconfigs stored in Kubernetes Secrets. No second operator instance is required on remote clusters.\n\n## Features\n\n- **Multi-cluster WireGuard mesh** — declarative `ClusterMesh` CRD bridges any number of clusters\n- **Fork-aware Kilo support** — accepts WireGuard IP annotations in both upstream (`\u003chost\u003e/32`) and Cozystack-patched (`\u003chost\u003e/\u003csubnet-mask\u003e`) form; normalises to host routes automatically\n- **Endpoint resolution chain** — per-node endpoint determined by priority: `clustermesh-endpoint` annotation → `force-endpoint` annotation → Node `ExternalIP` combined with `wireguardPort`; nodes with no resolvable endpoint are skipped cleanly\n- **Anchor peers** — a single per-cluster anchor `Peer` advertises the `allowedNetworks` entries that no individual node already carries (e.g. service and host-network ranges) so they are reachable across clusters\n- **Per-node host routes** — a node's `InternalIP` that falls within `allowedNetworks` is advertised as a `/32` (resp. `/128`) host route on that node's own `Peer`, so host-networked workloads (e.g. Ceph mons/OSDs) are reachable directly at each node instead of funnelling the whole range through the anchor\n- **Embedded CRD bootstrap** — the operator self-applies its CRD at startup; no separate CRD pre-install step required\n- **Safe cluster reconfiguration** — a change-watcher triggers a controlled pod restart when cluster topology or kubeconfig Secrets change, rebuilding the client registry from scratch\n- **Finalizer-based cleanup** — removing a `ClusterMesh` CR triggers deletion of all managed `Peer` objects on every cluster before the resource is released\n\n## Requirements\n\n- Kubernetes 1.28+ on every participating cluster\n- [Kilo](https://github.com/squat/kilo) installed and running on every cluster (both upstream and the Cozystack-patched build are supported)\n- Each node that participates in the mesh must expose its WireGuard UDP port on a network address reachable from every other cluster — by default port `51820`, configurable per cluster via `wireguardPort`\n- Each remote cluster's API server must be reachable from the cluster where the operator runs\n- A kubeconfig Secret for each non-local cluster, granting the operator read access to `nodes` and read/write access to `peers` on that cluster\n- Helm 3.x for chart-based installation\n\n## Quick Start\n\n### 1. Install the operator\n\nClone the repository and install with Helm:\n\n```bash\ngit clone https://github.com/cozystack/kilo-clustermesh-operator.git\ncd kilo-clustermesh-operator\nhelm install kilo-clustermesh-operator charts/kilo-clustermesh-operator \\\n  --namespace kilo-system \\\n  --create-namespace\n```\n\nContainer images are published to `ghcr.io/cozystack/kilo-clustermesh-operator` and tagged `sha-\u003cfull-commit\u003e` (e.g. `sha-43caba9978f26383593bedec79930c62e7ecead7`). Pin a specific build by overriding `image.tag` in your values file:\n\n```yaml\nimage:\n  tag: sha-\u003cfull-commit\u003e\n```\n\n### 2. Prepare remote-cluster credentials\n\nThe operator runs in a single cluster and reaches every other cluster in\na mesh through a kubeconfig stored as a `Secret` next to the\n`ClusterMesh` resource. Below is the minimal RBAC the operator needs on\neach remote cluster; nothing else is required and granting `cluster-admin`\nis **not** recommended for production.\n\nThe operator uses these permissions:\n\n- `nodes` get/list/watch — discover member nodes and read their Kilo\n  annotations\n- `nodes` patch — set `kilo.squat.ai/force-endpoint=\u003cInternalIP\u003e:\u003cport\u003e` on\n  member nodes whose endpoint cannot otherwise be derived (e.g. OpenStack\n  tenants without floating IPs)\n- `kilo.squat.ai/peers` full CRUD — publish and reconcile `Peer` objects\n  representing nodes from other clusters in the mesh\n- `apiextensions.k8s.io/customresourcedefinitions` get/create/update —\n  install the upstream Kilo `Peer` CRD on the remote cluster if it is\n  missing (so a brand-new cluster can join a mesh without manual CRD\n  bootstrap). The operator never deletes CRDs.\n\nOn every remote cluster, create a `ServiceAccount`, `ClusterRole`,\n`ClusterRoleBinding`, and a long-lived token `Secret`:\n\n```yaml\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: clustermesh-reader\n  namespace: kube-system\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\nmetadata:\n  name: kilo-clustermesh-remote\nrules:\n  - apiGroups: [\"\"]\n    resources: [nodes]\n    verbs: [get, list, watch, patch]\n  - apiGroups: [kilo.squat.ai]\n    resources: [peers]\n    verbs: [get, list, watch, create, update, patch, delete]\n  - apiGroups: [apiextensions.k8s.io]\n    resources: [customresourcedefinitions]\n    verbs: [get, create, update]\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n  name: clustermesh-reader\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: ClusterRole\n  name: kilo-clustermesh-remote\nsubjects:\n  - kind: ServiceAccount\n    name: clustermesh-reader\n    namespace: kube-system\n---\napiVersion: v1\nkind: Secret\nmetadata:\n  name: clustermesh-reader-token\n  namespace: kube-system\n  annotations:\n    kubernetes.io/service-account.name: clustermesh-reader\ntype: kubernetes.io/service-account-token\n```\n\n\u003e The annotated `Secret` of type `kubernetes.io/service-account-token`\n\u003e gives a non-expiring token. If you prefer short-lived credentials,\n\u003e swap in `kubectl create token --duration=8760h clustermesh-reader`\n\u003e and rotate before expiry; the operator only re-reads the Secret on\n\u003e reconcile, so a rotated token takes effect at the next mesh change\n\u003e or after a `flux reconcile`.\n\nBuild a kubeconfig from the token and store it as a Secret on the cluster where the operator runs:\n\n```bash\nTOKEN=$(kubectl --kubeconfig \"$REMOTE\" --namespace kube-system \\\n  get secret clustermesh-reader-token --output jsonpath='{.data.token}' | base64 --decode)\nCA=$(kubectl --kubeconfig \"$REMOTE\" --namespace kube-system \\\n  get secret clustermesh-reader-token --output jsonpath='{.data.ca\\.crt}')\nSERVER=$(kubectl --kubeconfig \"$REMOTE\" config view --minify \\\n  --output jsonpath='{.clusters[0].cluster.server}')\n\nTMP=$(mktemp); chmod 600 \"$TMP\"\ncat \u003e \"$TMP\" \u003c\u003cEOF\napiVersion: v1\nkind: Config\nclusters:\n- name: remote\n  cluster:\n    server: ${SERVER}\n    certificate-authority-data: ${CA}\nusers:\n- name: clustermesh-reader\n  user:\n    token: ${TOKEN}\ncontexts:\n- name: remote\n  context:\n    cluster: remote\n    user: clustermesh-reader\ncurrent-context: remote\nEOF\n\nkubectl --kubeconfig \"$TMP\" get nodes\nkubectl --kubeconfig \"$LOCAL\" --namespace kilo-system \\\n  create secret generic cluster-b-kubeconfig --from-file=kubeconfig=\"$TMP\"\nrm \"$TMP\"\n```\n\n### 3. Create a ClusterMesh resource\n\nThe two clusters must use **non-overlapping** pod CIDRs and WireGuard CIDRs. The example below uses distinct ranges for `cluster-a` and `cluster-b`:\n\n```yaml\napiVersion: kilo.squat.ai/v1alpha1\nkind: ClusterMesh\nmetadata:\n  name: my-mesh\n  namespace: kilo-system\nspec:\n  clusters:\n    - name: cluster-a\n      local: true\n      allowedNetworks:           # pod, WireGuard and service CIDRs, all in one flat list\n        - \"10.1.0.0/16\"\n        - \"10.200.0.0/24\"\n        - \"10.96.0.0/12\"\n      wireguardPort: 51820        # default; set explicitly if your cluster uses a different port\n    - name: cluster-b\n      kubeconfigSecretRef:\n        name: cluster-b-kubeconfig\n        key: kubeconfig\n      allowedNetworks:\n        - \"10.2.0.0/16\"\n        - \"10.200.1.0/24\"\n        - \"10.112.0.0/12\"\n      wireguardPort: 51820\n```\n\n\u003e **Warning:** Pod CIDRs, WireGuard CIDRs, and service CIDRs must not overlap between any two clusters in the same namespace. Overlapping CIDRs block reconciliation for all affected meshes.\n\u003e\n\u003e **Note:** The CRD is automatically installed by the operator at startup — you do not need to apply it separately.\n\n## How It Works\n\nOn each reconcile cycle, the operator connects to every cluster in the `ClusterMesh` spec, lists all `Node` objects, validates that each node's pod CIDR and WireGuard IP fall within the cluster's declared `allowedNetworks`, and creates or updates Kilo `Peer` objects accordingly. Nodes that fail validation or have no resolvable endpoint are skipped. Each node's `Peer` also carries any of its `InternalIP` addresses that fall within `allowedNetworks` as `/32` (resp. `/128`) host routes, so host-networked workloads are reachable directly at each node. Any `allowedNetworks` entry that no individual node already advertises (e.g. the service CIDR or host-network ranges with no node address in them) is folded into a single anchor `Peer` on every other cluster. The operator uses a finalizer to clean up all managed peers when a `ClusterMesh` resource is deleted.\n\nSee [./docs/architecture.md](./docs/architecture.md) for the full reconciliation flow and component details.\n\n\u003e **Note:** The operator watches `ClusterMesh` and `Secret` objects only — it does **not** watch `Node` objects. After changing a node annotation (endpoint, WireGuard IP, public key), trigger a reconcile manually:\n\u003e\n\u003e ```bash\n\u003e kubectl --namespace kilo-system annotate clustermesh \u003cname\u003e \\\n\u003e   reconcile-trigger=\"$(date +%s)\" --overwrite\n\u003e ```\n\n## Documentation\n\n| Page | Description |\n| --- | --- |\n| [Architecture](./docs/architecture.md) | Reconciliation flow, component internals, CRD bootstrap, change-watcher |\n| [Installation](./docs/installation.md) | Helm chart values, RBAC setup, image pinning, uninstall procedure |\n| [Configuration](./docs/configuration.md) | Full `ClusterMesh` CRD reference, field constraints, status conditions |\n| [Per-node setup](./docs/per-node-setup.md) | Endpoint resolution chain, node annotations, WireGuard IP requirements |\n| [Troubleshooting](./docs/troubleshooting.md) | Common failure modes, skip reasons, CIDR overlap, stale peers |\n| [Known Gaps](./docs/known-gaps.md) | Outstanding work and proposal divergences (for contributors) |\n\n## Project Status\n\nAlpha — the API is functional and in active use within Cozystack, but the CRD version is `v1alpha1` and breaking changes may occur before a stable release. See [docs/known-gaps.md](./docs/known-gaps.md) for outstanding work and divergences from the upstream proposal.\n\n## License\n\nCopyright 2026 The Kilo Authors. Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for the full text.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcozystack%2Fkilo-clustermesh-operator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcozystack%2Fkilo-clustermesh-operator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcozystack%2Fkilo-clustermesh-operator/lists"}