{"id":23160572,"url":"https://github.com/olga-mir/k8s-multi-cluster","last_synced_at":"2025-08-18T02:31:43.804Z","repository":{"id":38332119,"uuid":"482691462","full_name":"olga-mir/k8s-multi-cluster","owner":"olga-mir","description":"Experiments with multi cluster patterns","archived":false,"fork":false,"pushed_at":"2024-09-28T07:05:26.000Z","size":3039,"stargazers_count":7,"open_issues_count":6,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-05T18:11:32.261Z","etag":null,"topics":["clusterapi","fluxcd","gitops","k8s","multicluster"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/olga-mir.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-18T02:39:05.000Z","updated_at":"2024-01-27T21:04:06.000Z","dependencies_parsed_at":"2023-01-31T17:31:56.544Z","dependency_job_id":"b9e1a3f3-d23b-4474-acf0-d356eba87ac3","html_url":"https://github.com/olga-mir/k8s-multi-cluster","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/olga-mir/k8s-multi-cluster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olga-mir%2Fk8s-multi-cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olga-mir%2Fk8s-multi-cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olga-mir%2Fk8s-multi-cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olga-mir%2Fk8s-multi-cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/olga-mir","download_url":"https://codeload.github.com/olga-mir/k8s-multi-cluster/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olga-mir%2Fk8s-multi-cluster/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270933632,"owners_count":24670456,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-18T02:00:08.743Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clusterapi","fluxcd","gitops","k8s","multicluster"],"created_at":"2024-12-17T23:11:23.277Z","updated_at":"2025-08-18T02:31:43.348Z","avatar_url":"https://github.com/olga-mir.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multi Cluster Management\n\nThis repository contains manifests and scripts to bootstrap clusters with [Cluster API](https://github.com/kubernetes-sigs/cluster-api). Currently only AWS clusters are supported, but more types will be added later (EKS and GCP).\n\n# Tech Stack\n\n* GitOps. Cluster(s) manifests are managed by [FluxCD](https://fluxcd.io/) and the repo structure follows [\"repo per team example\"](https://fluxcd.io/docs/guides/repository-structure/#repo-per-team).\n\n* Infrastruture provisioning. Clusters are deployed using [\"Boostrap \u0026 Pivot\"](https://cluster-api.sigs.k8s.io/clusterctl/commands/move.html) approach with initial temporary management cluster running on `kind`.\nFlux manifests are installed on each CAPI cluster by Flux running on the management cluster.\nEven though it is not a recommended approach, in this project Flux is running in read-only mode (deploy key does not have write permissions).\n\n* CNI. [cilium](https://cilium.io/). Because CNI needs to be installed during cluster bootstrap before anything else runs on the cluster it can't be managed by Flux. CAPI implemented [Cluster API Addon Provider for Helm (CAAPH)](https://github.com/kubernetes-sigs/cluster-api-addon-provider-helm) which allows to install helm charts for workload cluster during its bootstrap.\n\nThis might be possible to implement with: https://fluxcd.io/flux/components/kustomize/kustomizations/#remote-clusterscluster-api instead.\n\n# Installation\n\n## One Time Setup\n\nCreate CAPI IAM user. This will ensure the least privilege principle and give the ability to audit CAPI requests separately.\nRefer to [aws/README.md](aws/README.md) for more details what required for initial AWS setup.\n\nSetup workload clusters config as described in [config/README.md](config/README.md). Workload clusters can be set and removed on the go, they don't need to exist before running the deploy script.\n\n## Deploy\n\nDeploy permanent management cluster on AWS (using temp `kind` cluster and then pivot)\n\n```\n./scripts/deploy.sh\n```\n\nKubeconfig file path can be provided in `K8S_MULTI_KUBECONFIG` env variable, it doesn't have to be an empty kubeconfig, but it will be modified by the script adding and removing entries for clusters and contexts that it manages.\nIf this variable is not provided then a kubeconfig in this repo root location will be created.\n\nBy choice, Flux is deployed in read only mode, and therefore it can't be bootstrapped as described in official documents. This requires additional steps including creating flux secret, this is done by separate script:\n\n```\n./scripts/helpers.sh -c\n```\n\n## Adding a new cluster\n\nTo add a new cluster create config env for it by copying existing file (`./config/cluster-\u003cnum\u003e.env`) and modifying values.\nThis is intended to be manual step because the values must be provided by the end user, not guessed by the system, e.g. CIDRs.\n\n```\n./scripts/helper.sh -c cluster-02\n```\n\nThis will generate all necessary files and add the cluster to mgmt kustomization list too. Then it will be pushed to the repo (example commit from the script: https://github.com/olga-mir/k8s-multi-cluster/pull/10/commits/92ee7e094881969736ed666a0e732f073ebc53c6), where flux will apply it and capi will provision it. The `./scripts/helper.sh` is still waiting for the cluster to come up and finalize the installation.\n\non mgmt cluster:\n```\n% k get cluster -A\nNAMESPACE      NAME           PHASE          AGE   VERSION\ncluster-01     cluster-01     Provisioned    12m\ncluster-02     cluster-02     Provisioning   60s\ncluster-mgmt   cluster-mgmt   Provisioned    13m\n```\n\n# Architecture\n\n## Clusters\n\n![clusters-design](./docs/clusters.png)\n\n## Directory structure\n\n![directory-structure](./docs/directory-structure.png)\n\n# Cleanup\n\nDelete clusters in clean CAPI way:\n```\n% ./scripts/cleanup.sh\n```\nThe script will move all cluster definitions, including mgmt cluster (which at this point is hosted on the mgmt cluster itself) to the `kind` cluster and delete them in parallel.\n\nWhen CAPI way is not working for some reasons (bugs), then you need to delete AWS resources that make up the clusters to avoid charges.\n\n* delete NAT gateways.\n* release Elastic IP(s).\n* terminate EC2 instances.\n* delete VPC.\n(Resrouces usually are named `\u003ccluster-name\u003e-\u003cresource-type\u003e` pattern, e.g `mgmt-nat`, `mgmt-vpc`)\n\nAlternatively, use script `./scripts/brutal-aws-cleanup.sh` - this script deletes everything it can find (in NATs, EIPs, EC2 instances, ELBs, but not VPCs) without checking if they are related to the clusters in this project. So it is not recommended to use if there are other resources in the account.\nApparenty deleting VPC is not for the faint-hearted, check out https://github.com/isovalent/aws-delete-vpc\n\n# Future Features\n\n* Cilium cluster mesh and Gateway API\n* Private clusters\n* Runtime Extentions\n* Eliminate all bash scripts - use SDK, extentions, CAPI operator, etc. Not all of these were available when I started this project\n\n# Resources\n\n* https://www.weave.works/blog/manage-thousands-of-clusters-with-gitops-and-the-cluster-api\n* [Weaveworks' The Power of GitOps with Flux youtube playlist](https://www.youtube.com/playlist?list=PL9lTuCFNLaD3fI_g-NXWVxopnJ0adn65d). One of the videos dedicated to CAPI\n* [Cluster API Addon Provider for Helm](https://github.com/kubernetes-sigs/cluster-api-addon-provider-helm)\n* [eCHO Episode 94: Cluster API and Cilium Cluster Mesh](https://www.youtube.com/live/HVqQhMRpUR4?si=UxZkr00vD2-pTSjF) - but you'll have to be very patient with this one. There is other interesting stuff like compiling linux kernel and `virtink`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folga-mir%2Fk8s-multi-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Folga-mir%2Fk8s-multi-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folga-mir%2Fk8s-multi-cluster/lists"}