{"id":22435578,"url":"https://github.com/allanger/mc-bootstrap","last_synced_at":"2026-02-23T08:34:19.238Z","repository":{"id":218749877,"uuid":"747229230","full_name":"allanger/mc-bootstrap","owner":"allanger","description":null,"archived":false,"fork":false,"pushed_at":"2024-01-23T18:31:25.000Z","size":33,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"init","last_synced_at":"2025-07-08T08:08:36.292Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Smarty","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/allanger.png","metadata":{"files":{"readme":"README.bak","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-01-23T14:24:40.000Z","updated_at":"2024-01-23T15:08:28.000Z","dependencies_parsed_at":"2025-07-08T08:06:41.201Z","dependency_job_id":"970c0914-e9af-490c-9537-0d813d3e7731","html_url":"https://github.com/allanger/mc-bootstrap","commit_stats":null,"previous_names":["allanger/mc-bootstrap"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/allanger/mc-bootstrap","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allanger%2Fmc-bootstrap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allanger%2Fmc-bootstrap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allanger%2Fmc-bootstrap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allanger%2Fmc-bootstrap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/allanger","download_url":"https://codeload.github.com/allanger/mc-bootstrap/tar.gz/refs/heads/init","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allanger%2Fmc-bootstrap/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29740026,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-23T07:44:07.782Z","status":"ssl_error","status_checked_at":"2026-02-23T07:44:07.432Z","response_time":90,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-05T23:14:39.964Z","updated_at":"2026-02-23T08:34:19.214Z","avatar_url":"https://github.com/allanger.png","language":"Smarty","funding_links":[],"categories":[],"sub_categories":[],"readme":"# New bootstrap\n\n## What is supposed to be improved?\n\n- Flux app should be improved, so it's supporting k8s api checks\n- Bootstrapping clusters should be easier\n- Application management can be changed to have a single source of truth\n\n## Flux app\n\n### What are problems?\n\n---\n\n#### Upgrading to 1.25 without workarounds is not possible\n\nIt's not possible, because flux is installed with kustomize, that doesn't support api checks. So after PSPs are removed from API, kustomization that is installing flux is broken. \n\nCurrent workflow is to add a new `if` condition to the chart, that will check for a certain value set `podSecurityStandards` and enable/disable PSPs installation.\n\nIt requires (not a lot):\n- Change in the helm chart\n- Release\n\nBut then we need to install that version to clusters. Since we need to test the whole upgrade process, we need co create a branch in the `cluster-management-bases` and then point `$CLUSTER-management-cluster` customization to the correct brahcn of `cluster-management-bases`. Since it's a matter of testing, each iteration requires:\n\n- Helm-chart change\n- Release\n- Change to the `cluster-management-bases`\n- Change to the `$CLUSTER-management-cluster`\n\nMore precisely (in a clean way):\n- Update the helm chart\n- Release a new version\n- Edit the main flux kustomization in the `cmb`, so it' installing another flux version\n- We need to add a patch for the `GirRepository/cluster-management-fleet` so it's creating a fleet out of the correct branch, or we can edit the manifest in place, both ways requires understanding of how flux works, second requires understanding of flux annotations. And since Flux is Honeybdager's responsibillity, we shouldn't expect that knowleges from a anybody else. So everytime there somebody has a problem, it's us being triggered. \n- After cluster-management-fleet is using a correct branch, we need to update the branch that is installing `flux-app-v2` in kustomizations of `$CLUSTER-management-cluster`. Since the default `GitRepositoty/cluster-management-fleet` is created by `cmb`, and then we reference a `cmb` branch in the `$CLUSTER-management-cluster` repo, it's already a chicken-egg condition. And the easiest solution that is editing in place is vialoting GitOps principles.\n\nSince flux app is not managed as an app, it can't be configured via the `config` repo, but as I've seen it's considered a default way of managing values, so it's already a behavior that is not expected by others. And it reuires others to get deeper in our setup. \n\nSince the spread of editing the main branch of the `cmb` is huge, we can't just go ahead an replace it there, also clusters are being updated one-by-one, ones who are upgrading will have to figure out a way to upgrade using two repos for each cluster + the default `config` repo for the rest of apps, because this value (that enables/disables PSPs) must be set manually.\n\n**What is that value is doing in out chart?**\nIt's enables/disables PSP creating and RBAC for that PSP too. \n\n**When should PSPs be not installed?**\nAfter a cluster version is higher that 1.25\n\n**What is the helm solution for that?**\n\n```\n{{- if le (atoi (.Capabilities.KubeVersion.Minor | replace \"+\" \"\")) 24 }}\nkind: PodSecurityPolicy\n...\n{{- end }}\n```\n\nIf we have this change, we will be able to deploy the same version of out helm chart to both `1.24` and `1.25` wiothout any manual change. Since the expected behaviour is to have PSPs until 1.25, it's fixing this problem for us.\n\nWorkflow\n- Release it\n- Deploy it everywhere\n- We have PSPs installed without modidifying any values everywhere\n\nThen if somebody is testing the cluster upgrade process, they only need take care of other apps, no changes are required to `cmb` and `$CLUSTER-management-cluster` repos. They only touch the `config` one to set that value evrywhere.\n\n**Why won't it work?** \nBecause kustomize is not able to run API checks. Once an API check is defined in a chart, kusomize won't render that manifest at all, so we won't have PSP. \n\n**Solution?...**\nHelm is a self-efficient tool, it doesn't need to have a kustomize, because it's a templater on its own. \n\nLet's have a look at what our kustomizations are doing\n\n- It's already handle by values, we can set it via them.\n```\nimages:\n  - name: docker.io/giantswarm/fluxcd-helm-controller\n    newName: giantswarm/fluxcd-helm-controller\n  - name: docker.io/giantswarm/fluxcd-image-automation-controller\n    newName: giantswarm/fluxcd-image-automation-controller\n  - name: docker.io/giantswarm/fluxcd-image-reflector-controller\n    newName: giantswarm/fluxcd-image-reflector-controller\n  - name: docker.io/giantswarm/fluxcd-notification-controller\n    newName: giantswarm/fluxcd-notification-controller\n  - name: docker.io/giantswarm/fluxcd-source-controller\n    newName: giantswarm/fluxcd-source-controller\n  - name: docker.io/giantswarm/fluxcd-kustomize-controller\n    newName: giantswarm/kustomize-controller\n  - name: docker.io/giantswarm/k8s-jwt-to-vault-token\n    newName: giantswarm/k8s-jwt-to-vault-token\n  - name: docker.io/giantswarm/docker-kubectl\n    newName: giantswarm/docker-kubectl\n```\n\n- These are specific to `giantswarm` flux installation, and they doesn't exist in the `customer` one. Why? Because out helm chart is creating those resources with static names. If we add something like `{{ .Release.Name }}` to a name of each resource that is in this list, we won't have to run kustomizations at all, and `giantswarm` and `customer` installation will look the same. Example: https://github.com/giantswarm/flux-app/blob/faa5f89120c052b3d5850504a5ed86ab93b89b55/helm/flux-app/templates/base/clusterrole-crd-controller.yaml\n```\n- target:\n    kind: ClusterRole\n    name: crd-controller\n  patch: |-\n    - op: replace\n      path: /metadata/name\n      value: crd-controller-giantswarm\n    - op: replace\n      path: /rules/10/resourceNames/0\n      value: flux-app-pvc-psp-giantswarm\n- target:\n    kind: PodSecurityPolicy\n    name: flux-app-pvc-psp\n  patch: |-\n    - op: replace\n      path: /metadata/name\n      value: flux-app-pvc-psp-giantswarm\n- target:\n    kind: ClusterRoleBinding\n    name: cluster-reconciler\n  patch: |-\n    - op: replace\n      path: /metadata/name\n      value: cluster-reconciler-giantswarm\n- target:\n    kind: ClusterRoleBinding\n    name: crd-controller\n  patch: |-\n    - op: replace\n      path: /metadata/name\n      value: crd-controller-giantswarm\n    - op: replace\n      path: /roleRef/name\n      value: crd-controller-giantswarm\n```\n\nWhy is it better? Because kustomize won't make sure that one resource has `giantswarm` postfix in a name if it's being refenreced by another. So if we change the PSP name, but won't change it in the CR, we will have to notice it in the runtime. If this value is set on the helm level, it can be testes with helm tools, and then we can be rather certain that names are correct everywhere. Also since kustomize doesn't know anything about cluster API, it will try rendering the PSP patch everytime, and it means that this state of a file won't work on both 1.24 and 1.25. \nIf we remove the PSP patch now, it will break all the installation, because flux will start to create resources with wrong names, and there will be conflicts between two fluxes\nIf we keep it now, we will have to switch to another branch while upgrading kubernetes. It's not an expected behaviour, as I've heard from people trying to upgrade k8s. They expect the working state be always in the main branch.\n\n- It can be handled on the helm chart level, even easier if we put CRDs to templates. But since we don't want it. We also can put it to manifests directly. Or it can be added as a `helm post-renderer kustomization`, if we want to keep patchingit with kustomize\n```yaml\n- target:\n    kind: CustomResourceDefinition\n    name: \".*\"\n  patch: |-\n    - op: add\n      # https://github.com/kubernetes-sigs/kustomize/issues/1256\n      path: /metadata/annotations/kustomize.toolkit.fluxcd.io~1prune\n```\n\n**Why helm post-renderer is better than kustomize**\n\nBecause helm-postrenderer that is applying kustomizations doesn't eliminate helm features, but just adds kustomize ones, while installing helm chart with kustomization eliminates helm features. \n\n- The big one. It can be handled by helm post-renderred since I think we don't want to edit these args in the helm chart directly. If we don't mind setting args via values, I'd go with it. \n```\n- target:\n    kind: Deployment\n    name: helm-controller\n    namespace: flux-giantswarm\n  patch: |-\n    - op: replace\n      path: /spec/template/spec/containers/0/args\n      value:\n        - --events-addr=http://notification-controller.$(RUNTIME_NAMESPACE).svc\n        - --watch-all-namespaces=false\n        - --log-level=info\n        - --log-encoding=json\n        - --enable-leader-election\n        - --concurrent=12\n- target:\n    kind: Deployment\n    name: image-automation-controller\n    namespace: flux-giantswarm\n  patch: |-\n    - op: replace\n      path: /spec/template/spec/containers/0/args\n      value:\n        - --events-addr=http://notification-controller.$(RUNTIME_NAMESPACE).svc\n        - --watch-all-namespaces=false\n        - --log-level=info\n        - --log-encoding=json\n        - --enable-leader-election\n- target:\n    kind: Deployment\n    name: image-reflector-controller\n    namespace: flux-giantswarm\n  patch: |-\n    - op: replace\n      path: /spec/template/spec/containers/0/args\n      value:\n        - --events-addr=http://notification-controller.$(RUNTIME_NAMESPACE).svc\n        - --watch-all-namespaces=false\n        - --log-level=info\n        - --log-encoding=json\n        - --enable-leader-election\n- target:\n    kind: Deployment\n    name: kustomize-controller\n    namespace: flux-giantswarm\n  patch: |-\n    - op: replace\n      path: /spec/template/spec/containers/0/args\n      value:\n        - --events-addr=http://notification-controller.$(RUNTIME_NAMESPACE).svc\n        - --watch-all-namespaces=false\n        - --log-level=info\n        - --log-encoding=json\n        - --enable-leader-election\n- target:\n    kind: Deployment\n    name: notification-controller\n    namespace: flux-giantswarm\n  patch: |-\n    - op: replace\n      path: /spec/template/spec/containers/0/args\n      value:\n        - --watch-all-namespaces=false\n        - --log-level=info\n        - --log-encoding=json\n        - --enable-leader-election\n- target:\n    kind: Deployment\n    name: source-controller\n    namespace: flux-giantswarm\n  patch: |-\n    - op: replace\n      path: /spec/template/spec/containers/0/args\n      value:\n        - --events-addr=http://notification-controller.$(RUNTIME_NAMESPACE).svc\n        - --watch-all-namespaces=false\n        - --log-level=info\n        - --log-encoding=json\n        - --enable-leader-election\n        - --storage-path=/data\n        - \"--storage-adv-addr=source-controller.$(RUNTIME_NAMESPACE).svc\"\n```\n\n- It's used to break kyverno, and once it's applied kyverno will block kustomize-controller, because expcetion policies won't be installed since we're not using helm. Adding policy exception all the time would mean that flux depends on kyverno, and it means that we depend on another team. \n```\n- patch-pvc-psp.yaml\n- patch-kustomize-controller.yaml\nresources:\n```\n\n- Resource that are added to a chart, that could be handled by helm.\n```\n- resource-rbac.yaml\n- resource-refresh-vault-token.yaml\n```\n\nBut this is not a real kustomization that is being used by a \"$CLUSTER-cluster-management\", we use it by other kustomizations.\n\nWe are using kustomizations from `/bases/provider/$PROVIDER` that are in most cases are used only to create one `ConfigMap`. It also can be handled by helm, if we add one manifests to templates. \n\nSo currently the gitflow path looks like that (and don't forget about branching strategy):\n```\n$CLUSTER-management-cluster/kustomization -\u003e cmb/providers/kustomizations -\u003e cmb/flux/kustomizations\n```\n\nSo what can be done with that? \nIf we stop installing flux with `Kustomization`, and replace it with `HelmRelease`, we will be able to have all these features included out of the box more or less. \n\nSo instead of:\n```\n$CLUSTER-management-cluster/kustomization -\u003e cmb/providers/kustomizations -\u003e cmb/flux/kustomizations -\u003e GitRepository/cluster-management-fleet -\u003e Kustomization/flux\n```\n\nWe can have: \n\n```\nHelmRepository/gs-catalog -\u003e HelmRelease/flux-app -\u003e GitRepository/cluster-management-fleet -\u003e Kustomization/flux\n```\n\nIf we need to configures values, we can set them with configmaps on the `Kustomization/flux` level. And use them in `HelmRelease/flux-app`. It's what was done by me as my hiring tasl, and we can have it automated easily with no problems.\n\nAfter flux-app is managed by helmrelease instead of kustomize, upgrade of a cluster will be upgrade of a k8s version. Since templates are generated dynamically, after we upgrade a version, helm-controller should stop rendering PSPs, and there will bo no chagnes required to any of GitOps repos. \n\n---\n\n#### Flux-app refactoring\n\nSo as you could see in the previous block, it requires some changes in the flux-app helm chart.\n\nYou could see the whole list of changes here: https://github.com/giantswarm/flux-app/pull/241/\n\nThese changes are not only related to the helm-controller requirenments, but also they are supposed to make changes easier in the future. \n\n---\n\n#### How to install flux?\n\n\u003e Here the bootstrap thing is started\n\nIf we want to replace kustomize with flux, we will most probabaly apply some changes to the bootstrappging. \n\nFlux requires only several components to be installed in the cluster: for example CNI and CoreDNS.\n\nBut flux doesn't requires Kyverno and VPA to be running. Even though security team may argue, a cluster (that only contains Flux, CNI, and CoreDNS) doesn't really have anything to be secured. Aslo, taking under consideration that flux-app us managed by us, we can be more or less sure that it's secure enough to be running without kyverno for 5 minutes without security issues. \n\nCurrentlly, we don't want to edit mc-bootsrap script, we only want to isntall flux ina different way. So let's replace that part only. \n\nSince we want flux to be managed by the same flux later, we will have to install it with helm and then create a `HelmRelease` resource. \n\nCurrently, we have to kustomizations that are installing flux. \n1. CRDs\n2. Flux-app\n\nWe need to install CRDs before flux, becase flux needs to have them to be installed and start managin itself.\n\nInstead. we can install `Flux` with CRDs during the bootstrap.\n\nWhat I'm proposing might not seem easier that what we have now, but I'll explain it later.\n\n**What do we need to do?**\n\n- Install flux\n- Make flux manage itself\n\n\n**How to install flux?**\n\nI would suggest having a helmfile in the bootstrap repo, that is being synced on bootstrap\n\n```yaml\nrepositories:\n  - name: gs-catalog\n    url: https://giantswarm.github.io/giantswarm-catalog/\nreleases:\n  - name: flux-app\n    chart: gs-catalog/flux-app\n    version: $SOME_VERSION\n```\n\nAfter we sync that helmfile against a cluster, we don't have it managing itself. We will have to create a `Kustomization` that will manage a `HelmRelease`. \n\nSo there are different ways of getting there. The easiest one that wouldn't be a big problem to implement, would be adding a `HelmRelease` in the `management-cluster-bases` repo and add a kustomization that will install it.\n\nSo we have a `flux` kustomization that instead of installing flux as a kustomization, it's going to apply a `HelmRelease`.\n\nAt this point we can keep using kustomizations without breaking things. We will just replace flux `Kustomization` with `Flux` `HelmRelease`\n\n**But we are adding an extra step, aren't we?**\n\nYes, at this point we are. And it seems to be more complex that it was before.\n\nSo let's check what that kustomization that is installing `HelmRelease` is.\n\nIt should contain 3 types of resources:\n\n- HelmRepository\n- HelmRelease\n- ConfigMap with values\n- (optional) Secret with secret values\n\nWhere `HelmRepository` is always the same, :w\n\n\n2. Flux requires a lot of CRDs to be installed, that produces a chicken-egg problem\n3. (IMO) Maintaining the chart is painfull. \n\n### How to solve them? \n\n# A new bootstrap and new config management\n\n## Bootstrap\n\nCreating a cluster should not take more that ~10 minutes, and it should never be so complicated,\nthat one is constantly repeating that creating a cluster `is very challenging`, because it's not \na challenge, to manage k8s, especially for a company that is selling k8s.\n\nWhy is it a challenge now? \n\n1. Helm charts are not using helm features to reduce installation complexity\n2. Kustomize, kustomize, kustomize\n3. Bootstrapping scipts seem to never be refactored\n4. A lot of sources of configs, when it should be just one\n\nNow let's go step by step\n\n1. ---\n\nSince we don't use ArgoCD, we're actually able to use helm features\n\n- Helm can check the k8s version, and it would save use PSP upgrade pain.\n- Helm can check which APIs are available in a cluster, and it save us from bootstrapping pain\n\nAPI check vs Switch Flag\n\nCurrenrly, to upgrade to 1.25, we need to have a release with some value property set to true that wouldn't make it render PSP, and then upgrade.\n\n\u003e My guess would be that it's there because helm is no used for installing helm charts, but since we're using flux, I'm not sure.\n\nIf we add `{{- if le (atoi (.Capabilities.KubeVersion.Minor | replace \"+\" \"\")) 24 }}`, helm-controller will not try to deploy PSPs anymore, once we're on 25.\n\nThat's it, we don't need to switch anything, it's done by helm. If we need to ensure `security standards`, it's not a helm responsibillity, but kyverno's\nIf our chart is not satisfying kyverno, we need to upgrade the chart, but it's not a matter of an additional switch anymore. \n\nFlux-app has a lot of CR bundled, and because of that we have an insane bootstrap, that is installing all the CRDs to the cluster, so flux can be installed. \n\nSince flux is managing a cluster, it should come right after all the system resources are installed, and VerticalPodAutoscaled and Kyverno are not system resources\nSo flux must not depend on any of those CRs, that's why we're adding `{{- if .Capabilities.APIVersions.Has \"autoscaling.k8s.io/v1/VerticalPodAutoscaler\" }}`\n\nNow helm will check, if vpa can be installed before firing its manifests against the cluster api\n\nIf all our charts have this, they are independant but they're still using all the features that we want them to use, if they're available\n\n2. ---\n\n\u003e Kustomize is always becoming a hell, I've never seen an exception\n\nWhat to do with kustomize? It depends on what it's supposed to be doing.\nIf it's creating a resource, it should be a part of a chart. Because managing kustomization will become eventually to complicated, I suggest modifying chart every time we need to have an additional resource. \nIf it's pathcing a resource and it can't become a part of a chart, we can use post-render kustomization that is supported dy flux.\n\nBut a halm chart is a bundle that should be installable and self-sufficient. That's why it's a package manager, and not just a tempalter\n\n3. ---\nThere must be no `helm install` in bootstarp scripts, because it's unmaintaineble, compliated and absolutely not required. Since we use flux, we need to use flux to isntall helm releases. If we need charts to be installed before flux is there, we should make it easier to maintain, and though it's not part of our responsibillity, I'd just suggest a change to teams that are resposible for that. \n\nInstead of using \n```\nhelm install $RELEASE $REPO/$CHART --version $VERSION \\\n  --set $SOMETHING=$SOMETHING\n  -f $SOME_FILE\nhelm install $RELEASE $REPO/$CHART --version $VERSION \\\n  --set $SOMETHING=$SOMETHING\n  -f $SOME_FILE\nhelm install $RELEASE $REPO/$CHART --version $VERSION \\\n  --set $SOMETHING=$SOMETHING\n  -f $SOME_FILE\n```\n\nwe can use helmfile during the bootstrap\n```yaml\nrepositoies: \n  - name: $REP\n    url: $URL\nreleases:\n  - name: $RELEASE\n    chart: $REPO/$CHART\n    version: $VERSION\n    values:\n        - $PATH_TO_VALUES\n  - name: $RELEASE\n    chart: $REPO/$CHART\n    version: $VERSION\n    values:\n        - $PATH_TO_VALUES\n  - name: $RELEASE\n    chart: $REPO/$CHART\n    version: $VERSION\n    values:\n        - $PATH_TO_VALUES\n```\n\nIn case, maintainers want their releases to be also managed by flux, they have to use same name and namespaces, and it's better to use same values and versions. Then we can take them over with no troubles\n\nBut what about namespaces and flux resources? \n\nI suggest using helm here too, it's going to be two simple as heck helm charts, that will be able to handle the whole config. So let's come up with a real example of a helm file\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallanger%2Fmc-bootstrap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fallanger%2Fmc-bootstrap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallanger%2Fmc-bootstrap/lists"}