{"id":34973900,"url":"https://github.com/flurdy/doubledragon","last_synced_at":"2026-03-17T12:07:23.563Z","repository":{"id":137247743,"uuid":"378006146","full_name":"flurdy/doubledragon","owner":"flurdy","description":"Flux v2 GitOps Kubernetes cluster configuration","archived":false,"fork":false,"pushed_at":"2025-05-07T01:52:48.000Z","size":225,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-07T02:42:24.463Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flurdy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-06-18T01:52:35.000Z","updated_at":"2025-05-07T01:52:52.000Z","dependencies_parsed_at":"2024-01-29T21:56:01.111Z","dependency_job_id":"384152bb-b5fe-4d38-94c7-41bd315b68f0","html_url":"https://github.com/flurdy/doubledragon","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/flurdy/doubledragon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flurdy%2Fdoubledragon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flurdy%2Fdoubledragon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flurdy%2Fdoubledragon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flurdy%2Fdoubledragon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flurdy","download_url":"https://codeload.github.com/flurdy/doubledragon/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flurdy%2Fdoubledragon/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30623568,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-17T11:26:08.186Z","status":"ssl_error","status_checked_at":"2026-03-17T11:24:37.311Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-26T23:56:04.199Z","updated_at":"2026-03-17T12:07:23.553Z","avatar_url":"https://github.com/flurdy.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Double Dragon\nFlux v2 scaffold\n\nKubernetes cluster configuration that uses GitOps to manage state.\n\nIncludes Flux, Helm, cert-manager, Nginx Ingress and Sealed Secrets.\n\n* [fluxcd.io](https://fluxcd.io)\n* [helm.sh](https://helm.sh)\n* [kubernetes.github.io/ingress-nginx/](https://kubernetes.github.io/ingress-nginx/)\n* [github.com/bitnami-labs/sealed-secrets](https://github.com/bitnami-labs/sealed-secrets)\n* [github.com/jetstack/cert-manager](https://github.com/jetstack/cert-manager)\n* [kustomize.io](https://kustomize.io)\n\n![Double Dragon](https://static.wixstatic.com/media/cf1e64_4736286d1baa49ee99802212ada59dee~mv2.png/v1/fill/w_498,h_664,al_c,usm_0.66_1.00_0.01/cf1e64_4736286d1baa49ee99802212ada59dee~mv2.png \"arcade!\")\n\n### Contributors\n\n* Ivar Abrahamsen : [@flurdy](https://twitter.com/flurdy) : [github.com/flurdy](https://github.com/flurdy) : [eray.uk](https://eray.uk)\n\n\n### Flux version\n\n* For a Flux v1 setup, please follow the older [Lemmings repository](https://github.com/flurdy/lemmings/)\n* For a Flux v2 setup, please follow this, the [Double Dragon repository](https://github.com/flurdy/doubledragon/)\n\n### Contents\n\n1. [Introduction](#doubledragon)\n1. [Pre requisites](#contributors)\n1. [Double Dragon install](#double-dragon-install)\n   1. [Clone repository](#forkclone-repository)\n   1. [Cluster environment variables](#cluster-environment-variables)\n   1. [Install Flux CLI](#install-flux-cli)\n   1. [Bootstrap Flux](#bootstrap-flux-on-your-cluster)\n   1. [File structure](#file-structure)\n   1. [Namespaces](#namespaces)\n   1. [Sealed Secrets](#sealed-secrets)\n   1. [Nginx Ingress](#nginx-ingress)\n   1. [Cert Manager](#cert-manager)\n   1. [Container registries](#container-registries)\n1. [Applications](#applications)\n   1. [Hello world application](#hello-world-application)\n   1. [Your first application](#your-first-application)\n   1. [Automatic image updates](#automatic-image-updates)\n1. [Go wild](#go-wild)\n1. [Further information](#further-information)\n   1. [Monitoring](#monitoring)\n   1. [Auto scaling](#auto-scaling)\n   1. [Advise: Don't touch](#advise-dont-touch)\n   1. [Troubleshooting](#troubleshooting)\n   1. [Add another cluster](#add-another-cluster)\n   1. [Providers and tools](#providers-tools)\n   1. [License](#license)\n\n## Pre requisites\n\n### Kubernetes tools and cluster\n\n*     brew install kubectl\n* Create Kubernetes cluster (see [Kubernetes as a Service providers](#Kubernetes-as-a-Service) below)\n* Set up Kubernetes context (see [provider CLIs](#Kubernetes-as-a-Service) and [kubectx CLI](#Kubernetes-as-a-Service)  below)\n* Test cluster connection:\n\n      kubectl cluster-info\n\n\n\n### Github token\n\nFlux uses your [Github Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) to access your repos.\nIf you need to create a new one you need to make sure it has the required accesses ticked. For example Flux will need to access to the deploy key for the repo which require *Admin* access.\n\nIn your dotfiles make sure you expose it as `GITHUB_TOKEN`.\nAt the same time set the `GITHUB_USER` env-var to your github username.\n(Nudge: [direnv](https://direnv.net/))\n\n## Double Dragon install\n\n---\n### Fork/Clone repository\n\n* Initialize an empty Double Dragon repo for your setup.\n\n      git clone git@github.com:flurdy/doubledragon.git;\n      mkdir doubledragon-fleet;\n      cp doubledragon/README.md doubledragon-fleet/;\n      cp doubledragon/LICENSE doubledragon-fleet/;\n      cd doubledragon-fleet;\n      git init;\n      git add README.md LICENSE;\n      git commit -m \"Starting our double dragon fleet\";\n\n  Replace _doubledragon-fleet_ with whatever you want to call your repository.\n\n  You may wish use the original `doubledragon` repo to compare.\n\n* Create a private github repository\n\n  Manually create a private `doubledragon-fleet` repo via [github.com](https://github.com)\n  or with the [Github CLI](https://cli.github.com/)\n\n      brew install gh;\n      gh auth login;\n      gh repo create --private doubledragon-fleet -r origin -s .\n\n  Note, the CLI for some reason does not like if Github PAT env-var is set so you may have to temporarily unset when using it.\n\n   In Bash:\n\n      unset GITHUB_TOKEN\n\n   In Fish:\n\n      set -e GITHUB_TOKEN\n\n  Make sure you set the `GITHUB_TOKEN` env-var again afterwards.\n\n* And push your local repo to the github repo\n\n      git push -u origin main\n\n* Edit the `README.md` as you see fit.\n\n* Edit the `LICENSE` as you see fit.\n\n* Note, Flux can also talk to Bitbucket, Gitlab, Github Enterprise and self-hosted git repositories\n\n### Cluster environment variables\n\n* Lets temporarily add the repo and cluster name as environment variables so that most commands in this howto can be copy-pasted directly\n\n  In Bash:\n\n      export DOUBLEDRAGON_REPO=doubledragon-fleet;\n      export DOUBLEDRAGON_NAME=doubledragon;\n      export DOUBLEDRAGON_CLUSTER=doubledragon-01\n\n  In Fish:\n\n      set -x DOUBLEDRAGON_REPO doubledragon-fleet;\n      set -x DOUBLEDRAGON_NAME doubledragon;\n      set -x DOUBLEDRAGON_CLUSTER doubledragon-01\n\n  Replace _doubledragon-fleet, doubledragon, doubledragon-01_ with whatever you decide to call your repository and cluster\n\n### Install Flux CLI\n\n*\n      brew install fluxcd/tap/flux\n\n* Test if Flux ir ready to be installed on your cluster\n\n      flux check --pre\n\n### Bootstrap Flux on your cluster\n\n    flux bootstrap github \\\n      --token-auth=false \\\n      --components-extra=image-reflector-controller,image-automation-controller \\\n      --owner=$GITHUB_USER \\\n      --repository=$DOUBLEDRAGON_REPO \\\n      --branch=main \\\n      --path=./clusters/$DOUBLEDRAGON_CLUSTER \\\n      --read-write-key=true \\\n      --personal\n\n* This will create a github repo set in `$DOUBLEDRAGON_REPO` if it does not already exist.\n   And names your initial cluster as set in `$DOUBLEDRAGON_CLUSTER`.\n\n* Update your local repo with the _origin_ changes\n\n      git pull\n\n### File structure\n\nUnlike Flux v1 which was a simpler one repo per cluster,\nFlux v2 is more flexible with potentially many clusters per repo and more abstractions if desired.\n\nFlux v2 also prefer to use [Kustomize](https://kustomize.io) for templating,\nbut you do not have to use it.\n\nFlux can act directly in your cluster,\nbut this setup does everything via config files and git.\nThat way we have a replayable paper trail.\n\nSo a Flux repo may look like this at the start:\n\n    |-- apps\n    |   |-- base\n    |   |-- overlays\n    |   |   |-- doubledragon\n    |-- clusters\n    |   |-- doubledragon-01\n    |-- infrastructure\n    |   |-- sources\n\n\n* `apps/base` is where you define your apps; deployments, services, etc.\n* `apps/overlays/doubledragon` is where you choose which apps a cluster has,\n  and any customization specific to that cluster.\n* `cluster/doubledragon-01` with links to apps and infrastructure active in your specific cluster.\n* `infrastructure/sources` where to find images from registries, Helm repos, etc.\n\n* Create some of these folders:\n\n      mkdir -p apps/base;\n      mkdir -p apps/overlays/$DOUBLEDRAGON_NAME;\n      mkdir -p infrastructure/sources\n\n* Check the file structure\n\n      tree\n\n\n### Namespaces\n\n* Lets separate some of our resources into two namespaces.\n\n  You may go with further specific namespaces if you prefer.\n\n  (For some reason _kustomize_ does not let you add several namespaces\n  in one _kustomization_ so we will add plain files to the cluster)\n\n* Edit `clusters/$DOUBLEDRAGON_CLUSTER/namespaces.yaml`\n\n      apiVersion: v1\n      kind: Namespace\n      metadata:\n        name: infrastructure\n      ---\n      apiVersion: v1\n      kind: Namespace\n      metadata:\n        name: apps\n\n* Push to _Flux_\n\n      git add clusters/$DOUBLEDRAGON_CLUSTER/namespaces.yaml;\n      git commit -m \"Namespaces\";\n      git push\n\n### Sealed Secrets\n\nSafely store encrypted secrets in the git repository.\n\nThere are several alternative encrypted secrets solutions, such as [Mozilla's SOPS](https://fluxcd.io/flux/guides/mozilla-sops/),\nbut Sealed Secrets works well for me.\n\n* Add a Helm repository for Sealed Secrets\n\n      flux create source helm sealed-secrets-source \\\n        --interval=1h \\\n        --namespace=infrastructure \\\n        --url=https://bitnami-labs.github.io/sealed-secrets \\\n        --export \u003e infrastructure/sources/sealed-secrets-source.yaml\n\n* Install Helm chart\n\n      mkdir -p infrastructure/sealed-secrets;\n\n      flux create helmrelease sealed-secrets \\\n        --interval=1h \\\n        --release-name=sealed-secrets-controller \\\n        --target-namespace=infrastructure \\\n        --source=HelmRepository/sealed-secrets-source \\\n        --chart=sealed-secrets \\\n        --chart-version=\"\u003e=1.15.0-0\" \\\n        --crds=CreateReplace \\\n        --export \u003e infrastructure/sealed-secrets/sealed-secrets.yaml\n\n* Add to git so Flux can act on it\n\n      git add infrastructure/sources/sealed-secrets-source.yaml \\\n        infrastructure/sealed-secrets/sealed-secrets.yaml;\n      git commit -m \"Added Sealed Secrets\"\n\n* Next we need to add simple _Kustomization_ files that activates Sealed Secrets for our cluster.\nThese will be very simple for now, later on they will be more elaborate and helpful\n\n\n* Edit   `infrastructure/sealed-secrets/kustomization.yaml`\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      resources:\n      - sealed-secrets.yaml\n\n* Edit   `infrastructure/sources/kustomization.yaml`\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      resources:\n      - sealed-secrets-source.yaml\n\n* Append to `infrastructure/kustomization.yaml`\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      resources:\n      - sources\n      - sealed-secrets\n\n* Create a pollable link in our cluster for all _infrastructure_:\n\n      flux create kustomization infrastructure \\\n        --target-namespace=infrastructure \\\n        --source=flux-system \\\n        --path=\"./infrastructure\" \\\n        --prune=true \\\n        --interval=10m \\\n        --export \u003e clusters/$DOUBLEDRAGON_CLUSTER/infrastructure.yaml\n\n\n* Push to git\n\n      git add infrastructure/sources/kustomization.yaml;\n      git add infrastructure/sealed-secrets/kustomization.yaml;\n      git add infrastructure/kustomization.yaml;\n      git add clusters/$DOUBLEDRAGON_CLUSTER/infrastructure.yaml;\n      git commit -m \"Activated Sealed Secrets\";\n      git push\n\n* Flux should pick this up and install the Helm chart for Sealed Secrets\n\n#### Using Sealed Secrets\n\n* Install `kubeseal` CLI\n\n      brew install kubeseal\n\n* Retrieve public key from this cluster\n\n      mkdir -p clusters/$DOUBLEDRAGON_CLUSTER/secrets;\n\n      kubeseal --fetch-cert \\\n        --controller-name=sealed-secrets-controller \\\n        --controller-namespace=infrastructure \\\n        \u003e clusters/$DOUBLEDRAGON_CLUSTER/secrets/sealed-secrets-cert.pem\n\n  * Some cluster setups may block access to your sealed-secrets-controller,\ne.g. a GKE cluster.\n\n     So instead we can temporarily proxy that locally like this:\n\n        kubectl --namespace infrastructure port-forward \\\n          service/sealed-secrets-controller 8081:8080\n\n  * And use `curl` to download the certificate instead:\n\n        curl localhost:8081/v1/cert.pem \\\n          \u003e clusters/$DOUBLEDRAGON_CLUSTER/secrets/sealed-secrets-cert.pem\n\n* Add it to source control\n\n      git add clusters/$DOUBLEDRAGON_CLUSTER/secrets/sealed-secrets-cert.pem;\n      git commit -m \"Sealed Secret public key\";\n      git push\n\n### Nginx Ingress\n\n* Add a Helm repository\n\n      flux create source helm ingress-nginx-source \\\n        --interval=1h \\\n        --namespace=infrastructure \\\n        --url=https://kubernetes.github.io/ingress-nginx \\\n        --export \u003e infrastructure/sources/ingress-nginx-source.yaml\n\n* Append it to the exiting sources kustomization `infrastructure/sources/kustomization.yaml`\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      resources:\n      - sealed-secrets-source.yaml\n      - ingress-nginx-source.yaml\n\n* Install the ingress controller with the Helm chart\n\n      mkdir -p infrastructure/ingress-nginx;\n\n      flux create helmrelease ingress-nginx \\\n        --interval=1h \\\n        --release-name=ingress-nginx \\\n        --target-namespace=apps \\\n        --namespace=infrastructure \\\n        --source=HelmRepository/ingress-nginx-source \\\n        --chart=ingress-nginx \\\n        --chart-version=\"\u003e=1.0-4\" \\\n        --crds=CreateReplace \\\n        --export \u003e infrastructure/ingress-nginx/ingress-nginx.yaml\n\n* Add kustomization `infrastructure/ingress-nginx/kustomization.yaml`\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      resources:\n      - ingress-nginx.yaml\n\n\n* Append it to `infrastructure/kustomization.yaml`\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      resources:\n      - sources\n      - sealed-secrets\n      - ingress-nginx\n\n* Add to git and push\n\n      git add infrastructure/sources/ingress-nginx-source.yaml;\n      git add infrastructure/sources/kustomization.yaml;\n      git add infrastructure/ingress-nginx/ingress-nginx.yaml;\n      git add infrastructure/ingress-nginx/kustomization.yaml;\n      git add infrastructure/kustomization.yaml;\n      git commit -m \"Nginx Ingress\";\n      git push\n\n### Cert manager\n#### Install cert-manager\n\n* Add a Jetstack source repo\n\n      flux create source helm jetstack-source \\\n        --interval=1h \\\n        --namespace=infrastructure \\\n        --url=https://charts.jetstack.io \\\n        --export \u003e infrastructure/sources/jetstack-source.yaml\n\n  * Append it to the exiting sources kustomization `infrastructure/sources/kustomization.yaml`\n\n        apiVersion: kustomize.config.k8s.io/v1beta1\n        kind: Kustomization\n        resources:\n        - sealed-secrets-source.yaml\n        - ingress-nginx-source.yaml\n        - jetstack-source.yaml\n\n* Define Cert Manager Helm properties\n\n\n      mkdir infrastructure/cert-manager;\n\n  * Edit `infrastructure/cert-manager/values.yaml`\n\n        crds:\n          enabled: true\n        \n\n* Install Cert Manager Helm\n\n      flux create helmrelease cert-manager \\\n        --chart=cert-manager \\\n        --source=HelmRepository/jetstack-source.infrastructure \\\n        --release-name=cert-manager \\\n        --namespace=infrastructure \\  \n        --target-namespace cert-manager \\\n        --create-target-namespace \\\n        --chart-version=\"\u003e=1.12.0\" \\\n        --interval=1h \\\n        --values=infrastructure/cert-manager/values.yaml \\\n        --export \u003e infrastructure/cert-manager/cert-manager.yaml\n\n\n  * Create kustomization `infrastructure/cert-manager/kustomization.yaml`\n\n        apiVersion: kustomize.config.k8s.io/v1beta1\n        kind: Kustomization\n        resources:\n        - cert-manager.yaml\n\n  * Append it to infrastructure kustomization `infrastructure/kustomization.yaml`\n\n        apiVersion: kustomize.config.k8s.io/v1beta1\n        kind: Kustomization\n        resources:\n        - sources\n        - sealed-secrets\n        - ingress-nginx\n        - cert-manager\n\n* Add to repo and push\n\n      git add infrastructure/sources/jetstack-source.yaml;\n      git add infrastructure/sources/kustomization.yaml;\n      git add infrastructure/cert-manager/cert-manager.yaml;\n      git add infrastructure/cert-manager/kustomization.yaml;\n      git add infrastructure/kustomization.yaml;\n      git commit -m \"Cert-manager\";\n      git push\n\n* Verify Cert manager works\n\n  * Install the cert-manager CLI\n\n    Optional but handy\n\n        brew install cmctl\n\n  * Verify\n\n        cmctl check api\n\n    Hopefully that will return \"`The cert-manager API is ready`\"\n\n\n#### Certificate issuers\n\nLets create a _staging_ and _production_ certificate issuers with [Lets Encrypt](https://letsencrypt.org/), so that testing in _staging_ does not flood the _prod_ instance.\n\n    mkdir -p clusters/$DOUBLEDRAGON_CLUSTER/certificate-issuers\n\n* Create and edit the staging issuer at\n\n  `clusters/$DOUBLEDRAGON_CLUSTER/certificate-issuers/letsencrypt-issuer-staging.yaml`\n\n      apiVersion: cert-manager.io/v1\n      kind: ClusterIssuer\n      metadata:\n        name: letsencrypt-staging\n      spec:\n        acme:\n          server: https://acme-staging-v02.api.letsencrypt.org/directory\n          email: youremail@example.com\n          privateKeySecretRef:\n            name: letsencrypt-staging-secret\n          solvers:\n          - http01:\n              ingress:\n                class: nginx\n\n* Replace `youremail@example.com` with an email address you have access to\n\n* Add to _flux_ and watch till active\n\n      git add clusters/$DOUBLEDRAGON_CLUSTER/certificate-issuers/letsencrypt-issuer-staging.yaml;\n      git commit -m \"Staging issuer\";\n      git push;\n      kubectl get clusterissuer -A --watch\n\n* Secure an app\n\n  I.e. add a TLS certificate to an ingress.\n\n  \u003e [!NOTE] \n  \u003e This step may have to wait until you add your own apps later on in the tutorial.\n\n  * Edit your app's _ingress_ `apps/base/someapp/ingress.yaml`\n\n    Add the annotation and _tls_ sections\n\n        apiVersion: networking.k8s.io/v1\n        kind: Ingress\n        metadata:\n          annotations:\n            cert-manager.io/cluster-issuer: letsencrypt-staging\n          name: someapp-ingress\n          namespace: apps\n        spec:\n          rules:\n          - host: someapp.example.com\n            http:\n              paths:\n              - pathType: Prefix\n                path: /\n                backend:\n                  service:\n                    name: someapp-service\n                    port:\n                      number: 80\n          tls:\n          - hosts:\n            - someapp.example.com\n            secretName: someapp-cert-staging\n\n  * Add to git\n\n        git add apps/base/someapp/ingress.yaml;\n        git commit -m \"Secured someapp\";\n        git push;\n        kubectl get ingress -n apps --watch\n\n    Soon the `someapp.example.com` line will show 443 as available port. It all ok.\n\n    Note, your browser will throw a warning when accessing this site\n    as the certificate for staging is not signed. Unlike prod.\n\n* Now lets add a prod issuer, create and edit\n\n  `clusters/$DOUBLEDRAGON_CLUSTER/certificate-issuers/letsencrypt-issuer-prod.yaml`\n\n      apiVersion: cert-manager.io/v1\n      kind: ClusterIssuer\n      metadata:\n        name: letsencrypt-prod\n      spec:\n        acme:\n          server: https://acme-v02.api.letsencrypt.org/directory\n          email: youremail@example.com\n          privateKeySecretRef:\n            name: letsencrypt-prod-secret\n          solvers:\n          - http01:\n              ingress:\n                class: nginx\n\n* Add to _flux_ and watch till active\n\n      git add clusters/$DOUBLEDRAGON_CLUSTER/certificate-issuers/letsencrypt-issuer-prod.yaml;\n      git commit -m \"Prod issuer\";\n      git push;\n      kubectl get clusterissuer -A --watch\n\n* Update the certificate for your app\n\n  \u003e [!NOTE] \n  \u003e Again, only after you have added your own apps later on in the tutorial.\n\n  Change the `cluster-issuer` annotation and `secretName` in\n\n  `apps/base/someapp/ingress.yaml`\n\n      apiVersion: networking.k8s.io/v1\n      kind: Ingress\n      metadata:\n        annotations:\n          cert-manager.io/cluster-issuer: letsencrypt-prod\n        name: someapp-ingress\n        namespace: apps\n      spec:\n        rules:\n        - host: someapp.example.com\n          http:\n            paths:\n            - pathType: Prefix\n              path: /\n              backend:\n                service:\n                  name: someapp-service\n                  port:\n                    number: 80\n        tls:\n        - hosts:\n          - someapp.example.com\n          secretName: someapp-cert-prod\n\n  * Push and check when the certificate change goes live\n\n        git add apps/base/someapp/ingress.yaml;\n        git commit -m \"Secured someapp with prod cert\";\n        git push;\n        kubectl describe ingress someapp-ingress -n apps --watch\n        \n### Container Registries\n\nTo access private Docker container image repositories\nwe need to setup some more sources, image sources.\nAnd some secrets to access those.\n\n\n* Please follow [flurdy's 'kubernetes-docker-registry guide'](https://flurdy.com/docs/kubernetes/registry/kubernetes-docker-registry.html) for your relevant registries.\n\n#### GCR Google Container Registry\n\n\u003e[!WARNING]\n\u003e GCR has been deprecated and replaced by Google Artifact Registry.\n\u003e This is only as an example of how you would seal and add any registry to your cluster\n\n* For example if you needed [GCR](https://cloud.google.com/container-registry/),\n  and have followed the guide above and got a `gcr-registry.yml` file (or `.yaml`),\n\n  and maybe the initial `gcp-service-account.json` source as well.\n\n* Make sure the raw secrets do not get added to *git* by accident\n\n      echo gcp-service-account.json \u003e\u003e .gitignore;\n      echo gcr-registry.yml \u003e\u003e .gitignore;\n      git add .gitignore\n\n* Seal the secrets\n\n      mkdir -p clusters/$DOUBLEDRAGON_CLUSTER/registries/apps;\n      mkdir -p clusters/$DOUBLEDRAGON_CLUSTER/registries/infrastructure\n\n  A registry secret for both the _apps_ namespace\n\n      kubeseal --format=yaml --namespace=apps \\\n      --cert=clusters/$DOUBLEDRAGON_CLUSTER/secrets/sealed-secrets-cert.pem \\\n      \u003c gcr-registry.yml \\\n      \u003e clusters/$DOUBLEDRAGON_CLUSTER/registries/apps/sealed-gcr-registry.yaml\n\n   and the _infrastructure_ namespace\n\n      kubeseal --format=yaml --namespace=infrastructure \\\n      --cert=clusters/$DOUBLEDRAGON_CLUSTER/secrets/sealed-secrets-cert.pem \\\n      \u003c gcr-registry.yml \\\n      \u003e clusters/$DOUBLEDRAGON_CLUSTER/registries/infrastructure/sealed-gcr-registry.yaml\n\n* Add the secrets to Flux\n\n      git add clusters/$DOUBLEDRAGON_CLUSTER/registries/apps/sealed-gcr-registry.yaml;\n      git add clusters/$DOUBLEDRAGON_CLUSTER/registries/infrastructure/sealed-gcr-registry.yaml;\n      git commit -m \"GCR registry\";\n      git push\n\n   You may later need more for other and future namespaces, e.g. `default` and `flux-system`\n\n* Remove `gcr-registry.yml` (and `gcp-service-account.json`)\n\n  Later on when you have tested the registry by confirming that the cluster can download actual deployment images for your apps,\n  you should delete the unencrypted registry files\n\n      rm gcr-registry.yml gcp-service-account.json\n\n\n* Lets set up repo image scanning\n\n  To check when a new repo tag and image has been added to a registry.\n\n  For example if you have an app that stores its images in a private repo like _GCR_.\n\n  Otherwise you can wait to do this step later.\n\n      flux create image repository someapp-source \\\n      --image=ghcr.io/someorg/someuser/somerepo \\\n      --interval=5m \\\n      --namespace=infrastructure \\\n      --secret-ref=gcr-registry \\\n      --export \u003e infrastructure/sources/someapp-source.yaml\n\n  * (Change _somerepo_ to your app name.\n  And use the correct GCR image path.)\n\n  * This example refers to the `gcr-registry` sealed secret\n\n  * Append this to the YAML in `infrastructure/sources/someapp-source.yaml`:\n\n        accessFrom:\n          namespaceSelectors:\n            - matchLabels:\n                kubernetes.io/metadata.name: apps\n\n    Note, indentation is under `spec`.\n\n* Note, for _GCR_ there is the alternative option of a more secure short-lived _acccess token_ instead.\n\n  This can be done with Flux. [You need to set up a cronjob to refresh it](https://fluxcd.io/flux/guides/cron-job-image-auth/).\n\n* Add these to `infrastructure/sources/kustomization.yaml`\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      resources:\n      ...\n      - someapp-source.yaml\n\n* Add to git/flux\n\n      git add infrastructure/sources/someapp-source.yaml;\n      git add infrastructure/sources/kustomization.yaml;\n      git commit -m \"Sources for someapp\"\n      git push\n\n## Applications\n\n---\n## Hello world application\n\nLets create a Hello World app.\n\n### Base layer\n\n* First lets create a _base layer_\n\n      mkdir -p apps/base/hello\n\n* And an initial deployment yaml for an _Hello_ app\n\n      kubectl create deployment hello-deployment \\\n      --image=nginxdemos/hello:0.3 \\\n      --namespace=apps \\\n      --dry-run=client -o yaml \\\n      \u003e apps/base/hello/deployment.yaml\n\n* Lets prune the output a bit: `apps/base/hello/deployment.yaml`,\nand change the app labels to just `hello`\n\n      apiVersion: apps/v1\n      kind: Deployment\n      metadata:\n        labels:\n          app: hello\n        name: hello-deployment\n        namespace: apps\n      spec:\n        replicas: 1\n        selector:\n          matchLabels:\n            app: hello\n        template:\n          metadata:\n            labels:\n              app: hello\n          spec:\n            containers:\n              - image: nginxdemos/hello:0.3\n                name: hello\n\n* Add a service at `apps/base/hello/service.yaml`\n\n      apiVersion: v1\n      kind: Service\n      metadata:\n        name: hello-service\n        namespace: apps\n      spec:\n        selector:\n          app: hello\n        ports:\n          - protocol: TCP\n            port: 80\n            targetPort: 80\n\n* And an ingress at `apps/base/hello/ingress.yaml`\n\n      apiVersion: networking.k8s.io/v1\n      kind: Ingress\n      metadata:\n        name: hello-ingress\n        namespace: apps\n        annotations:\n          kubernetes.io/ingress.class: nginx\n      spec:\n        rules:\n          - host: hello.example.com\n            http:\n              paths:\n                - path: /\n                  pathType: Prefix\n                  backend:\n                    service:\n                      name: hello-service\n                      port:\n                        number: 80\n\n* Bundle these in `apps/base/hello/kustomization.yaml`\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      namespace: apps\n      resources:\n        - deployment.yaml\n        - service.yaml\n        - ingress.yaml\n\n### Hello app overlay\n\n     mkdir -p apps/overlays/$DOUBLEDRAGON_NAME/hello\n\n* Edit `apps/overlays/$DOUBLEDRAGON_NAME/hello/kustomization.yaml`\n\n  In more complicated apps this may have some overrides but for now very simple.\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      namespace: apps\n      bases:\n        - ../../../base/hello\n\n\n* Edit `apps/overlays/$DOUBLEDRAGON_NAME/kustomization.yaml`\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      namespace: apps\n      bases:\n        - hello\n\n* Add it all to the repo\n\n      git add apps/base/hello/deployment.yaml;\n      git add apps/base/hello/service.yaml;\n      git add apps/base/hello/ingress.yaml;\n      git add apps/base/hello/kustomization.yaml;\n      git add apps/overlays/$DOUBLEDRAGON_NAME/hello/kustomization.yaml;\n      git add apps/overlays/$DOUBLEDRAGON_NAME/kustomization.yaml;\n      git commit -m \"Hello app files\";\n      git push\n\n### Add Hello app to cluster\n\n* Create a _kustomization_ for all apps in the overlay\n\n      flux create kustomization apps \\\n        --target-namespace=apps \\\n        --source=flux-system \\\n        --path=\"./apps/overlays/$DOUBLEDRAGON_NAME\" \\\n        --depends-on=infrastructure \\\n        --prune=true \\\n        --interval=10m \\\n        --export \u003e clusters/$DOUBLEDRAGON_CLUSTER/apps.yaml\n\n* Add update the repo\n\n      git add clusters/$DOUBLEDRAGON_CLUSTER/apps.yaml;\n      git commit -m \"Adding apps to the cluster\";\n      git push\n### Test Hello\n\n* Find the ingress controller's `External IP`\n\n      kubectl get services -n apps ingress-nginx-controller\n\n* Use `curl` to resolve the URL. Replace `11.22.33.44` with the external IP,\n  and `lynx` to view it\n\n      curl -H \"Host: hello.example.com\" \\\n        --resolve hello.example.com:80:11.22.33.44 \\\n        --resolve hello.example.com:443:11.22.33.44 \\\n        http://hello.example.com | lynx -stdin\n\n* This should show a basic hello world page, with an Nginx logo and some server address, name and date details.\n\n### Delete Hello\n\n* Remove / comment out the app\n\n  Normally you just edit `apps/overlay/$DOUBLEDRAGON_NAME/kustomizaton.yaml`\n  And comment out\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      namespace: apps\n      bases:\n      #  - hello\n\n   But since this is the only app in there it might break the YAML.\n\n   So the easiert is to delete cluster's `apps.yaml` file\n\n      git rm clusters/$DOUBLEDRAGON_CLUSTER/apps.yaml;\n      git commit -m \"Removing apps from the cluster\";\n      git push\n\n  That should cascade the changes via the aggregated _kustomize_, and remove the ingress, service and deployment from the live cluster.\n\n* If permanent, remove the app's `apps/overlays` and `apps/base` folders as well as a tidy-up chore.\n\n\n## Your first application\n\nOk, so a _Hello world_ is not what you intended to use your cluster for.\nLets see how you could add a real application to the cluster.\n\nFor simplification lets call the app `myfirstapp`.\nReplace any reference to it with a correct name.\n\n### Deploy, Service, Ingress, Overlay\n\nThese will be very similar to the _Hello_ app.\n\n* Names and labels\n\n  Change all the names and lables from `hello` to `myfirstapp`.\n\n* Deployment\n\n  One thing to change is the image name and tag, and adding a registry secret\n\n  E.g.:\n\n      spec:\n        containers:\n          - image: gcr.io/somethingsomething:1.2.3\n            name: myfirstapp-container\n         ...\n        imagePullSecrets:\n          - name: gcr-registry\n\n\n* More _deployment_ config\n\n  For the _Hello_ deployment the basic config was sufficient,\n  but for more normal workflows you will most likely add more e.g. resource limits, env-vars, secrets etc.\n\n  E.g.:\n\n      spec:\n        containers:\n          - image: gcr.io/somethingsomething:1.2.3\n            ...\n            resources:\n              requests:\n                memory: \"250Mi\"\n                cpu: \"50m\"\n              limits:\n                memory: \"800Mi\"\n                cpu: \"250m\"\n            env:\n              - name: SOMEVAR\n                value: \"5\"\n            envFrom:\n              - secretRef:\n                  name: some-secret\n\n   These are out-of-scope for this tutorial though.\n\n* Ingress\n\n  You will need to change the hostname in `hosts`, and possibly the paths if necessary.\n\n* Overlay\n\n  You could be clever with the overlay but for now just copy what _Hello_ did\n\n* Add it all to git\n\n      git add apps/base/myfirstapp/deployment.yaml;\n      git add apps/base/myfirstapp/service.yaml;\n      git add apps/base/myfirstapp/ingress.yaml;\n      git add apps/base/myfirstapp/kustomization.yaml;\n      git add overlay/$DOUBLE_DRAGON_CLUSTER/myfirstapp/kustomization.yaml;\n      git add overlay/$DOUBLE_DRAGON_CLUSTER/kustomization.yaml;\n      git commit -m \"Added myfirstapp\"\n\n### Registry and image repository\n\n* Registry\n\n  You would probably store your app's images in a private Docker registry.\n\n  Revisit the [Container registries](#container-registries) section to add relevant registry secrets.\n\n* Image repository\n\n  Lets add an __Image Repository__ for your application.\n  So that the _deployment_ below can scan and find the _Docker_ image it requires\n\n      flux create image repository myfirstapp-source \\\n        --image=gcr.io/somethingsomething \\\n        --interval=5m \\\n        --namespace=infrastructure \\\n        --export \u003e infrastructure/sources/myfirstapp-source.yaml\n\n* Append this to the YAML in infrastructure/sources/myfirstapp-source.yaml:\n\n      ...\n      accessFrom:\n        namespaceSelectors:\n          - matchLabels:\n              kubernetes.io/metadata.name: apps\n\n* Append this source to the _Kustomization_ at `infrastructure/sources/kustomization.yaml`\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      resources:\n        ...\n        - myfirstapp-source.yaml\n\n* Add to git\n\n      git add infrastructure/sources/myfirstapp-source.yaml;\n      git add infrastructure/sources/kustomization.yaml;\n      git commit -m \"myfirstapp source\";\n      git push\n\n\n### Test the application\n\n* Like _hello_ use `curl`\n\n      curl -H \"Host: myfirstapp.example.com\" \\\n        --resolve myfirstapp.example.com:80:11.22.33.44 \\\n        --resolve myfirstapp.example.com:443:11.22.33.44 \\\n        http://myfirstapp.example.com | lynx -stdin\n\n   Replace `myfirstapp.example.com` with your hostname\n\nThat should be the basics to get your first application added\n\n## Automatic image updates\n\nLetting _Flux_ automatically update the image if a newer one get uploaded to the _docker registry_ is a handy feature.\n\n_Flux_ allows different policies on when to update it, such as only when approved, only on major version upgrades and more.\n\nHere is how to update on every new [semver](https://semver.org/) tag:\n\n\n* Image repository\n\n  As shown above in the [Your first application](#your-first-application) section, you need an _image repository_ source(s) for your application images\n\n* Image policies\n\n  Similarily we need a policy on how to act on any changes to the source repository\n\n      flux create image policy myfirstapp-policy \\\n        --image-ref=myfirstapp-source \\\n        --namespace=apps \\\n        --select-semver=0.3.x \\\n        --export \u003e ./apps/base/myfirstapp/image-policy.yaml\n\n  This policy allows updates on any new `0.3.x` semver versions.\n  So if on `0.3.1` if `0.3.2` gets uploaded it will trigger this policy.\n  A new lower version of `0.3.0` would not.\n\n  However `0.4.0` will not get uploaded. Nor would `1.0.0`.\n  For that the `--select-semver` would have to be `0.x` or just `x` I think\n\n* Add source namespace to the policy\n\n  The CLI does not have that option so please change the policy to refer to the \"infrastructure\" namespace:\n\n      ---\n      apiVersion: image.toolkit.fluxcd.io/v1beta1\n      kind: ImagePolicy\n      metadata:\n        name: myfirstapp-policy\n        namespace: apps\n      spec:\n        imageRepositoryRef:\n          name: myfirstapp-source\n          namespace: infrastructure\n        policy:\n          semver:\n            range: 0.3.x\n\n\n* Append this policy to the _Kustomization_ of the app `apps/base/myfirstapp/kustomization.yaml`\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      resources:\n        ...\n        - image-policy.yaml\n\n* Apply the policy\n\n  We need to tell _Flux_ where this policy would apply\n\n  Edit `apps/base/myfirstapp/deployment.yaml`\n  and modify the `image` line by appending the policy name\n\n      ...\n      spec:\n        containers:\n          - image: gcr.io/something:0.3.1 # {\"$imagepolicy\": \"apps:myfirstapp-policy \"}\n            ...\n\n  This seems a bit hacky but this is how it works\n\n* Image Update\n\n  We also need to tell the `apps` kustomization about how to update any the images.\n  I.e. create a _git commit_.\n\n      flux create image update apps-image-update --namespace=apps \\\n        --git-repo-ref=flux-system \\\n        --git-repo-namespace=flux-system \\\n        --git-repo-path=\"./\" \\\n        --checkout-branch=main \\\n        --push-branch=main \\\n        --author-name=fluxcdbot \\\n        --author-email=fluxcdbot@users.noreply.github.com \\\n        --commit-template=\"{{range .Updated.Images}}{{println .}}{{end}}\" \\\n        --export \u003e apps/overlays/$DOUBLE_DRAGON_NAME/image-updates.yaml\n\n* Append to `apps/overlays/$DOUBLE_DRAGON_NAME/kustomization.yaml`\n\n      apiVersion: kustomize.config.k8s.io/v1beta1\n      kind: Kustomization\n      bases:\n        - myfirstapp\n        ...\n      resources:\n        ...\n        - image-updates.yaml\n\n* Add to the repo\n\n      git add apps/base/myfirstapp/deployment.yaml;\n      git add apps/base/myfirstapp/image-policy.yaml;\n      git add apps/base/myfirstapp/kustomization.yaml;\n      git add apps/overlays/$DOUBLE_DRAGON_NAME/image-updates.yaml;\n      git add apps/overlays/$DOUBLE_DRAGON_NAME/kustomization.yaml;\n      git commit -m \"myfirstapp image policy\";\n      git push\n\n\n* New versions\n\n  Any new images in the docker repository that is in the chosen semver version range will now automatically update the deployment.\n\n  Note: You need to wait for the various polling of the image repository, the policy, flux itself etc. So changes might take awhile.\n\n  An alternative is that _Flux_ also let policy changes be prompted via webhook, which is also possible.\n\n* Tail if the policy has updated\n\n      flux get image policy -n apps --watch\n\n  Or any image states\n\n      flux get images all --all-namespaces\n## Go wild\n\n* Add/update your deployments, services, charts, docker registries, secrets, kustomizations etc\n\n### Usual steps for a simple web app\n\n1. Add source if in a private repo. And add/append to source _kustomization_.\n\n   * `infrastructure/sources/someapp-source.yaml`\n   * `infrastructure/sources/kustomization.yaml`\n   * `infrastructure/kustomization.yaml`\n\n1. Add deploy, service, ingress to new app base folder.\n\n   * `apps/base/someapp/deployment.yaml`\n   * `apps/base/someapp/service.yaml`\n   * `apps/base/someapp/ingress.yaml`\n   * `apps/base/someapp/image-policy.yaml`\n\n1. Add/append to apps _kustomization_ and overlay.\n\n   * `apps/base/someapp/kustomization.yaml`\n   * `apps/base/kustomization.yaml`\n   * `apps/overlay/somecluster/someapp/kustomization.yaml`\n   * `apps/overlay/somecluster/kustomization.yaml`\n\n   (Some of the _Kustomization_ files can be short-cutted if they do nothing but redirect)\n\n\n## Further information\n\n## Monitoring\n\nOptionally you may want to set up monitoring of the cluster, with metrics and logging.\n\nFlux has a ready made example of how to set up a Prometheus and Loki stack to achieve this.\n\n* https://github.com/fluxcd/flux2-monitoring-example\n\nAnd some genearl guidance on setting this up \n\n* https://fluxcd.io/flux/monitoring/\n* https://fluxcd.io/flux/monitoring/metrics/#monitoring-setup\n\n### Setting up Monitoring \n\n* Clone the example repo locally.\n* Copy the `monitoring.yaml` from that repo's cluster folder into your cluster.\n  * [flux2-monitoring-example/clusters/test/monitoring.yaml](https://github.com/fluxcd/flux2-monitoring-example/blob/main/clusters/test/monitoring.yaml)\n    =\u003e `cluster/$DOUBLE_DRAGON_NAME/monitoring.yaml` \n* Copy over the entire `monitoring` folder into your Flux repo.\n  * [flux2-monitoring-example/monitoring](https://github.com/fluxcd/flux2-monitoring-example/tree/main/monitoring)\n   =\u003e `./monitoring`\n* Modify the Prometheus Helm values for `podMonitorNamespaceSelector` to include other namespaces if tagged\n  * [monitoring/controllers/kube-prometheus-stack/release.yaml](https://github.com/fluxcd/flux2-monitoring-example/blob/main/monitoring/controllers/kube-prometheus-stack/release.yaml#L36)\n  * ```\n    podMonitorNamespaceSelector: \n      matchLabels:\n        kubernetes.io/metadata.name: monitoring  \n    ```\n* Modify the existing namespaces to include a *monitoring* label\n  * `cluster/doubledragon/namespaces.yaml`\n  * ``` \n    apiVersion: v1\n    kind: Namespace\n    metadata:\n    name: infrastructure\n    labels:\n      app.kubernetes.io/component: monitoring\n    ---\n    apiVersion: v1\n    kind: Namespace\n    metadata:\n      name: apps\n      labels:\n        app.kubernetes.io/component: monitoring\n    ```\n* Add to git, push and wait\n  ```\n  git add cluster/$DOUBLE_DRAGON_NAME/monitoring.yaml;\n  git add monitoring;\n  git add -p cluster/$DOUBLE_DRAGON_NAME/namespaces.yaml;\n  git commit -m \"Added monitoring\";\n  git push;\n  flux reconcile kustomization infrastructure --with-source;\n  kubectl get pods -A --watch\n  ```\n* Patience. It takes a while to set up and syncronise itself.\n\n* You do not need to expose the Grafana UI to the world. \n  Setting up a tunnel instead works fine\n  *  ```\n     kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80\n     ```\n  * [localhost:3000/d/flux-cluster](http://localhost:3000/d/flux-cluster/flux-cluster-stats)\n  \n## Auto scaling\n\nKubernetes can be configured to auto scale your nodes if increased loads/pods.\nThis is usually configured using the provider's UI, or CLI tool.\n\nHowever many will rely on your cluster having the [metrics-server](https://github.com/kubernetes-sigs/metrics-server/) installed.\n\nContrary to the name, this service is only metrics related for auto-scaling. \nActual metrics will be better served with the Prometheus stack mentioned above.\n\n* Add the Helm source\n\n  ```\n  flux create source helm metrics-server-source \\\n  --interval=1h \\\n  --namespace=infrastructure \\\n  --url=https://kubernetes-sigs.github.io/metrics-server/ \\\n  --export \u003e infrastructure/sources/metrics-server-source.yaml\n  ```\n\n* Edit `infrastructure/sources/kustomization.yaml`\n\n  and append `- metrics-server-source.yaml` to it\n\n  ```\n  apiVersion: kustomize.config.k8s.io/v1beta1\n  kind: Kustomization\n  resources:\n  ....\n  - metrics-server-source.yaml\n  ```\n\n* Add source to git\n\n  ```\n  git add infrastructure/sources/metrics-server-source.yaml;\n  git add infrastructure/sources/kustomization.yaml;\n  git commit -m \"Metrics server source\"\n  ```\n\n* Install the Helm chart\n\n  ```\n  mkdir -p infrastructure/metrics-server;\n\n  flux create helmrelease metrics-server \\\n    --interval=1h \\\n    --release-name=metrics-server \\\n    --namespace=infrastructure \\\n    --target-namespace=infrastructure \\\n    --source=HelmRepository/metrics-server-source \\\n    --chart=metrics-server \\\n    --chart-version=\"\u003e=0.7.2\" \\\n    --crds=CreateReplace \\\n    --export \u003e infrastructure/metrics-server/metrics-server.yaml\n  ```\n\n* Create `infrastructure/metrics-server/kustomization.yaml`\n\n  ```\n  apiVersion: kustomize.config.k8s.io/v1beta1\n  kind: Kustomization\n  resources:\n  - metrics-server.yaml\n  ```\n\n* Edit `infrastructure/kustomization.yaml`\n\n  and append `- metrics-server` to it\n\n  ```\n  apiVersion: kustomize.config.k8s.io/v1beta1\n  kind: Kustomization\n  resources:\n  ....\n  - metrics-server\n  ```\n \n* Push to git\n\n  ```\n  git add infrastructure/metrics-server/metrics-server.yaml;\n  git add infrastructure/metrics-server/kustomization.yaml;\n  git add infrastructure/kustomization.yaml;\n  git commit -m \"Adding metrics server\";\n  git push\n  ```\n\n## Advise: Don't touch\n\n* Once Flux is running, by convention avoid using `kubectl create|apply` etc.\n* And by the same convention avoid using `flux create`.\n\n  i.e avoid acting directly for any _write_ operations.\n\n  Nearly all changes should be via Git. Export any changed YAML to Git as above.\n\n  Otherwise the git source and cluster state will start to diverge and hard to recreate.\n\n* Any `kubectl` and  `flux` interaction should be read only. Those are fine.\n\n* Sometimes whilst troubleshooting you will have to use the scalpel and use `kubectl create|apply|delete` or `flux create`.\n\n   But minimise the usage, and try to update the yaml to reflect any permanent changes.\n\n## Troubleshooting\n\nFrequent issues and how to monitor.\n\n### 1 Tail the logs\n\n* _Flux_ logs\n\n      flux logs -Af --since 3h\n\n* _Kubernetes_ logs\n\n      kubectl logs podname -f\n\n### 2 Watch statuses\n\n* _Flux_ kustomization status\n\n      flux get kustomizations --watch\n\n* _Kubernetes_ status\n\n      kubectl get deploy,service,ingress,pods,secret,imagerepository,clusterissuer -n apps\n\n  Or watch a single resource type\n\n      kubectl get deploy -A --watch\n\n### 3 Known possible issues\n\n* Certain operation takes a few minutes, e.g. pod creation, waiting on Flux scan polling\n\n* Nginx: `x509 certificate is not valid`\n\n  Happens sometimes with the Nginx ingresses.\n  Seems to be a known problem that require [manual patching](https://github.com/jet/kube-webhook-certgen#patch).\n  Or as I fix it:\n\n  * Comment out the _nginx controller_ and the ingresses from the _kustomization_ files.\n  * Wait until _Flux_ has removed them from the cluster.\n  * Uncomment and add them back in.\n  * Note this may change the external IP assigned to the cluster's load balancer.\n\n\n### 4 Remove and add\n\n* Fixed typos, and nothing changes?\n\n  Sometimes some resources gets added with a typo,\n  but you fixed it and pushed the change to the repo,\n  yet _Flux_ or _Kubernetes_ do not pick up the change?\n\n  Most of the time _Flux_ and _Kubernetes_ notices and changes the resources. But sometimes not.\n\n* Force the change.\n\n  Simply remove the resource, push to git, let the system catch up, add it back with the typo corrected, and the change gets picked up\n\n  Most of the time the _\"removal\"_ can be done by commenting out the reference to it in a `kustomization.yaml` file.\n  Instead of removing actual _deployment_ etc git files and history.\n\n* Scale the deployments down and back up\n\n      kubectl scale deploy -n apps --replicas=0 myfirstapp-deployment\n\n  Wait until pods are destroyed\n\n      kubectl get pods -n apps --watch\n\n  Then scale back up\n\n      kubectl scale deploy -n apps --replicas=2 myfirstapp-deployment\n\n  Note, sometimes _flux_ notices the inconsistency and scales the app back up as well before you do.\n\n* Force _flux_ do reconcile the cluster state and the repository state\n\n  If impatient\n\n       flux reconcile kustomization apps --with-source\n\n## Removing _Flux_\n\nThe nuclear option. But sometimes neccessary.\n\n    flux uninstall --namespace=flux-system\n\nThough usually I just spin up another cluster instead.\n\nNote, any encrypted secrets will have to be re-sealed when re-installing _flux_ with the new certificate\n\n## Add another cluster\n\n* Now that you have a working cluster, scrap it. If you want to.\n\n  Create a new cluster without all the mistakes from setting up the first cluster.\n\n* Or when you just need another cluster naturally, you can do the same.\n\n* In only a few steps, you do not have to do it all again\n\n### Create and bootstrap another cluster\n\n* Create the cluster with your provider\n\n* Authenticate `kubectl` with the new cluster\n\n* Set as the current kubernetes context\n\n* Maybe export a new env-var (and old ones if no longer set)\n\n  In Bash:\n\n       export DOUBLEDRAGON_REPO=doubledragon-fleet;\n       export DOUBLEDRAGON_CLUSTER=doubledragon-01;\n       export DOUBLEDRAGON_CLUSTER_NEW=doubledragon-02\n\n  In Fish:\n\n       set -x DOUBLEDRAGON_REPO doubledragon-fleet;\n       set -x DOUBLEDRAGON_CLUSTER doubledragon-01;\n       set -x DOUBLEDRAGON_CLUSTER_NEW doubledragon-02\n\n* Bootstrap the new cluster with `doubledragon-02` or `$DOUBLEDRAGON_CLUSTER_NEW` as the name\n\n      flux bootstrap github \\\n        --token-auth=false \\\n        --components-extra=image-reflector-controller,image-automation-controller \\\n        --owner=$GITHUB_USER \\\n        --repository=$DOUBLEDRAGON_REPO \\\n        --branch=main \\\n        --path=./clusters/$DOUBLEDRAGON_CLUSTER_NEW \\\n        --read-write-key \\\n        --personal\n\n   After a while pull the changes\n\n      git pull\n\n### Copy and re-initialise infrastructure\n\n* Add namespaces to the new cluster\n\n      cp clusters/$DOUBLEDRAGON_CLUSTER/namespaces.yaml clusters/$DOUBLEDRAGON_CLUSTER_NEW/;\n      git add clusters/$DOUBLEDRAGON_CLUSTER_NEW/namespaces.yaml;\n      git commit -m \"Double Dragon II namespaces\";\n      git push\n\n* Copy the `infrastructure.yaml` kustomization to the new cluster\n\n      cp clusters/$DOUBLEDRAGON_CLUSTER/infrastructure.yaml clusters/$DOUBLEDRAGON_CLUSTER_NEW/;\n      git add clusters/$DOUBLEDRAGON_CLUSTER_NEW/infrastructure.yaml;\n      git commit -m \"Double Dragon II infrastructure\";\n      git push\n\n  This will add the _Sealed Secrets_, _Nginx_, and everything in _sources_ to the new cluster.\n  And more if you have extended it.\n\n  The _sources_ may cause issues initially until we re-encrypt any secrets.\n\n* Download the _Sealed Secrets_ public key for this cluster\n\n      mkdir -p clusters/$DOUBLEDRAGON_CLUSTER_NEW/secrets;\n\n      kubeseal --fetch-cert \\\n      --controller-name=sealed-secrets-controller \\\n      --controller-namespace=infrastructure \\\n      \u003e clusters/$DOUBLEDRAGON_CLUSTER_NEW/secrets/sealed-secrets-cert.pem;\n\n      git add clusters/$DOUBLEDRAGON_CLUSTER_NEW/secrets/sealed-secrets-cert.pem;\n      git commit -m \"Sealed Secret public key\";\n      git push\n\n* Re-encrypt secrets such as the GCR registry secret if needed\n\n  E.g.\n\n      mkdir -p clusters/$DOUBLEDRAGON_CLUSTER_NEW/registries/apps;\n      mkdir -p clusters/$DOUBLEDRAGON_CLUSTER_NEW/registries/infrastructure;\n\n      kubeseal --format=yaml --namespace=apps \\\n       --cert=clusters/$DOUBLEDRAGON_CLUSTER_NEW/secrets/sealed-secrets-cert.pem \\\n       \u003c gcr-registry.yml \\\n       \u003e clusters/$DOUBLEDRAGON_CLUSTER_NEW/registries/apps/sealed-gcr-registry.yml;\n\n      kubeseal --format=yaml --namespace=infrastructure \\\n       --cert=clusters/$DOUBLEDRAGON_CLUSTER_NEW/secrets/sealed-secrets-cert.pem \\\n       \u003c gcr-registry.yml \\\n       \u003e clusters/$DOUBLEDRAGON_CLUSTER_NEW/registries/infrastructure/sealed-gcr-registry.yml;\n\n      git add clusters/$DOUBLEDRAGON_CLUSTER_NEW/registries/apps/sealed-gcr-registry.yml;\n      git add clusters/$DOUBLEDRAGON_CLUSTER_NEW/registries/infrastructure/sealed-gcr-registry.yml;\n      git commit -m \"GCR registry for cluster DD-02 ns\";\n      git push\n\n### Copy/tweak over other cluster specifics\n\n* Such as *certificate-manager* issuers\n\n      mkdir -p clusters/$DOUBLEDRAGON_CLUSTER_NEW/certificate-issuers;\n      cp clusters/$DOUBLEDRAGON_CLUSTER/certificate-issuers/* \\\n         clusters/$DOUBLEDRAGON_CLUSTER_NEW/certificate-issuers/;\n\n      git add clusters/$DOUBLEDRAGON_CLUSTER_NEW/certificate-issuers/*;\n      git commit -m \"Staging and Prod issuer\";\n      git push\n\n### Copy/tweak apps overlay\n\n* Optionally create a new overlay\n\n  Or share the same common one in `apps/overlays/doubledragon`\n  linked in `apps.yaml`\n\n      cp clusters/$DOUBLEDRAGON_CLUSTER/apps.yaml clusters/$DOUBLEDRAGON_CLUSTER_NEW/;\n      git add clusters/$DOUBLEDRAGON_CLUSTER_NEW/apps.yaml;\n      git commit -m \"Apps overlay for cluster DD-02\";\n      git push\n\n* And that will be it\n\n* Note, the exposed load balancer external IP will be different.\n\n## Providers and tools\n\n### Kubernetes as a Service\n\n* Cloud providers\n\n  * Amazon AWS EKS: [aws.amazon.com/eks/](https://aws.amazon.com/eks/)\n  * Google Cloud GKE: [cloud.google.com/kubernetes-engine/](https://cloud.google.com/kubernetes-engine/)\n  * Microsoft Azure AKS: [azure.microsoft.com/en-us/services/kubernetes-service/](https://azure.microsoft.com/en-us/services/kubernetes-service/)\n  * DigitalOcean Kubernetes: [www.digitalocean.com/products/kubernetes/](https://www.digitalocean.com/products/kubernetes/)\n\n* Cloud provider CLIs\n\n  * [cloud.google.com/sdk/](https://cloud.google.com/sdk/)\n\n        brew cask install google-cloud-sdk\n\n  * [github.com/digitalocean/doctl](https://github.com/digitalocean/doctl)\n\n        brew install doctl\n  * [eksctl.io](https://eksctl.io/)\n\n        brew tap weaveworks/tap\n        brew install weaveworks/tap/eksctl\n\n  * [github.com/Azure/azure-cli](https://github.com/Azure/azure-cli)\n\n        brew install azure-cli\n\n### Tools\n\n* [github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)\n\n      brew install kubectx\n\n* [github.com/vmware-tanzu/octant](https://github.com/vmware-tanzu/octant)\n\n      brew install octant\n\n* [k9ss.io](https://k9ss.io)\n\n      brew install derailed/k9s/k9s\n\n* [keel.sh](https://keel.sh)\n\n* [headlamp.dev](https://headlamp.dev)\n   \n  * [headlamp.dev/blog/2025/03/26/flux-ui-updates](https://headlamp.dev/blog/2025/03/26/flux-ui-updates)\n\n* [github.com/stern/stern](https://github.com/stern/stern)\n\n      brew install stern\n\n* [flurdy.com/docs/kubernetes/registry/kubernetes-docker-registry.html](https://flurdy.com/docs/kubernetes/registry/kubernetes-docker-registry.html)\n* [ramitsurana.github.io/awesome-kubernetes/](https://ramitsurana.github.io/awesome-kubernetes/)\n\nClient tools are also available on Linux, Windows and more.\n\n### License\n\n\nThe _Lemmings_ and _Double Dragon_ code bases are licensed under the _MIT license_ which lets you pretty much do as you please with it.\n\nThough please attribute back if possible.\n\n### Attributions\n\n* This guide heavily used the docs and example project available on the official Flux website\n   * [fluxcd.io/flux](https://fluxcd.io/flux/)\n* Kubernetes official docs\n   * [kubernetes.io/docs/](https://kubernetes.io/docs/)\n* [cert-manager.io/docs/](https://cert-manager.io/docs/installation/continuous-deployment-and-gitops/)\n\n### Versions\n\n* 2025-03-25 Updated for Flux 2.5\n* 2023-03-21 Your first app and image updates\n* 2023-02-09 Double Dragon tweaks and env-vars\n* 2022-11-10 Double Dragon refreshed\n* 2021-07-10 Flux 2. Lemmings =\u003e Double Dragon\n* 2020-02-13 Flux 1.1, fluxcd.io annotations, and Helm 3\n* 2019-11-07 Flux 0.16, flux.weave.works annotations and Helm 2\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflurdy%2Fdoubledragon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflurdy%2Fdoubledragon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflurdy%2Fdoubledragon/lists"}