{"id":19835422,"url":"https://github.com/equinor/sdp-flux","last_synced_at":"2025-05-01T17:33:46.592Z","repository":{"id":34153544,"uuid":"154460139","full_name":"equinor/sdp-flux","owner":"equinor","description":"Flux continuous delivery to k8s","archived":false,"fork":false,"pushed_at":"2023-07-04T08:12:08.000Z","size":1049,"stargazers_count":23,"open_issues_count":1,"forks_count":2,"subscribers_count":5,"default_branch":"prod","last_synced_at":"2023-11-01T11:26:13.117Z","etag":null,"topics":["aks","flux","gitops","helm","k8s","kubernetes"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/equinor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-10-24T07:48:00.000Z","updated_at":"2023-08-29T11:03:31.000Z","dependencies_parsed_at":"2023-02-14T06:00:37.383Z","dependency_job_id":null,"html_url":"https://github.com/equinor/sdp-flux","commit_stats":null,"previous_names":[],"tags_count":13,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equinor%2Fsdp-flux","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equinor%2Fsdp-flux/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equinor%2Fsdp-flux/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equinor%2Fsdp-flux/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/equinor","download_url":"https://codeload.github.com/equinor/sdp-flux/tar.gz/refs/heads/prod","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224270267,"owners_count":17283649,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aks","flux","gitops","helm","k8s","kubernetes"],"created_at":"2024-11-12T12:08:03.164Z","updated_at":"2024-11-12T12:08:03.950Z","avatar_url":"https://github.com/equinor.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.com/equinor/sdp-flux.svg?token=yR5pmi3sbtpmzTWwTfNG\u0026branch=master)](https://travis-ci.com/equinor/sdp-flux)\n\n# Equinor SDP-Teams K8s GitOps\n\nThis is the repository where we define the manifest to be run in our Kubernetes cluster.  \nWe use [Flux](https://github.com/weaveworks/flux) as the basis for our GitOps workflow.\n\n## Related Repositories\n\nIaC and Bootstrap - https://github.com/equinor/sdp-omnia\n\n## How it works\n\nIn essence we create manifests to be run on the Kubernetes cluster, commit them to this repository and then the Flux controller notices the new commit and applies all the YAML files (can be simplified down to `kubectl apply -f FILENAME`).\n\nWe have integrated [Kustomize](https://kustomize.io/) support with Flux. This means that the [/base](/base) folder contains common configuration for all our clusters. Any changes between the clusters (mostly DNS config), are made in \"patches\" to the base file found in the [/dev](/dev) and [/prod](/prod) folders. Different cluster's Flux operator are subscribed to a specific repo, branch, and kustomize-path-path for an effective GitOps workflow.\n\nTypically we set base equals to the values of prod, and patches in dev are mostly used for overwriting ingresses.\n\nDeleting helm releases is currently not done through Flux.\n\n1) Firstly, remove references of the chart from the Git repo. Remember to update the kustomization.yaml files.\n2) Then eiher delete the entire namespace of the helm release you wish to remove `kubectl delete helmrelease xxx -n yyy`\n\n## How we use it\n\nBy utilizing Kustomize, we are able to use the same manifests in different clusters, with context aware patches. The biggest benefit of this is that we can use a standard git branch strategy, which involves pushing changes to a _dev_ branch, and merging into a _prod_ branch when it's tested OK in dev. For optimal GitOps workflow! :+1\n\nThis simplistic branching model will likely not work in the same way for larger teams, but for our use case it works well. Usually the flow is\n\n- Make changes to dev\n- Test\n- PR from dev branch to prod branch\n- Squash and merge if latest dev commits are messy, or add merge commit if history is somewhat clean\n- Rebase dev branch on prod branch to keep commits in sync for optimal overview.\n\nSometimes you make changes to the dev branch, but are not ready to merge into prod yet, but your team member wants to merge something else in.\nIn these cases, either move your unready changes to a separate branch, and let the other team member merge dev into prod. Alternatively you can keep the changes, intermix commits to dev with your team member, before making a PR, your team member should now take his changes to a separate branch (which has now been tested through the dev branch), and be ready to merge from the feature branch into prod.\n\n## Naming conventions\n\nThe `custom-charts` folder contains the charts we have created ourselves and and is needed for our cluster.  \n\nThe creation of new namespaces is done by creating a new file with the same name as the namespace(.yaml) and place it under the folder `namespaces`.\n\nAll other k8s manifests are placed in the `base` folder, organized by _namespace_ and service.  \nWe keep one k8s resource in each file, and name the file the type of k8s resource it is. So an _ingress_ resource for the _demo_ application, in the namespace _apps_ will be named; `./base/apps/demo/ingress.yaml`\n\n## Using SealedSecrets\n\nSometimes we need to store secrets in the Git repository to make sure our repository is the primary source of truth. In these cases we use a tool called [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets). With this we can store secrets enrypted in the repository and be sure that Flux manages and puts them in the Kubernetes cluster.\n\nNote that some secrets must be in place before deployments such as flux are in place. These are created in the sdp-omnia repo, with secrets stored to Azure Keyvault.\n\n### Install client\n\n- Download [kubeseal](https://github.com/bitnami-labs/sealed-secrets/releases)\n- Put `kubeseal-linux-amd64` or `kubeseal-darwin-amd64` in your path and rename it to `kubeseal`.\n\n### First time\n\nAfter first run we need to export the private and public key. The public key is what we all use to encrypt the secrets and the private key is used by the controller to decrypt the secrets and put them in the cluster. It is important to have a backup of the private key, but _NEVER_ commit that to the repository.\n\n- To get the __private key__ (remember to store this file somewhere safe so we can restore in case of emergency)\n  `kubectl get secret -n sealed-secrets sealed-secret-custom-key -o yaml \u003e sealedsecrets.yaml`\n\n- Use the kubeseal tool to get the __public key__\n  `kubeseal --controller-namespace sealed-secrets --controller-name sealed-secrets --fetch-cert \u003e secret.pem`\n\n### Creating SealedSecrets\n\nYou need to use the kubectl to initially create the secret and then pipe this to kubeseal to make it encrypted.\n\n``` bash\n# From literal\nkubectl -n \u003cNAMESPACE\u003e create secret generic \u003cSECRETNAME\u003e --namespace=\u003cNAMESPACE\u003e --dry-run=client --from-literal=\u003cKEY\u003e=\u003cVALUES\u003e -o json | kubeseal --format yaml --cert sealed-secret.pem \u003e sealed-secret.yaml\n\n# From file\nkubectl -n \u003cNAMESPACE\u003e create secret generic \u003cSECRETNAME\u003e --namespace=\u003cNAMESPACE\u003e --dry-run=client --from-file=\u003cFILENAME\u003e -o json | kubeseal --cert sealed-secret.pem --format yaml \u003e sealed-secret.yaml\n\n# TLS secret\n# Remember to include intermediate certificates if any (goes on the end of the .crt file)\nkubectl create secret tls \u003cSECRETNAME\u003e --namespace=\u003cNAMESPACE\u003e --key myTLSCert.key --cert myTLSCert.crt --dry-run=client -o json | kubeseal --format yaml --cert sealed-secret.pem \u003e sealed-secret-tls.yaml\n\n# Validate that the SS controller can decrypt the sealed-secret\ncat ./sealed-secret-tls.yaml |kubeseal --controller-namespace sealed-secrets --controller-name sealed-secrets --validate\n```\n\nMake sure to place the secrets in the appropriate namespace folder to keep the repository organised.\n\n## Oauth2 Proxy\n\nTo utilize oauth2-proxy to authenticate users before they can access a web application, add these lines to the **ingress annotation**:\n\n``` yaml\nnginx.ingress.kubernetes.io/auth-url: \"https://$host/oauth2/auth\"\nnginx.ingress.kubernetes.io/auth-signin: \"https://$host/oauth2/start?rd=$escaped_request_uri\"\n```\n\nYou also need a **host specific ingress for the proxy**. That could look like [this](examples/oauth2-ingress.yaml).\n\nThere is also a few more things to note;\n\n- The oauth2-proxy need to be deployed in the same namespace\n- The proxy must be configured with the specific Azure app. This is where we define access. Each proxy can only authenticate *one* application.\n- Note that in Helm version \u003c v3, the name of a helm release must be uniq in the *cluster*, NOT within a *namespace*. So to deploy multiple oauth2-proxy instances, the helm release name must be different, the service name will also change.\n\n### Configure Oauth2-proxy\n\nEach oath2 HelmRelease need a secret containing there keys;\n\n- client-id\n- client-secret\n- cookie-secret\n\nThe cookie secret can be created like this;  \n\n```bash\ndocker run -ti --rm python:3-alpine python -c 'import secrets,base64; print(base64.b64encode(base64.b64encode(secrets.token_bytes(16))));'`\n```\n\nYou only need to copy the content after \"b'\".\n\n## Tips and tricks\n\nHow to do something we do a lot? If you know, type them up here and we shall all be the wiser for it.\n\n### Using a Private Container Registry\n\nWe assume the ACR has been created and set up in the bootstrapping portion of the Kubernetes cluster. Take a look in sdp-aks repository for information on how to create and set up a new ACR. To use the ACR we need a service principal that should have been created by ARM templates.\n\n- Start by creating the secret which stores docker registry information;  \n  `kubectl -n \u003cNAMESPACE\u003e create secret docker-registry \u003cSECRET_NAME\u003e --docker-server=\u003cREGISTRY_URL\u003e --docker-username=\u003cSERVICE_PRINCIPAL_ID\u003e --docker-password=\u003cSERVICE_PRINCIPAL_PASSWORD\u003e --docker-email=gm_sds_rdi@equinor.com`\n\n- To use this secret in the image pull, use it in a manifest like so;\n\n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  name: private-reg\nspec:\n  containers:\n  - name: private-reg-container\n    image: REGISTRY_URL/\u003cyour-private-image\u003e\n  imagePullSecrets:\n  - name: SECRET_NAME\n```\n\n### Produce certificates from ingress config (Cert Manager)\n\nWe use [CertManager](https://github.com/jetstack/cert-manager) for creating, validating and deploying Let'sEncrypt certificates. CertManager can shortcut the usual procedure of creating a Certificate resource and then referencing the secret from this Certificate in the ingress by adding a few annotations to the ingress.\n\nIn the shortest terms, add these [annotations to your ingress](http://docs.cert-manager.io/en/latest/reference/ingress-shim.html#supported-annotations)\n\n```yaml\nkubernetes.io/ingress.class: nginx\nkubernetes.io/tls-acme: \"true\"\n```\n\nIf you want a Let'sEncrypt testing certificate (to not expend the cert quota), you can specify another certificate issuer. For a cluster issuer add this:\n\n```yaml\ncertmanager.k8s.io/cluster-issuer: \"letsencrypt-staging\"\n```\n\nThis works because we have defined a default issuer and protocol when we deployed [CertManager](releases/infrastructure/hr-cert-manager.yaml).\n\n### Automaticly create DNS entries (External DNS)\n\nWe have implemented External DNS with the Kubernetes cluster and connected it to the Azure DNS Zone. This is done with a Azure AD Service Principal that has rights to only change this DNS Zone.\n\n[External DNS](https://github.com/kubernetes-incubator/external-dns) updates DNS Zone entries with the help of annotations in ingress resources. To create and update a DNS entry use the following annotation\n\n```yaml\nexternal-dns.alpha.kubernetes.io/hostname: demo.example.com.\n```\n\n### Full example using Cert Manager and External DNS\n\nThis example manifest uses the techniques in the two previous sections and shows how these works together to automate some tedious tasks.\nMore info can be found on [HelmRelease](https://github.com/weaveworks/flux/blob/master/site/helm-integration.md), in this example we use a git repository\nfor the Helm chart, but you could use a Helm repository.\n\n```yaml\n---\napiVersion: helm.fluxcd.io/v1 \nkind: HelmRelease\nmetadata:\n  name: sdp-demo\n  namespace: prod\n  annotations:\n    flux.weave.works/automated: \"true\"\nspec:\n  releaseName: sdp-demo\n  chart:\n    git: ssh://git@github.com/equinor/sdp-flux.git\n    ref: prod\n    path: charts/sdp-demo\n  values:\n    ingress:\n      enabled: true\n      annotations:\n        kubernetes.io/ingress.class: nginx\n        kubernetes.io/tls-acme: \"true\"\n        external-dns.alpha.kubernetes.io/hostname: demo.example.com.\n      hosts:\n      - demo.example.com\n      tls:\n      - secretName: demo-tls\n        hosts:\n        - demo.example.com\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fequinor%2Fsdp-flux","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fequinor%2Fsdp-flux","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fequinor%2Fsdp-flux/lists"}