{"id":13521793,"url":"https://github.com/stefanprodan/gitops-istio","last_synced_at":"2025-05-14T23:06:35.062Z","repository":{"id":37677609,"uuid":"131903780","full_name":"stefanprodan/gitops-istio","owner":"stefanprodan","description":"A GitOps recipe for Progressive Delivery with Flux v2, Flagger and Istio","archived":false,"fork":false,"pushed_at":"2025-05-09T08:02:26.000Z","size":455,"stargazers_count":664,"open_issues_count":3,"forks_count":441,"subscribers_count":22,"default_branch":"main","last_synced_at":"2025-05-09T09:22:13.255Z","etag":null,"topics":["ab-testing","flagger","flux","gitops","istio","kubernetes","progressive-delivery"],"latest_commit_sha":null,"homepage":"https://flagger.app","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stefanprodan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-05-02T20:42:22.000Z","updated_at":"2025-04-15T20:32:32.000Z","dependencies_parsed_at":"2023-10-04T13:58:17.907Z","dependency_job_id":"248090bd-7124-4ffc-bed9-7bf506ea4f9d","html_url":"https://github.com/stefanprodan/gitops-istio","commit_stats":{"total_commits":196,"total_committers":9,"mean_commits":21.77777777777778,"dds":0.2908163265306123,"last_synced_commit":"3851e1802537cd5f7f833b05502f6db1f4528f47"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanprodan%2Fgitops-istio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanprodan%2Fgitops-istio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanprodan%2Fgitops-istio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanprodan%2Fgitops-istio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stefanprodan","download_url":"https://codeload.github.com/stefanprodan/gitops-istio/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254243360,"owners_count":22038046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-testing","flagger","flux","gitops","istio","kubernetes","progressive-delivery"],"created_at":"2024-08-01T06:00:38.094Z","updated_at":"2025-05-14T23:06:30.053Z","avatar_url":"https://github.com/stefanprodan.png","language":null,"readme":"# gitops-istio\n\n[![e2e](https://github.com/stefanprodan/gitops-istio/workflows/e2e/badge.svg)](https://github.com/stefanprodan/gitops-istio/actions)\n[![e2analyzee](https://github.com/stefanprodan/gitops-istio/workflows/analyze/badge.svg)](https://github.com/stefanprodan/gitops-istio/actions)\n[![license](https://img.shields.io/github/license/stefanprodan/gitops-istio.svg)](https://github.com/stefanprodan/gitops-istio/blob/main/LICENSE)\n\nThis is a guide where you will get hands-on experience with GitOps and\nProgressive Delivery using Kubernetes and Istio.\n\n## Introduction\n\n### What is GitOps?\n\nGitOps is a way to do Continuous Delivery, it works by using Git as a source of truth\nfor declarative infrastructure and workloads.\nFor Kubernetes this means using `git push` instead of `kubectl apply/delete` or `helm install/upgrade`.\n\nIn this workshop you'll be using GitHub to host the config repository and [Flux](https://fluxcd.io)\nas the GitOps delivery solution.\n\n### What is Progressive Delivery?\n\nProgressive delivery is an umbrella term for advanced deployment patterns like canaries, feature flags and A/B testing.\nProgressive delivery techniques are used to reduce the risk of introducing a new software version in production\nby giving app developers and SRE teams a fine-grained control over the blast radius.\n\nIn this workshop you'll be using [Flagger](https://flagger.app), Istio and Prometheus to automate\nCanary Releases and A/B Testing for your applications.\n\n![Progressive Delivery GitOps Pipeline](/docs/images/flux-flagger-gitops.png)\n\n## Prerequisites\n\nYou'll need a Kubernetes cluster **v1.20** or newer with `LoadBalancer` support. \n\nFor testing purposes you can use Minikube with 2 CPUs and 4GB of memory:\n```bash\nminikube start --cpus='2' --memory='4g'\n```\nIf using Minikube, run the following in a separate terminal window/tab for the duration of this workshop:\n```bash\nminikube tunnel\n```\nThis assigns an External-IP to the istio-gateway service and allows the helm install to complete successfully.\n\nInstall `jq`, `yq` and the `flux` CLI with Homebrew:\n\n```bash\nbrew install jq yq fluxcd/tap/flux\n```\n\nVerify that your cluster satisfies the prerequisites with:\n\n```bash\nflux check --pre\n```\n\nFork this repository and clone it:\n\n```bash\ngit clone https://github.com/\u003cYOUR-USERNAME\u003e/gitops-istio\ncd gitops-istio\n```\n\n## Cluster bootstrap\n\nWith `flux bootstrap` command you can install Flux on a Kubernetes cluster\nand configure it to manage itself from a Git repository.\nIf the Flux components are present on the cluster,\nthe bootstrap command will perform an upgrade if needed.\n\nBootstrap Flux by specifying your GitHub repository fork URL:\n\n```bash\nflux bootstrap git \\\n  --author-email=\u003cYOUR-EMAIL\u003e \\\n  --url=ssh://git@github.com/\u003cYOUR-USERNAME\u003e/gitops-istio \\\n  --branch=main \\\n  --path=clusters/my-cluster\n```\n\nThe above command requires ssh-agent, if you're using Windows see\n[flux bootstrap github](https://fluxcd.io/docs/guides/installation/#github-and-github-enterprise) documentation.\n\nAt bootstrap, Flux generates an SSH key and prints the public key.\nIn order to sync your cluster state with git you need to copy the public key and create a deploy key with write \naccess on your GitHub repository. On GitHub go to _Settings \u003e Deploy keys_ click on _Add deploy key_, \ncheck _Allow write access_, paste the Flux public key and click _Add key_.\n\nWhen Flux has access to your repository it will do the following:\n\n* installs Istio using the Istio `base`, `istiod` and `gateway` Helm charts\n* waits for Istio control plane to be ready\n* installs Flagger, Prometheus and Grafana\n* creates the Istio public gateway\n* creates the `prod` namespace\n* creates the load tester deployment\n* creates the frontend deployment and canary\n* creates the backend deployment and canary\n\nWhen bootstrapping a cluster with Istio, it is important to control the installation order.\nFor the applications pods to be injected with Istio sidecar,\nthe Istio control plane must be up and running before the apps.\n\nWith Flux v2 you can specify the execution order by defining dependencies between objects.\nFor example, in [clusters/my-cluster/apps.yaml](https://github.com/stefanprodan/gitops-istio/blob/main/clusters/my-cluster/apps.yaml)\nwe tell Flux that the `apps` reconciliation depends on the `istio-system` one:\n\n```yaml\napiVersion: kustomize.toolkit.fluxcd.io/v1beta2\nkind: Kustomization\nmetadata:\n  name: apps\n  namespace: flux-system\nspec:\n  interval: 30m0s\n  dependsOn:\n    - name: istio-system\n  sourceRef:\n    kind: GitRepository\n    name: flux-system\n  path: ./apps\n```\n\nWatch Flux installing Istio first, then the demo apps:\n\n```bash\nwatch flux get kustomizations\n```\n\nYou can tail the Flux reconciliation logs with:\n\n```bash\nflux logs --all-namespaces --follow --tail=10\n```\n\nList all the Kubernetes resources managed by Flux with:\n\n```bash\nflux tree kustomization flux-system\n```\n\n## Istio customizations\n\nYou can customize the Istio installation using the Flux `HelmReleases` located at\n[istio/system/istio.yaml](https://github.com/stefanprodan/gitops-istio/blob/main/istio/system/istio.yaml):\n\n```yaml\napiVersion: helm.toolkit.fluxcd.io/v2beta1\nkind: HelmRelease\nmetadata:\n  name: istio-gateway\n  namespace: istio-system\nspec:\n  dependsOn:\n    - name: istio-base\n    - name: istiod\n  # source: https://github.com/istio/istio/blob/master/manifests/charts/gateway/values.yaml\n  values:\n    autoscaling:\n      enabled: true\n```\n\nAfter modifying the Helm release values, you can push the change to git and Flux\nwill reconfigure the Istio control plane according to your changes.\n\nYou can monitor the Helm upgrades with:\n\n```bash\nflux -n istio-system get helmreleases --watch\n```\n\nTo troubleshoot upgrade failures, you can inspect the Helm release with:\n\n```bash\nkubectl -n istio-system describe helmrelease istio-gateway\n```\n\nFlux issues Kubernetes events containing all the errors encountered during reconciliation.\nYou could also configure Flux to publish the events to Slack, MS Team, Discord and others;\nplease the [notification guide](https://fluxcd.io/docs/guides/notifications/) for more details.\n\n## Istio control plane upgrades\n\nIstio upgrades are automated using GitHub Actions and Flux.\n\n![Flux Istio Operator](docs/images/flux-istio-gitops.png)\n\nWhen a new Istio version is available, the\n[`update-istio` GitHub Action workflow](https://github.com/stefanprodan/gitops-istio/blob/main/.github/workflows/update-istio.yaml)\nwill open a pull request with the manifest updates needed for upgrading Istio.\nThe new Istio version is tested on Kubernetes Kind by the\n[`e2e` workflow](https://github.com/stefanprodan/gitops-istio/blob/main/.github/workflows/e2e.yaml)\nand when the PR is merged into the main branch, Flux will upgrade Istio on the production cluster.\n\n## Application bootstrap\n\nWhen Flux syncs the Git repository with your cluster, it creates the frontend/backend deployment, HPA and a canary object.\nFlagger uses the canary definition to create a series of objects: Kubernetes deployments, \nClusterIP services, Istio destination rules and virtual services. These objects expose the application on the mesh and drive \nthe canary analysis and promotion.\n\n```bash\n# applied by Flux\ndeployment.apps/frontend\nhorizontalpodautoscaler.autoscaling/frontend\ncanary.flagger.app/frontend\n\n# generated by Flagger\ndeployment.apps/frontend-primary\nhorizontalpodautoscaler.autoscaling/frontend-primary\nservice/frontend\nservice/frontend-canary\nservice/frontend-primary\ndestinationrule.networking.istio.io/frontend-canary\ndestinationrule.networking.istio.io/frontend-primary\nvirtualservice.networking.istio.io/frontend\n```\n\nCheck if Flagger has successfully initialized the canaries: \n\n```\nkubectl -n prod get canaries\n\nNAME       STATUS        WEIGHT\nbackend    Initialized   0\nfrontend   Initialized   0\n```\n\nWhen the `frontend-primary` deployment comes online, \nFlagger will route all traffic to the primary pods and scale to zero the `frontend` deployment.\n\nFind the Istio ingress gateway address with:\n\n```bash\nkubectl -n istio-system get svc istio-ingressgateway -ojson | jq .status.loadBalancer.ingress\n```\n\nOpen a browser and navigate to the ingress address, you'll see the frontend UI.\n\n## Canary releases\n\nFlagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators\nlike HTTP requests success rate, requests average duration and pod health.\nBased on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack.\n\nA canary analysis is triggered by changes in any of the following objects:\n* Deployment PodSpec (container image, command, ports, env, etc)\n* ConfigMaps and Secrets mounted as volumes or mapped to environment variables\n\nFor workloads that are not receiving constant traffic Flagger can be configured with a webhook, \nthat when called, will start a load test for the target workload. The canary configuration can be found\nat [apps/backend/canary.yaml](https://github.com/stefanprodan/gitops-istio/blob/main/apps/backend/canary.yaml).\n\n![Flagger Canary Release](https://raw.githubusercontent.com/fluxcd/flagger/main/docs/diagrams/flagger-canary-steps.png)\n\nPull the changes from GitHub:\n\n```sh\ngit pull origin main\n```\n\nTo trigger a canary deployment for the backend app, bump the container image:\n\n```bash\nyq e '.images[0].newTag=\"6.1.1\"' -i ./apps/backend/kustomization.yaml\n```\n\nCommit and push changes:\n\n```bash\ngit add -A \u0026\u0026 \\\ngit commit -m \"backend 6.1.1\" \u0026\u0026 \\\ngit push origin main\n```\n\nTell Flux to pull the changes or wait one minute for Flux to detect the changes on its own:\n\n```bash\nflux reconcile source git flux-system\n```\n\nWatch Flux reconciling your cluster to the latest commit:\n\n```bash\nwatch flux get kustomizations\n```\n\nAfter a couple of seconds, Flagger detects that the deployment revision changed and starts a new rollout:\n\n```bash\n$ kubectl -n prod describe canary backend\n\nEvents:\n\nNew revision detected! Scaling up backend.prod\nStarting canary analysis for backend.prod\nPre-rollout check conformance-test passed\nAdvance backend.prod canary weight 5\n...\nAdvance backend.prod canary weight 50\nCopying backend.prod template spec to backend-primary.prod\nPromotion completed! Scaling down backend.prod\n```\n\nDuring the analysis the canary’s progress can be monitored with Grafana. You can access the dashboard using port forwarding:\n\n```bash\nkubectl -n istio-system port-forward svc/flagger-grafana 3000:80\n```\n\nThe Istio dashboard URL is \nhttp://localhost:3000/d/flagger-istio/istio-canary?refresh=10s\u0026orgId=1\u0026var-namespace=prod\u0026var-primary=backend-primary\u0026var-canary=backend\n\n![Canary Deployment](https://raw.githubusercontent.com/fluxcd/flagger/main/docs/screens/demo-backend-dashboard.png)\n\nNote that if new changes are applied to the deployment during the canary analysis, Flagger will restart the analysis phase.\n\n## A/B testing\n\nBesides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions. \nIn an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users. \nThis is particularly useful for frontend applications that require session affinity.\n\nYou can enable A/B testing by specifying the HTTP match conditions and the number of iterations:\n\n```yaml\n  analysis:\n    # schedule interval (default 60s)\n    interval: 10s\n    # max number of failed metric checks before rollback\n    threshold: 10\n    # total number of iterations\n    iterations: 12\n    # canary match condition\n    match:\n      - headers:\n          user-agent:\n            regex: \".*Firefox.*\"\n      - headers:\n          cookie:\n            regex: \"^(.*?;)?(type=insider)(;.*)?$\"\n```\n\nThe above configuration will run an analysis for two minutes targeting Firefox users and those that \nhave an insider cookie. The frontend configuration can be found at `apps/frontend/canary.yaml`.\n\nTrigger a deployment by updating the frontend container image:\n\n```bash\nyq e '.images[0].newTag=\"6.1.1\"' -i ./apps/frontend/kustomization.yaml\n\ngit add -A \u0026\u0026 \\\ngit commit -m \"frontend 6.1.1\" \u0026\u0026 \\\ngit push origin main\n\nflux reconcile source git flux-system\n```\n\nFlagger detects that the deployment revision changed and starts the A/B testing:\n\n```bash\n$ kubectl -n istio-system logs deploy/flagger -f | jq .msg\n\nNew revision detected! Scaling up frontend.prod\nWaiting for frontend.prod rollout to finish: 0 of 1 updated replicas are available\nPre-rollout check conformance-test passed\nAdvance frontend.prod canary iteration 1/10\n...\nAdvance frontend.prod canary iteration 10/10\nCopying frontend.prod template spec to frontend-primary.prod\nWaiting for frontend-primary.prod rollout to finish: 1 of 2 updated replicas are available\nPromotion completed! Scaling down frontend.prod\n```\n\nYou can monitor all canaries with:\n\n```bash\n$ watch kubectl get canaries --all-namespaces\n\nNAMESPACE   NAME      STATUS        WEIGHT\nprod        frontend  Progressing   100\nprod        backend   Succeeded     0\n```\n\n## Rollback based on Istio metrics\n\nFlagger makes use of the metrics provided by Istio telemetry to validate the canary workload.\nThe frontend app [analysis](https://github.com/stefanprodan/gitops-istio/blob/main/apps/frontend/canary.yaml)\ndefines two metric checks: \n\n```yaml\n    metrics:\n      - name: error-rate\n        templateRef:\n          name: error-rate\n          namespace: istio-system\n        thresholdRange:\n          max: 1\n        interval: 30s\n      - name: latency\n        templateRef:\n          name: latency\n          namespace: istio-system\n        thresholdRange:\n          max: 500\n        interval: 30s\n```\n\nThe Prometheus queries used for checking the error rate and latency are located at\n[flagger-metrics.yaml](https://github.com/stefanprodan/gitops-istio/blob/main/istio/gateway/flagger-metrics.yaml).\n\nBump the frontend version to `6.1.2`, then during the canary analysis you can generate\nHTTP 500 errors and high latency to test Flagger's rollback.\n\nGenerate HTTP 500 errors:\n\n```bash\nwatch curl -b 'type=insider' http://\u003cINGRESS-IP\u003e/status/500\n```\n\nGenerate latency:\n\n```bash\nwatch curl -b 'type=insider' http://\u003cINGRESS-IP\u003e/delay/1\n```\n\nWhen the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, \nthe canary is scaled to zero and the rollout is marked as failed.\n\n```text\n$ kubectl -n istio-system logs deploy/flagger -f | jq .msg\n\nNew revision detected! Scaling up frontend.prod\nPre-rollout check conformance-test passed\nAdvance frontend.prod canary iteration 1/10\nHalt frontend.prod advancement error-rate 31 \u003e 1\nHalt frontend.prod advancement latency 2000 \u003e 500\n...\nRolling back frontend.prod failed checks threshold reached 10\nCanary failed! Scaling down frontend.prod\n```\n\nYou can extend the analysis with custom metric checks targeting\n[Prometheus](https://docs.flagger.app/usage/metrics#prometheus),\n[Datadog](https://docs.flagger.app/usage/metrics#datadog) and\n[Amazon CloudWatch](https://docs.flagger.app/usage/metrics#amazon-cloudwatch).\n\nFor configuring alerting of the canary analysis for Slack, MS Teams, Discord or Rocket see the\n[docs](https://docs.flagger.app/usage/alerting#canary-configuration).\n\n## Getting Help\n\nIf you have any questions about progressive delivery:\n\n* Invite yourself to the [CNCF community slack](https://slack.cncf.io/)\n  and join the [#flux](https://cloud-native.slack.com/messages/flux/) and [#flagger](https://cloud-native.slack.com/messages/flagger/) channels.\n* Check out the [Flux talks section](https://fluxcd.io/community/#talks) and to see a list of online talks,\n  hands-on training and meetups.\n\nYour feedback is always welcome!\n","funding_links":[],"categories":["Others","GitOps","Tutorials"],"sub_categories":["Secrets"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefanprodan%2Fgitops-istio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstefanprodan%2Fgitops-istio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefanprodan%2Fgitops-istio/lists"}