{"id":17211760,"url":"https://github.com/runyontr/k8s-canary","last_synced_at":"2026-03-27T02:12:29.963Z","repository":{"id":105092316,"uuid":"107264831","full_name":"runyontr/k8s-canary","owner":"runyontr","description":"Walkthrough of Canary deployment on Kubernetes","archived":false,"fork":false,"pushed_at":"2017-11-06T17:53:33.000Z","size":15,"stargazers_count":18,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-23T01:37:04.670Z","etag":null,"topics":["canary-deployment","golang","golang-application","kubernetes","kubernetes-tutorials","tutorial","tutorials"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/runyontr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-17T12:21:41.000Z","updated_at":"2023-08-21T14:40:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"6a02da84-9be7-44c7-b21b-37d92999141d","html_url":"https://github.com/runyontr/k8s-canary","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/runyontr/k8s-canary","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runyontr%2Fk8s-canary","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runyontr%2Fk8s-canary/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runyontr%2Fk8s-canary/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runyontr%2Fk8s-canary/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/runyontr","download_url":"https://codeload.github.com/runyontr/k8s-canary/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/runyontr%2Fk8s-canary/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31009588,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-27T01:56:05.093Z","status":"online","status_checked_at":"2026-03-27T02:00:08.055Z","response_time":164,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["canary-deployment","golang","golang-application","kubernetes","kubernetes-tutorials","tutorial","tutorials"],"created_at":"2024-10-15T02:58:26.193Z","updated_at":"2026-03-27T02:12:29.935Z","avatar_url":"https://github.com/runyontr.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# K8s Canary\n\nThis tutorial will walk through how to perform a canary deployment on [Kubernetes](https://kubernetes.io/). \nWhen updating an application, a slow rollout will allow for monitoring of traffic to the new\ndeployment and allow a rollback of the deployment with minimal impact to operations.\n\n\n\n\n\n\n## Kubernetes Services\n\nA [Service](https://kubernetes.io/docs/concepts/services-networking/service/) is an abstraction of the concept of pods. \nsince pods can be created and destroyed, a service provides a single host:port to talk to the pods\nconfigured for this service.  Services are created with a set of label selectors, and all requests\nmade to the service will be balanced to pods running with the configured label selectors.\n\n\nThere is no restriction that the pods behind a service need to come from the same deployment, and this will\nallow us to create a canary deployment to test an update before performing a full update.\n\n\n\n# The App\n\nThe applications we'll use to show the canary deployment are located in the `app` folder. The application\n uses an interface to abstract out the implementation of the AppInfo service.  \n \n \nThe application is supposed to provide information about how the application has been deployed on\nKubernetes.  It looks for environment variables containing metadata about the application and \nlabels on the container and provides them back as a json message.\n\n\n```go\npackage models \n\ntype AppInfo struct {\n\tPodName string //Extracts the MY_POD_NAME environment variable\n\tAppName string //Extracts the value for the app label \n\tNamespace string //Extracts the MY_POD_NAMESPACE environment variable\n\tRelease string //Extracts the value for the release label\n\tLabels  map[string]string //Contains other labels for app\n}\n```\n\n## App Structure\n\n### Models\n\nThe `app/models` folder contains the data transfer objects for the application.\n\n\n### Service\n\nThe `app/service` folder contains the interface definition and the implemenattions of the interface that \nextract the AppInfo from the runtime.  This folder contains three implementations of the interface.\n\n\n\n### Transport\n\nThe `app/transport` folder contains the [go-kit](https://github.com/go-kit/kit) code to host the \ninterface implementation as a service.\nThe `http.Handler` returned from `MakeInfoServiceHandler` is hosted to translate a request to `/v1/appinfo` to\nthe `GetAppInfo` function of the interface implementation and serialize the response back.\n\n \n \n### Dockerfiles\n\nThere are three dockerfiles in the `app` folder that correspond to images built with different implementations \nof the interface.  The following table shows which structs are being used in which Dockerfile, what tag these\nimages are pushed up with, and a brief description of what should be expected when running the application.\n\n\n\n Struct   |Dockerfile | Docker Image| Description |\n ---|---|---|---|\n appInfoBaseline | app/Dockerfile.v1 | `runyonsolutions/appinfo:1` | Does not return Namespace value.\n appInfoBroken | app/Dockerfile.v2 |`runyonsolutions/appinfo:2` | Returns an error\n appInfoWithNamespace | app/Dockerfile.v3 |`runyonsolutions/appinfo:3` | populates Namespace value correctly\n \n\n\n \n In practice, the application being deployed would probably not have three separate implementations\n  of the same  interface, but might represent iterations made to one implementation. \n   The use of three different structs simplifies showing the code running in each image.\n \n The application layout does show how easily it would be to change how a service functions and interchange different storage\n systems for basic CRUD applications, or different implementations of analytics. \n \n \n\n\n## Deployment\n\nAs a baseline, we assume there is a Kubernetes deployment named appinfo that was deployed from the \n`deployment/appinfo.yaml` file.  This deployment can be created via\n\n```\nkubectl create -f https://raw.githubusercontent.com/runyontr/k8s-canary/master/deployment/appinfo.yaml\n```\n\nThe deployment is very basic except for two additional configuration.  First is  [Exposing Pod Information through \nEnvironment Variables](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/). \nThese are created from the yaml in [Deployment YAML](deployment/appinfo.yaml)\n\n```yaml\n        env:\n        - name: MY_POD_NAME\n          valueFrom:\n            fieldRef:\n              fieldPath: metadata.name\n        - name: MY_POD_NAMESPACE\n          valueFrom:\n            fieldRef:\n              fieldPath: metadata.namespace\n```\n\nWhich create an environment variable `MY_POD_NAME` that contains the pod name and `MY_POD_NAMESPACE` \nwhich contains the namespace where the pod is running.\n\nThe second customization is [Exposing pod labels into a container](https://kubernetes-v1-4.github.io/docs/user-guide/downward-api/).\nwhich creates a file at `/etc/labels` containing the pods labels.\n\n\nNow get the name of the pod that was deployed:\n\n```\nkubectl get pods -l app=appinfo\n```\n\noutput:\n\n```\nNAME                       READY     STATUS    RESTARTS   AGE\nappinfo-567b989978-vvzlg   1/1       Running   0          19s\n\n```\n\n## Talking to the deployment\n\nNow we can port forward localhost:8080 to the pod's container port 8080 via\n\n```\nPOD_NAME=$(kubectl get pods -l app=appinfo -o jsonpath=\"{.items[0].metadata.name}\")\nkubectl port-forward $POD_NAME 8080:8080\n```\noutput\n```\nForwarding from 127.0.0.1:8080 -\u003e 80\nForwarding from [::1]:8080 -\u003e 80\n```\n\n\nOpen a new terminal and we can send an HTTP request:\n\n\n```\ncurl -s localhost:8080/v1/appinfo | jq .\n```\nand get a response like\n```\n{\n  \"PodName\": \"appinfo-567b989978-vvzlg\",\n  \"AppName\": \"appinfo\",\n  \"Namespace\": \"\",\n  \"Release\": \"stable\",\n  \"Labels\": {\n    \"pod-template-hash\": \"1236545534\"\n  }\n}\n\n``` \n\nIn a new terminal run this command to be constantly refreshing the AppInfo, which will be useful to show\nthe real time updates in the runtime in the following section.\n\n```\n while true; do clear; curl -s localhost:8080/v1/appinfo | jq . ; sleep 1; done;\n```\n\n\n### Changing labels\n\n\nTo see how the `/etc/labels` file is updated dynamically, we can add a new label to the pod and see\nthe output of our while loop get adjusted in real time\n\n```\nkubectl label pods $POD_NAME newlabel=realtime\n``` \n\n\n\nSwitch back to the first terminal to stop the port forwarding with CTRL+C.\n\n\n## Create Service\n\nA service provides load balancing to a selection of pods based on a particular label. \n The label we're going\nto filter by is `app=appinfo`.  To see the pods that satisfy this, run\n\n```\nkubectl get pods -l app=appinfo\n```\n\n\nThe service defined in `deployment/appinfo-service.yaml` has the label selector defined as `app=appinfo`. \n This will create\na load balancer (service) that routes requests to pods with the label `app=appinfo`. \n\n```\nkubectl create -f https://raw.githubusercontent.com/runyontr/k8s-canary/master/deployment/appinfo-service.yaml\n```\n\nand see it\n\n```\nkubectl get svc appinfo\n```\n\noutput:\n\n```\nNAME      CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE\nappinfo   10.32.0.232   \u003cnodes\u003e       8080:30753/TCP   14s\n```\n\nThere will now be an environment variables in all running containers that start after the creation\nof this service:\n\n```\nAPPINFO_SERVICE_PORT=8080\nAPPINFO_PORT_8080_TCP_PORT=8080\nAPPINFO_PORT=tcp://10.32.0.232:8080\nAPPINFO_PORT_8080_TCP_ADDR=10.32.0.232\n```\n\nwhich will allow all pods running in the cluster to have the connection information for this service readily available.\n\n\n\n### Accessing the Service\n\nThe type of service we created was [NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport).\n  \n  When this type of service is created, each node will proxy requests to a specific port on the node\n   (the same port on each node) to the configured port on the pod (8080 for our application).  \n \n To obtain the port on the node, run\n ```\n NODE_PORT=$(kubectl get svc appinfo \\\n   --output=jsonpath='{range .spec.ports[0]}{.nodePort}')\n ```\n \n \n### Cloud Firewall\n  When running on a cloud provider, a firewall rule might need to be added to allow TCP traffic into the\n  nodes at the service port.  To enable traffic in Google Cloud, the following command will open\n  traffic to the node port.\n```\ngcloud compute firewall-rules create appinfo-service \\\n  --allow=tcp:${NODE_PORT} \\\n  --network ${KUBERNETES_NETWORK_NAME}\n```\n\nwhere `${KUBERNETES_NETWORK_NAME}` is the name of the network Kubernetes is deployed on.\n\n\nNow if `${EXTERNAL_IP}` is the public address of one of the nodes in the cluster, then\nthe service is available at `http://${EXTERNAL_IP}:${NODE_PORT}`\n\n\n## Connecting\n\nSimilar to the while loop monitoring the port forwarded traffic of a particular pod in \n[Talking to the Deployment](#talking-to-the-deployment), the loop\n\n```\n while true; do clear; curl -s ${EXTERNAL_IP}:${NODE_PORT}/v1/appinfo | jq . ; sleep 1; done;\n```\n\nwill show the output of the service.  In order to monitor the deployments in the following section, \nthis command should be run in a new terminal.\n\n# Deploy an Update (Canary)\n\nLooking at the output of the loop, we see that the namespace field is not being populated correctly by the\ndeployment.  One proposed (failed) solution is captured in the implementation `appInfoBroken`, which\nsimulates a developers failed attempt at fixing the issue.  As new software can sometimes contain bugs, a slow\nrollout of the new version of the software will allow for minimal impact if there is an issue.\n\n\n\nTo simulate a real deployment,  we should have the currently deployed application scaled to handle\nthe current traffic.  When looking at how large to scale the current deployment (n), its helpful to \nunderstand the system's SLOs.  The new canary deployment will be getting 1/(n+1) of the traffic\ngoing to the service, and reducing the canary's load by increasing the value of n\n lower the impact to the error budget when\nthings go wrong.\n\nFor this tutorial, we scale to 3 replicas.\n\n```\nkubectl scale --replicas=3 deployment appinfo\ndeployment \"appinfo\" scaled\n```\n\n```\nkubectl get pods -l app=appinfo\nNAME                       READY     STATUS    RESTARTS   AGE\nappinfo-567b989978-6hbm8   1/1       Running   0          15s\nappinfo-567b989978-q58nt   1/1       Running   0          15s\nappinfo-567b989978-vvzlg   1/1       Running   0          14m\n```\n\nLooking at the output of the while loop look should now show the responses coming from pods with different names. \n Additionally,\nif any labels were applied to the first pod in [Chaning Labels](#changing-labels), the newly created pods\n will not have those labels.  This should be seen as responses will have different sets of labels.\n\n\nNow we are ready to deploy our broken canary deployment:\n\n\n```\nkubectl create -f https://raw.githubusercontent.com/runyontr/k8s-canary/master/deployment/canary-broken.yaml \ndeployment \"appinfo-canary-broken\" created\n```\n\nLooking at the running pods, we can now see 4 pods with the `app=appinfo` label:\n\n```\nkubectl get pods -l app=appinfo\nNAME                                    READY     STATUS    RESTARTS   AGE\nappinfo-567b989978-6hbm8                1/1       Running   0          10m\nappinfo-567b989978-q58nt                1/1       Running   0          10m\nappinfo-567b989978-vvzlg                1/1       Running   0          25m\nappinfo-canary-broken-c66665c44-7cfk6   1/1       Running   0          19s\n```\n\n\nNow looking at the loop in [Connection](#connecting) should have about 1/4 of the response coming back with a message of\n```json\n{\"error\": \"something went wrong\"}\n```\n and the other 3/4 of response should be responding as before.\n\n\n### Monitoring\n\nThis is when application monitoring (e.g. Prometheus/Grafana) would be able to split the metrics between canary\n and stable pods and show any difference in performance.  A future iteration of this tutorial will demonstrate\n the performance differences with a monitoring solution.\n\n\n### Rollback\n\nFor this demo, the while loop will be used to demonstrate the health of the system, and since\n  there are errors being returned in some responses, a rollback of the deployment is required:\n  \n\n```\nkubectl delete -f https://raw.githubusercontent.com/runyontr/k8s-canary/master/deployment/canary-broken.yaml\n```\n\nAt this point, the broken pod should be terminating:\n\n```\nkubectl get pods -l app=appinfo\nNAME                                    READY     STATUS        RESTARTS   AGE\nappinfo-567b989978-6hbm8                1/1       Running       0          12m\nappinfo-567b989978-q58nt                1/1       Running       0          12m\nappinfo-567b989978-vvzlg                1/1       Running       0          26m\nappinfo-canary-broken-c66665c44-7cfk6   1/1       Terminating   0          2m\n```\n\nand responses will return to being valid 100% of the time. In most shops, this would be enough to justify \na [Post Mortem](https://landing.google.com/sre/book/chapters/postmortem-culture.html) and improve any\ntesting process prior to being considered for a release.\n                                                             \n\n\n\n\n## Fix\n\nAfter figuring out the issue, a new fix has been created and is ready to be rolled out.  Following a similar\nprocess, a new canary deployment is created:\n\n```\nkubectl create -f https://raw.githubusercontent.com/runyontr/k8s-canary/master/deployment/canary-fixed.yaml \ndeployment \"appinfo-canary-fixed\" created\n```\n\n\n\nLooking at the output of the while loop now shows about 1/4 of the requests have the correct namespace value (`default`), and\nare reporting the `release=canary` label.  \n\n\n### Acceptance\nAt the point the team is willing to accept the new version formally, the configuration on the \n stable app would need to be updated to the docker image of the canary deployment:\n\n\n```\nkubectl set image deployment/appinfo appinfo-containers=runyonsolutions/appinfo:3\n```\n\nSince the image is being update this should fire off a rolling update of the deployment and new pods should be created\n\n```\nkubectl get pods -l app=appinfo\nNAME                                    READY     STATUS              RESTARTS   AGE\nappinfo-567b989978-6hbm8                1/1       Terminating         0          16m\nappinfo-567b989978-q58nt                1/1       Terminating         0          16m\nappinfo-567b989978-vvzlg                1/1       Running             0          30m\nappinfo-84d5cf794d-9dpfz                1/1       Running             0          5s\nappinfo-84d5cf794d-dd67v                1/1       Running             0          10s\nappinfo-84d5cf794d-lq45c                0/1       ContainerCreating   0          2s\nappinfo-95bccb844-jnfzn                 0/1       Terminating         3          1m\nappinfo-canary-fixed-744f96dc75-zbxr9   1/1       Running             0          2m\n```\n\nDescribing any of the newly created pods should show the update image.  The monitoring loop in [Connection](#connecting)\nshould should show labels `release=stable` having the namespace value correctly set.\n\nFinally, we need to clean up the canary app.\n\n```\nkubectl delete -f https://raw.githubusercontent.com/runyontr/k8s-canary/master/deployment/canary-fixed.yaml\n```\n\nNow all pods running behind the service are updated.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frunyontr%2Fk8s-canary","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frunyontr%2Fk8s-canary","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frunyontr%2Fk8s-canary/lists"}