{"id":25969392,"url":"https://github.com/thesephi/k8s-safe-deploy-poc","last_synced_at":"2025-03-04T22:48:03.506Z","repository":{"id":193933433,"uuid":"689760581","full_name":"Thesephi/k8s-safe-deploy-poc","owner":"Thesephi","description":"Demo for a safe k8s deployment practice","archived":false,"fork":false,"pushed_at":"2023-09-10T20:27:43.000Z","size":8,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-02T00:12:39.191Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Thesephi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-10T20:13:40.000Z","updated_at":"2023-09-10T20:18:50.000Z","dependencies_parsed_at":"2023-09-10T21:28:56.817Z","dependency_job_id":null,"html_url":"https://github.com/Thesephi/k8s-safe-deploy-poc","commit_stats":null,"previous_names":["thesephi/k8s-safe-deploy-poc"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Thesephi%2Fk8s-safe-deploy-poc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Thesephi%2Fk8s-safe-deploy-poc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Thesephi%2Fk8s-safe-deploy-poc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Thesephi%2Fk8s-safe-deploy-poc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Thesephi","download_url":"https://codeload.github.com/Thesephi/k8s-safe-deploy-poc/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241935238,"owners_count":20044826,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-04T22:48:02.759Z","updated_at":"2025-03-04T22:48:03.492Z","avatar_url":"https://github.com/Thesephi.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# POC: k8s atomic deployment (safety first)\n\nThis repo demonstrates how k8s deployment can be safe or unsafe depending on\nhow the `kubectl apply` command is used.\n\n# Prerequisites\n\nTo jump into the demo as smoothly as possible, it's recommended that the\nfollowing are installed beforehand:\n1. k8s itself (e.g. if using Docker Desktop, the built-in Kubernetes is enough)\n2. [yq](https://github.com/mikefarah/yq/releases) - a library to parse yaml (.yml) files\n3. [ingress-nginx for k8s](https://kubernetes.github.io/ingress-nginx/deploy/#quick-start) or an equivalence approach to help reach (e.g. `curl`) the k8s service\n\n# Setup\n\n### 1. Initialize everything\n\n```bash\n# run the initial deployment script\n./deploy.sh\n\n# expose the service via an ingress;\n# @NOTE this step is only needed if\n# `ingress-nginx` was used to access\n# the k8s service; if you use something\n# else, please expose your service in\n# accordance to your own setup\nkubectl apply -f ingress.yml\n```\n\n### 2. Check that everything works\n```\ncurl -s -I localhost/baz\nHTTP/1.1 200 OK\n```\n\n### 3. Take it further\nAssuming we now want to add a label called `version`\nto the k8s resources, and we've prepared the\nmanifest file `deploy-version.yml`, and now all\nthat's left is to run the deployment script. Let's\nsee how it works out as we do it the UNSAFE and SAFE\nways.\n\n#### 3.1. Run the UNSAFE deployment script\n```\n# this basically calls `kubectl apply -f deploy-version.yml` in an unsafe way\n./deploy-version-unsafe.sh\n\nnamespace/baz-service unchanged\nservice/baz-service configured\nThe Deployment \"baz-deployment\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app\":\"baz-service\", \"version\":\"2.0\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable\n```\n\nWhat happened here was that: k8s reconfigured the `baz-service` (to add the label `version=\"2.0\"`)\nbut it then failed to reconfigure the `baz-deployment`, due to the `field is immutable` error.\nWhy k8s behaves this way is beyond the scope of this demo, here we only care about the consequence\nof this behavior.\n\nIf we curl again, we no longer get the successful 200 OK response:\n```\ncurl -s -I localhost/baz\nHTTP/1.1 503 Service Temporarily Unavailable\n```\n\nThis proves that our previous deployment script was _destructive_ i.e. UNSAFE.\n\n#### 3.2. Reset to the last working state\n```bash\n# run the initial deployment script again\n./deploy.sh\n\n# side-note: we don't need to deploy the `Ingress` resource again as it's not\n# part of `deploy-version.yml`, and thus was not \"corrupted\" by the previous\n# unsafe deployment script\n```\n\nEnsure everything works again\n```\ncurl -s -I localhost/baz\nHTTP/1.1 200 OK\n```\n\n#### 3.3. Run the SAFE deployment script\n\n```\n./deploy-version-safe.sh\n\nnamespace/baz-service configured (dry run)\nservice/baz-service configured (dry run)\ndeployment.apps/baz-deployment configured (dry run)\nnamespace/baz-service unchanged (server dry run)\nservice/baz-service configured (server dry run)\nThe Deployment \"baz-deployment\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app\":\"baz-service\", \"version\":\"2.0\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable\n```\n\nAs you can see, the deployment still failed. The idea of this demo is not to fix this k8s error,\nbut to fail safely. This time, the `--dry-run` argument did its job well: the script \"failed early\",\nso it exited before it had the chance to corrupt the running system.\n\nLet's curl again to make sure things are still running well:\n```\ncurl -s -I localhost/baz\nHTTP/1.1 200 OK\n```\n\n### 4. Conclusion\n\nThis demo reproduced a scenario where applying a collection of k8s manifest (.yml file)\nwithout any special validation mechanism may lead to service disruption (system downtime).\n\nAnd one \"quick win\" is to make use of the `--dry-run` argument. It is literally free and effortless\nto do so, while the peace of mind it brings is hugely valuable. No service disruption (financial loss)\ncan ever justify the lack of properly applying an officially supported failsafe mechanism.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthesephi%2Fk8s-safe-deploy-poc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthesephi%2Fk8s-safe-deploy-poc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthesephi%2Fk8s-safe-deploy-poc/lists"}