{"id":20424950,"url":"https://github.com/stackzoo/khaos","last_synced_at":"2025-04-12T18:54:02.492Z","repository":{"id":209662118,"uuid":"724198777","full_name":"stackzoo/khaos","owner":"stackzoo","description":"A lightweight kubernetes operator to test cluster resilience via chaos engineering 💣  ☸️","archived":false,"fork":false,"pushed_at":"2025-03-13T00:26:41.000Z","size":1736,"stargazers_count":27,"open_issues_count":2,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-26T13:11:20.454Z","etag":null,"topics":["chaos-engineering","golang","kubebuilder","kubernetes","kubernetes-operator"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stackzoo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-27T15:41:00.000Z","updated_at":"2024-11-06T15:20:55.000Z","dependencies_parsed_at":"2023-11-28T15:30:53.479Z","dependency_job_id":"4d39d671-5ec5-4ef3-8491-ef412ffeb70e","html_url":"https://github.com/stackzoo/khaos","commit_stats":null,"previous_names":["stackzoo/khaos"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackzoo%2Fkhaos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackzoo%2Fkhaos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackzoo%2Fkhaos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackzoo%2Fkhaos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stackzoo","download_url":"https://codeload.github.com/stackzoo/khaos/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248618263,"owners_count":21134200,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chaos-engineering","golang","kubebuilder","kubernetes","kubernetes-operator"],"created_at":"2024-11-15T07:11:36.055Z","updated_at":"2025-04-12T18:54:02.472Z","avatar_url":"https://github.com/stackzoo.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KHAOS\n[![CI](https://github.com/stackzoo/khaos/actions/workflows/ci.yaml/badge.svg)](https://github.com/stackzoo/khaos/actions/workflows/ci.yaml)  [![releaser](https://github.com/stackzoo/khaos/actions/workflows/release.yaml/badge.svg)](https://github.com/stackzoo/khaos/actions/workflows/release.yaml)  ![GitHub last commit (branch)](https://img.shields.io/github/last-commit/stackzoo/khaos/main)  \n[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)  [![Latest Release](https://img.shields.io/github/v/release/stackzoo/khaos?logo=github)](https://github.com/stackzoo/khaos/releases/latest)  [![Go version](https://img.shields.io/github/go-mod/go-version/stackzoo/khaos.svg)](https://github.com/stackzoo/khaos)  [![Go Report Card](https://goreportcard.com/badge/github.com/stackzoo/khaos)](https://goreportcard.com/report/github.com/stackzoo/khaos)  \n\n\n\n\u003cimg src=\"docs/images/logo4.png\" alt=\"logo\" width=\"210\" height=\"210\"\u003e  \n\nA lightweight kubernetes operator to test cluster and application resilience via chaos engineering 💣 ☸️  \n\n## Abstract\n**Khaos** (pun intended) is a straightforward Kubernetes [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) made with [kubebuilder](https://github.com/kubernetes-sigs/kubebuilder) and designed for executing [Chaos Engineering](https://en.wikipedia.org/wiki/Chaos_engineering) activities.  \nThrough the implementation of custom controllers and resources, Khaos facilitates the configuration and automation of operations such as the targeted deletion of pods within a specified namespace, the removal of nodes from the cluster, the deletion of secrets and more.  \nKhaos is an **unopinionated** operator, meaning that it only provides simple and *atomic primitives* that engineers can use as building blocks in order to compose their preferred chaos strategy.  \nCurrently, Khaos does not implement *cronjobs*; any scheduling of Khaos Custom Resources is delegated to external logic outside the cluster, possibly through a *GitOps* approach.  \n\n\u003e [!WARNING]  \n\u003e This operator will introduce faults and unpredicatbility in your infrastructure, use with caution.  \n\n## Supported features\n- [X] Delete pods\n- [X] Random scaling pod replicas\n- [x] Delete cluster nodes\n- [X] Delete secrets\n- [X] Delete configmaps\n- [X] Cordon nodes\n- [X] Taint nodes\n- [X] Inject resource constraints in pods\n- [X] Add o remove labels in pods\n- [X] Flood api server with calls\n- [X] Consume resources in a namespace\n- [X] Create custom kubernetes events\n- [X] Exec commands inside pods (**experimental**).  \n\n\n\n## Local Testing and Debugging\nFirst of all clone the repository:  \n```console\ngit clone https://github.com/stackzoo/khaos \u0026\u0026 cd khaos\n```  \n\nThe repo contains a `Makefile` with all that you need.  \nInspect the make targets with the following command:  \n```console\nmake help\n\nUsage:\n  make \u003ctarget\u003e\n\nGeneral\n  help             Display this help.\n\nDevelopment\n  manifests        Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.\n  generate         Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.\n  fmt              Run go fmt against code.\n  vet              Run go vet against code.\n  cluster-up       Create a kind cluster named \"test-operator-cluster\" with a master and 3 worker nodes.\n  cluster-down     Delete the kind cluster named \"test-operator-cluster\".\n  test             Run tests.\n  lint             Run golangci-lint linter \u0026 yamllint\n  lint-fix         Run golangci-lint linter and perform fixes\n\nBuild\n  build            Build manager binary.\n  run              Run a controller from your host.\n  docker-build     Build docker image with the manager.\n  docker-push      Push docker image with the manager.\n  docker-buildx    Build and push docker image for the manager for cross-platform support\n\nDeployment\n  install          Install CRDs into the K8s cluster specified in ~/.kube/config.\n  uninstall        Uninstall CRDs from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.\n  deploy           Deploy controller to the K8s cluster specified in ~/.kube/config.\n  undeploy         Undeploy controller from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.\n  helmify          Download helmify locally if necessary.\n  helm             Produce operator helm charts\n\nBuild Dependencies\n  kustomize        Download kustomize locally if necessary. If wrong version is installed, it will be removed before downloading.\n  controller-gen   Download controller-gen locally if necessary. If wrong version is installed, it will be overwritten.\n  envtest          Download envtest-setup locally if necessary.\n```   \n\nYou can spin up a local dev cluster with [KinD](https://kind.sigs.k8s.io/) via the following command:  \n```console\nmake cluster-up\n```   \n\nInstall and list all the available operator's CRDs with the following command:  \n```console\nmake manifests \u0026\u0026 make install \u0026\u0026 kubectl get crds\n\nNAME                                       CREATED AT\napiserveroverloads.khaos.stackzoo.io          2024-01-21T15:29:36Z\ncommandinjections.khaos.stackzoo.io           2024-01-21T15:29:36Z\nconfigmapdestroyers.khaos.stackzoo.io         2024-01-21T15:29:36Z\nconsumenamespaceresources.khaos.stackzoo.io   2024-01-21T15:29:36Z\ncontainerresourcechaos.khaos.stackzoo.io      2024-01-21T15:29:36Z\ncordonnodes.khaos.stackzoo.io                 2024-01-21T15:29:36Z\neventsentropies.khaos.stackzoo.io             2024-01-21T15:29:36Z\nnodedestroyers.khaos.stackzoo.io              2024-01-21T15:29:36Z\nnodetainters.khaos.stackzoo.io                2024-01-21T15:29:36Z\npoddestroyers.khaos.stackzoo.io               2024-01-21T15:29:36Z\npodlabelchaos.khaos.stackzoo.io               2024-01-21T15:29:36Z\nrandomscalings.khaos.stackzoo.io              2024-01-21T15:29:36Z\nsecretdestroyers.khaos.stackzoo.io            2024-01-21T15:29:36Z\n```  \n\nIn order to run the operator on your cluster (current context - i.e. whatever cluster `kubectl cluster-info` shows) run:  \n```console\nmake run\n```  \n\n\nIn order to debug this project locally, I strongly suggest using [vscode](https://code.visualstudio.com/).  \n\nIn vscode you need to create a `.vscode/launch.json` file similar to the following:  \n```json\n{\n    \"version\": \"0.2.0\",\n    \"configurations\": [\n      {\n        \"name\": \"Debug Khaos Operator\",\n        \"type\": \"go\",\n        \"request\": \"launch\",\n        \"mode\": \"auto\",\n        \"program\": \"${workspaceFolder}/cmd/main.go\",\n        \"args\": []\n      }\n    ]\n  }\n```   \n\n\n\n## Some Examples\n\nIn order to test the following examples, you can use the local *KinD* cluster (see the `Local Testing and Debugging` section).  \nOnce you have the cluster up and running, procede to create a new namespace called `prod` and apply an example deployment:  \n\n```console\nkubectl create namespace prod \u0026\u0026 kubectl apply -f examples/test-deployment.yaml\n```  \n\nNow you can procede with the examples!  \n\n\u003cdetails\u003e\n  \u003csummary\u003eDELETE PODS\u003c/summary\u003e\n\nWait for all the pods in the `prod` namespace to be up and running and then apply the `PodDestroyer` manifest:  \n\n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: PodDestroyer\nmetadata:\n  name: nginx-destroyer\nspec:\n  selector:\n    matchLabels:\n      app: nginx\n  maxPods: 3\n  namespace: prod\n```  \n\n\n\n```console\nkubectl apply -f examples/pod-destroyer.yaml\n```\n\nNow you can observe 2 things:  \n1. the pods in prod namespace are being Terminated (and recreated by the replicaset):  \n```console\nNAME                                READY   STATUS              RESTARTS   AGE\nnginx-deployment-7bf8c77b5b-5fvrc   1/1     Running             0          6s\nnginx-deployment-7bf8c77b5b-5qcx4   1/1     Running             0          6s\nnginx-deployment-7bf8c77b5b-6kmbd   0/1     ContainerCreating   0          6s\nnginx-deployment-7bf8c77b5b-75bg6   1/1     Running             0          6s\nnginx-deployment-7bf8c77b5b-bcbk5   1/1     Running             0          6s\nnginx-deployment-7bf8c77b5b-f5wkh   1/1     Running             0          6s\nnginx-deployment-7bf8c77b5b-gfdzl   1/1     Running             0          6s\nnginx-deployment-7bf8c77b5b-gmhr2   1/1     Running             0          6s\nnginx-deployment-7bf8c77b5b-gsprh   1/1     Terminating         0          6s\nnginx-deployment-7bf8c77b5b-hvsff   1/1     Running             0          6s\nnginx-deployment-7bf8c77b5b-v4j9v   0/1     ContainerCreating   0          6s\nnginx-deployment-7bf8c77b5b-zxxv7   0/1     Terminating         0          6s\nnginx-deployment-7bf8c77b5b-6kmbd   1/1     Running             0          6s\nnginx-deployment-7bf8c77b5b-zxxv7   0/1     Terminating         0          6s\nnginx-deployment-7bf8c77b5b-zxxv7   0/1     Terminating         0          6s\nnginx-deployment-7bf8c77b5b-zxxv7   0/1     Terminating         0          6s\nnginx-deployment-7bf8c77b5b-v4j9v   1/1     Running             0          7s\nnginx-deployment-7bf8c77b5b-gsprh   0/1     Terminating         0          32s\nnginx-deployment-7bf8c77b5b-gsprh   0/1     Terminating         0          33s\nnginx-deployment-7bf8c77b5b-gsprh   0/1     Terminating         0          33s\nnginx-deployment-7bf8c77b5b-gsprh   0/1     Terminating         0          33s\n```  \n2. Our operator shows the reconciliation logic's logs:  \n```console   \n2023-11-28T14:07:18+01:00       INFO    Reconciling PodDestroyer: default/nginx-destroyer       {\"controller\": \"poddestroyer\", \"controllerGroup\": \"khaos.stackzoo.io\", \"controllerKind\": \"PodDestroyer\", \"PodDestroyer\": {\"name\":\"nginx-destroyer\",\"namespace\":\"default\"}, \"namespace\": \"default\", \"name\": \"nginx-destroyer\", \"reconcileID\": \"1e16a7d2-825a-4b46-b4e5-ac1228bc1c36\"}\n2023-11-28T14:07:18+01:00       INFO    Selector: {map[app:nginx] []}   {\"controller\": \"poddestroyer\", \"controllerGroup\": \"khaos.stackzoo.io\", \"controllerKind\": \"PodDestroyer\", \"PodDestroyer\": {\"name\":\"nginx-destroyer\",\"namespace\":\"default\"}, \"namespace\": \"default\", \"name\": \"nginx-destroyer\", \"reconcileID\": \"1e16a7d2-825a-4b46-b4e5-ac1228bc1c36\"}\n2023-11-28T14:07:18+01:00       INFO    MaxPods: 3      {\"controller\": \"poddestroyer\", \"controllerGroup\": \"khaos.stackzoo.io\", \"controllerKind\": \"PodDestroyer\", \"PodDestroyer\": {\"name\":\"nginx-destroyer\",\"namespace\":\"default\"}, \"namespace\": \"default\", \"name\": \"nginx-destroyer\", \"reconcileID\": \"1e16a7d2-825a-4b46-b4e5-ac1228bc1c36\"}\n2023-11-28T14:07:18+01:00       INFO    Namespace: prod {\"controller\": \"poddestroyer\", \"controllerGroup\": \"khaos.stackzoo.io\", \"controllerKind\": \"PodDestroyer\", \"PodDestroyer\": {\"name\":\"nginx-destroyer\",\"namespace\":\"default\"}, \"namespace\": \"default\", \"name\": \"nginx-destroyer\", \"reconcileID\": \"1e16a7d2-825a-4b46-b4e5-ac1228bc1c36\"}\n```  \n\nNow we can inspect the status of our PodDestroyer object:  \n```console \nkubectl get poddestroyer\n\nNAME              AGE\nnginx-destroyer   4m51s\n```  \n\n```console\nkubectl get poddestroyer nginx-destroyer -o yaml\n```  \nThis will retrieve our resource in `yaml` format:  \n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: PodDestroyer\nmetadata:\n  annotations:\n    kubectl.kubernetes.io/last-applied-configuration: |\n      {\"apiVersion\":\"khaos.stackzoo.io/v1alpha1\",\"kind\":\"PodDestroyer\",\"metadata\":{\"annotations\":{},\"name\":\"nginx-destroyer\",\"namespace\":\"default\"},\"spec\":{\"maxPods\":9,\"namespace\":\"prod\",\"selector\":{\"matchLabels\":{\"app\":\"nginx\"}}}}\n  creationTimestamp: \"2023-11-28T13:07:18Z\"\n  generation: 1\n  name: nginx-destroyer\n  namespace: default\n  resourceVersion: \"2009\"\n  uid: fbba6287-6f70-406b-821e-9000f097afc5\nspec:\n  MaxPods: 3\n  namespace: prod\n  selector:\n    matchLabels:\n      app: nginx\nstatus:\n  numPodsDestroyed: 7\n```  \n\nThe `status` spec tells you how many pods have been successfully destroyed.  \n\n\n\u003c/details\u003e  \n\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eRANDOM SCALING POD REPLICAS\u003c/summary\u003e\n\n\nApply an example deployment:  \n\n\n```console\nkubectl apply -f examples/random-scaling-test-deployment.yaml\n```  \n\nRetrieve our deployment's pods in the default namespace:  \n```console\nkubectl get pods\n\nNAME                                         READY   STATUS    RESTARTS   AGE\nrandom-scaling-deployment-56c5d5bb74-pcgm6   1/1     Running   0          52s\nrandom-scaling-deployment-56c5d5bb74-rw4sp   1/1     Running   0          52s\nrandom-scaling-deployment-56c5d5bb74-tpvxb   1/1     Running   0          52s\n```  \n\nNow apply the following `RandomScaling` manifest:  \n\n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: RandomScaling\nmetadata:\n  name: example-randomscaling\nspec:\n  deployment: random-scaling-deployment\n  minReplicas: 2\n  maxReplicas: 13\n```   \n\n```console\nkubectl apply -f examples/random-scaling.yaml\n```   \n\nThis will scale our deployment by randomly picking a number between `minReplicas` and `maxReplicas`.  \n\n\nCheck again our pods:  \n```console\nkubectl get pods\n\nNAME                                         READY   STATUS    RESTARTS   AGE\nrandom-scaling-deployment-56c5d5bb74-2tcss   1/1     Running   0          2m49s\nrandom-scaling-deployment-56c5d5bb74-8c5gf   1/1     Running   0          2m49s\nrandom-scaling-deployment-56c5d5bb74-bjpkc   1/1     Running   0          2m49s\nrandom-scaling-deployment-56c5d5bb74-cctcz   1/1     Running   0          2m49s\nrandom-scaling-deployment-56c5d5bb74-pcgm6   1/1     Running   0          5m44s\nrandom-scaling-deployment-56c5d5bb74-rw4sp   1/1     Running   0          5m44s\nrandom-scaling-deployment-56c5d5bb74-tpvxb   1/1     Running   0          5m44s\n```  \n\nYou can notice that there are 4 more pods!  \n\nOur operator shows the reconciliation logic's logs:  \n```console   \n2024-01-21T17:47:32+01:00       INFO    Starting reconcile for random scaling - deployment random-scaling-deployment    {\"controller\": \"randomscaling\", \"controllerGroup\": \"khaos.stackzoo.io\", \"controllerKind\": \"RandomScaling\", \"RandomScaling\": {\"name\":\"example-randomscaling\",\"namespace\":\"default\"}, \"namespace\": \"default\", \"name\": \"example-randomscaling\", \"reconcileID\": \"4cda6061-a893-470a-8a43-1a222256d987\"}\n2024-01-21T17:47:32+01:00       INFO    RandomReplicas 7       {\"controller\": \"randomscaling\", \"controllerGroup\": \"khaos.stackzoo.io\", \"controllerKind\": \"RandomScaling\", \"RandomScaling\": {\"name\":\"example-randomscaling\",\"namespace\":\"default\"}, \"namespace\": \"default\", \"name\": \"example-randomscaling\", \"reconcileID\": \"4cda6061-a893-470a-8a43-1a222256d987\"}\n```  \n\nNow we can inspect the status of our PodDestroyer object:  \n```console \nkubectl get randomscaling example-randomscaling -o yaml\n```  \n\nThis will retrieve our resource in `yaml` format:  \n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: RandomScaling\nmetadata:\n  annotations:\n    kubectl.kubernetes.io/last-applied-configuration: |\n      {\"apiVersion\":\"khaos.stackzoo.io/v1alpha1\",\"kind\":\"RandomScaling\",\"metadata\":{\"annotations\":{},\"name\":\"example-randomscaling\",\"namespace\":\"default\"},\"spec\":{\"deployment\":\"random-scaling-deployment\",\"maxReplicas\":10,\"minReplicas\":1}}\n  creationTimestamp: \"2024-01-21T16:46:54Z\"\n  generation: 5\n  name: example-randomscaling\n  namespace: default\n  resourceVersion: \"1865\"\n  uid: 4197f351-4557-4033-b996-fd5f0a8e25fc\nspec:\n  deployment: random-scaling-deployment\n  maxReplicas: 10\n  minReplicas: 1\nstatus:\n  operationResult: true\n```  \n\nThe `status` spec tells you that the last trigger has been succesfully completed.  \n\n\n\u003c/details\u003e  \n\n\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eDELETE NODES\u003c/summary\u003e\n\nFirst, retrieve nodes info for your cluster:  \n```console\nkubectl get nodes\n\nNAME                                  STATUS   ROLES           AGE   VERSION\ntest-operator-cluster-control-plane   Ready    control-plane   24m   v1.27.3\ntest-operator-cluster-worker          Ready    \u003cnone\u003e          24m   v1.27.3\ntest-operator-cluster-worker2         Ready    \u003cnone\u003e          24m   v1.27.3\ntest-operator-cluster-worker3         Ready    \u003cnone\u003e          24m   v1.27.3\n\n```  \n\nNow apply the following `NodeDestroyer` manifest:  \n\n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: NodeDestroyer\nmetadata:\n  name: example-node-destroyer\nspec:\n  nodeNames:\n    - test-operator-cluster-worker\n    - test-operator-cluster-worker3\n```\n\n```console\nkubectl apply -f examples/node-destroyer.yaml\n```\n\nNow, once again, retrieve the node list from the kuber-apiserver:  \n```console\nkubectl get nodes\n\nNAME                                  STATUS   ROLES           AGE   VERSION\ntest-operator-cluster-control-plane   Ready    control-plane   25m   v1.27.3\ntest-operator-cluster-worker2         Ready    \u003cnone\u003e          25m   v1.27.3\n\n```  \n\nAs you can see the operator succesfully removed the specified nodes.  \n\n\n\u003c/details\u003e  \n\n\u003cdetails\u003e\n  \u003csummary\u003eTAINT NODES\u003c/summary\u003e\n\nFirst, retrieve nodes info from your cluster:  \n```console\nkubectl get nodes\n\nNAME                                  STATUS   ROLES           AGE   VERSION\ntest-operator-cluster-control-plane   Ready    control-plane   2m37s   v1.27.3\ntest-operator-cluster-worker          Ready    \u003cnone\u003e          2m15s   v1.27.3\ntest-operator-cluster-worker2         Ready    \u003cnone\u003e          2m16s   v1.27.3\ntest-operator-cluster-worker3         Ready    \u003cnone\u003e          2m17s   v1.27.3\n\n```  \n\nRetrieve the annotations for the test-operator-cluster-worker3 node:  \n```console\nkubectl get node test-operator-cluster-worker3 -o=jsonpath='{.spec.taints}' | jq\n```  \nThe previous command should return nothing as our node has no taints.  \n\n\nNow apply the following `NodeTainter` manifest:  \n\n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: NodeTainter\nmetadata:\n  name: example-node-tainter\nspec:\n  nodeNames:\n    - test-operator-cluster-worker\n    - test-operator-cluster-worker3\n```\n\n```console\nkubectl apply -f examples/node-tainter.yaml\n```  \nCheck the operator's logs:  \n```console\n2024-01-17T08:54:47+01:00\tINFO\tReconciling NodeTainter: default/example-node-tainter\t{\"controller\": \"nodetainter\", \"controllerGroup\": \"khaos.stackzoo.io\", \"controllerKind\": \"NodeTainter\", \"NodeTainter\": {\"name\":\"example-node-tainter\",\"namespace\":\"default\"}, \"namespace\": \"default\", \"name\": \"example-node-tainter\", \"reconcileID\": \"1c270341-0b1d-4675-8188-38e82f3ccc9e\"}\n2024-01-17T08:54:47+01:00\tINFO\tNode Names: [test-operator-cluster-worker test-operator-cluster-worker3]\t{\"controller\": \"nodetainter\", \"controllerGroup\": \"khaos.stackzoo.io\", \"controllerKind\": \"NodeTainter\", \"NodeTainter\": {\"name\":\"example-node-tainter\",\"namespace\":\"default\"}, \"namespace\": \"default\", \"name\": \"example-node-tainter\", \"reconcileID\": \"1c270341-0b1d-4675-8188-38e82f3ccc9e\"}\n```  \n\n\nNow, once again, retrieve the tain on the node:  \n```json\nkubectl get node test-operator-cluster-worker3 -o=jsonpath='{.spec.taints}' | jq\n\n[\n  {\n    \"effect\": \"NoSchedule\",\n    \"key\": \"khaos.io/tainted\",\n    \"value\": \"true\"\n  }\n]\n\n```  \n\nAs you can see the operator succesfully tainted the specified nodes.  \n\n\n\u003c/details\u003e  \n\n\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eDELETE SECRETS\u003c/summary\u003e\n\nFirst create a new kubernetes secret (empty secret is fine):  \n\n```console\nkubectl -n prod create secret generic test-secret\n\nsecret/test-secret created\n```  \n\nNow apply the following `SecretDestroyer` manifest:  \n\n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: SecretDestroyer\nmetadata:\n  name: example-secret-destroyer\nspec:\n  namespace: prod\n  secretNames:\n    - test-secret\n```\n\n```console\nkubectl apply -f examples/secret-destroyer.yaml\n```  \n\nTry to list all the secrets in the `prod` namespace:  \n```console\nkubectl -n prod get secrets\n\nNo resources found in prod namespace.\n```  \n\nThe specified secret was successfully removed.  \n\n\n\n\u003c/details\u003e  \n\n\n\u003cdetails\u003e\n  \u003csummary\u003eDELETE CONFIGMAPS\u003c/summary\u003e\n\nFirst create a new kubernetes configmap:  \n\n```console\nkubectl create configmap test-configmap --namespace=prod --from-literal=message=ready \u0026\u0026 kubectl -n prod get configmap\n\nconfigmap/test-configmap created\n\nNAME               DATA   AGE\nkube-root-ca.crt   1      2m24s\ntest-configmap     1      1s\n\n```  \n\nNow apply the following `ConfigMapDestroyer` manifest:  \n\n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: ConfigMapDestroyer\nmetadata:\n  name: example-configmap-destroyer\nspec:\n  namespace: prod\n  configMapNames:\n    - test-configmap\n```\n\n```console\nkubectl apply -f examples/config-map-destroyer.yaml\n```  \n\nTry to list all the configmaps in the `prod` namespace:  \n```console\nkubectl -n prod get configmap\n\nNAME               DATA   AGE\nkube-root-ca.crt   1      9m26s\n```  \n\nThe specified configmap was successfully removed.  \n\n\u003c/details\u003e  \n\n\n\u003cdetails\u003e\n  \u003csummary\u003eAPPLY NEW CONTAINER RESOURCE LIMITS\u003c/summary\u003e  \n\nApply the following `ContainerResourceChaos` manifest:  \n\n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: ContainerResourceChaos\nmetadata:\n  name: example-container-resource-chaos\n  namespace: prod\nspec:\n  namespace: prod\n  DeploymentName: nginx-deployment\n  containerName: nginx\n  maxCPU: \"666m\"\n  maxRAM: \"512Mi\"\n\n```  \n\n```console\nkubectl apply -f examples/container-resource-chaos.yaml\n```  \n\nNow retrieve one of the pod in the prod namespace in `yaml` format and take a look at the resources:  \n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  creationTimestamp: \"2023-11-28T13:43:37Z\"\n  generateName: nginx-deployment-c54b8b4b4-\n  labels:\n    app: nginx\n    pod-template-hash: c54b8b4b4\n  name: nginx-deployment-c54b8b4b4-jvw4k\n  namespace: prod\n  ownerReferences:\n  - apiVersion: apps/v1\n    blockOwnerDeletion: true\n    controller: true\n    kind: ReplicaSet\n    name: nginx-deployment-c54b8b4b4\n    uid: a73e8483-a51b-4f43-806d-38b8976ee61d\n  resourceVersion: \"6128\"\n  uid: 6be9fe17-f6b8-418b-96a1-bdf70da8eb95\nspec:\n  containers:\n  - image: nginx:latest\n    imagePullPolicy: Always\n    name: nginx\n    resources: # modified\n      limits:\n        cpu: 666m\n        memory: 512Mi\n      requests:\n        cpu: 666m\n        memory: 512Mi\n```   \n\n\n\u003c/details\u003e  \n\n\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eMODIFY POD LABELS\u003c/summary\u003e  \n\nApply the following `PodLabelChaos` manifest:  \n\n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: PodLabelChaos\nmetadata:\n  name: podlabelchaos-test\nspec:\n  deploymentName: nginx-deployment\n  namespace: prod\n  labels:\n    chaos: \"true\"\n  addLabels: true\n\n```  \n\n```console\nkubectl apply -f examples/pod-label-chaos.yaml\n```  \n\nNow retrieve one of the pod in the prod namespace in `yaml` format and take a look at the labels:  \n```yaml\n\napiVersion: v1\nkind: Pod\nmetadata:\n  creationTimestamp: \"2023-11-28T15:27:22Z\"\n  generateName: nginx-deployment-6bb89bf6cd-\n  labels:\n    app: nginx\n    chaos: \"true\"\n    pod-template-hash: 6bb89bf6cd\n  name: nginx-deployment-6bb89bf6cd-52j42\n  namespace: prod\n\n```   \n\n\n\u003c/details\u003e  \n\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eCONSUME NAMESPACE RESOURCES\u003c/summary\u003e  \nThis feature of the operator will spin up a busybox deployment with the specified replicas in the specified namespace.  \nAll the busybox's pod will execute the following command:  \n\n```console\nwhile true; do echo 'Doing extensive tasks'; sleep 1; done\n```  \n\n\nFirst of all we need to install the **metrics server** on our cluster:  \n```console\nkubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml  \\\n\u0026\u0026 kubectl patch -n kube-system deployment metrics-server --type=json -p '[{\"op\":\"add\",\"path\":\"/spec/template/spec/containers/0/args/-\",\"value\":\"--kubelet-insecure-tls\"}]'\n```   \nWait for the metric server pod to be up and running and check cluster (nodes) resources:  \n```console\nkubectl top nodes\n\nNAME                                  CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%\ntest-operator-cluster-control-plane   221m         2%     697Mi           4%\ntest-operator-cluster-worker          31m          0%     230Mi           1%\ntest-operator-cluster-worker2         29m          0%     253Mi           1%\ntest-operator-cluster-worker3         42m          0%     242Mi           1%\n```  \n\n\nNow apply the following `ConsumeNamespaceResources` manifest:  \n\n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: ConsumeNamespaceResources\nmetadata:\n  name: example-consume-resources\nspec:\n  targetNamespace: prod\n  numPods: 200\n\n```  \n\n```console\nkubectl apply -f examples/consume-namespace-resources.yaml  \n```  \n\nLet's inspect the deployment in the `prod` namespace:  \n```console\nkubectl -n prod get deployment\n\nNAME                 READY   UP-TO-DATE   AVAILABLE     AGE\nbusybox-deployment   200/200   80           80          44s\nnginx-deployment     10/10     10           10          10m\n```  \n\nLet's now review the nodes usagge:  \n```console\nkubectl top nodes\n\nNAME                                  CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   \ntest-operator-cluster-control-plane   845m         10%    904Mi           5%        \ntest-operator-cluster-worker          1790m        22%    938Mi           5%        \ntest-operator-cluster-worker2         1494m        18%    1039Mi          6%        \ntest-operator-cluster-worker3         1673m        20%    1045Mi          6%\n```  \n\nAs we can see, our deployment in the *prod* namespace is consuming resources!  \nNow try deleting the `ConsumeNamespaceResources` object:  \n```console\nkubectl delete -f examples/consume-namespace-resources.yaml\n\nconsumenamespaceresources.khaos.stackzoo.io \"example-consume-resources\" deleted\n```   \n\nCheck the operator's logs:  \n\n```console\n2023-11-30T15:45:40+01:00       INFO    Object deleted, finalizing resources    {\"controller\": \"consumenamespaceresources\", \"controllerGroup\": \"khaos.stackzoo.io\", \"controllerKind\": \"ConsumeNamespaceResources\", \"ConsumeNamespaceResources\": {\"name\":\"example-consume-resources\",\"namespace\":\"default\"}, \"namespace\": \"default\", \"name\": \"example-consume-resources\", \"reconcileID\": \"b35fdd79-5308-4080-a718-027e2d9d7d13\"}\n```  \n\nThe resource's controller contains a finalizer and it is deleting our busybox deployment in the *prod* namespace!  \nCheck the deployments in the *prod* namespace:  \n```console\nkubectl -n prod get deployment\n\nNAME               READY   UP-TO-DATE   AVAILABLE   AGE\nnginx-deployment   10/10   10           10          21m\n```  \nCool, our deployment has been successfully deleted.  \n\n\n\n\u003c/details\u003e  \n\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eCREATE CUSTOM KUBERNETES EVENTS\u003c/summary\u003e  \n\nApply the following `EventsEntropy` manifest:  \n\n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: EventsEntropy\nmetadata:\n  name: example-eventsentropy\nspec:\n  events:\n    - \"Custom event 1 with some gibberish - dfsdfsdffdgt egeg4e 😊\"\n    - \"Custom event 2 - with some gibberish dfsdfsdffdgt 676565 🥴\"\n    - \"Custom event 3 - with some gibberish 8/ihfwgf sufdh  🤪\"\n\n```  \n\n```console\nkubectl apply -f examples/events-entropy.yaml\n```  \n\nNow retrieve kubernetes events via kubectl:  \n```console\nkubectl get events | grep gibberish\n\n\u003cunknown\u003e               Custom event 1 with some gibberish - dfsdfsdffdgt egeg4e 😊\n\u003cunknown\u003e               Custom event 3 - with some gibberish 8/ihfwgf sufdh  🤪\n\u003cunknown\u003e               Custom event 2 - with some gibberish dfsdfsdffdgt 676565 🥴\n\n```   \n\n\n\u003c/details\u003e  \n\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eCORDON NODES\u003c/summary\u003e  \n\nApply the following `CordonNodes` manifest:  \n\n```yaml\napiVersion: khaos.stackzoo.io/v1alpha1\nkind: CordonNode\nmetadata:\n  name: example-cordon-node\nspec:\n  nodesToCordon:\n    - test-operator-cluster-worker\n    - test-operator-cluster-worker2\n    - test-operator-cluster-worker3\n\n```  \n\n```console\nkubectl apply -f examples/cordon-nodes.yaml\n```  \n\nNow check the status of the resource:  \n\n```console\nkubectl describe cordonnodes.khaos.stackzoo.io example-cordon-node | grep \"Nodes Cordoned\"\n\nNodes Cordoned:  3\n```   \n\n\nNow run a busybox pod:  \n```console\nkubectl apply -f examples/test-node-cordon-pod.yaml\n\npod/busybox-pod created\n```   \n\nLet's check that pod:  \n```console\nkubectl -n default describe pod busybox-pod | grep Warning\n\nWarning  FailedScheduling  63s   default-scheduler  0/4 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 node(s) were unschedulable. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling..\n```  \n\n\u003c/details\u003e  \n\n\n\u003cbr/\u003e  \n\n\n## Operator Installation\n\n### Via Makefile\nThis repo contains a [github action](https://github.com/stackzoo/khaos/blob/main/.github/workflows/release.yaml) to publish  the operator *oci image*  to *github registry* when new release tags are pushed to the main branch.  \nIn order to install the operator as a pod in the cluster you can leverage one of the *make* targets:  \n```console\nmake deploy IMG=ghcr.io/stackzoo/khaos:0.0.28\n```  \n\nThis command will install all the required *CRDs* and *RBAC manifests* and then start the operator as a pod:  \n```console\nkubectl get pods -n khaos-system\n\nNAME                                       READY   STATUS             RESTARTS   AGE\nkhaos-controller-manager-8887957bf-5b8g9   2/2     Running               0       107s\n```  \n\n\u003e [!NOTE]  \n\u003e If you encounter RBAC errors, you may need to grant yourself cluster-admin privileges or be logged in as admin.  \n  \n### Via Helm (Recommended)\nThe *Makefile* also contains a target to build the operator's *Helm chart* with [*helmify*](https://github.com/arttor/helmify).  \nYou can build the helm chart locally with the following command (once you are inside the project's root):  \n```console\nmake helm\n```  \nThis will put the charts inside the `charts/khaos` folder.  \nThe release action also push the charts on the github registry.  \nYou can install the operator via helm with the following single command:  \n```console\nhelm upgrade --install khaos oci://ghcr.io/stackzoo/khaos/helm-charts/khaos --version v0.0.28  \n```  \n\n\n\n\n## Operator Image Signature Verification\nThe `realease` pipeline sign the operator's *OCI image* with [cosign](https://docs.sigstore.dev/signing/quickstart/).  \nIn order to verify the signature, use the following command:  \n```console\ncosign verify --key cosign/cosign.pub ghcr.io/stackzoo/khaos:0.0.28\n```  \nVerification output:  \n```console\n\nVerification for ghcr.io/stackzoo/khaos:0.0.28 --\nThe following checks were performed on each of these signatures:\n  - The cosign claims were validated\n  - Existence of the claims in the transparency log was verified offline\n  - The signatures were verified against the specified public key\n\n[{\"critical\":{\"identity\":{\"docker-reference\":\"ghcr.io/stackzoo/khaos\"},\"image\":{\"docker-manifest-digest\":\"sha256:3b6d72f646820225943d401a6bea795925e0714d75d6c5c5b7e0de0a3c9178b2\"},\"type\":\"cosign container image signature\"},\"optional\":{\"Bundle\":{\"SignedEntryTimestamp\":\"MEUCIQCLufLLbhbHa+rawlztjHOP7goS30ekP25Q4wtmflob/gIgMGBIVWMeSMgJEfBbPXPd+YV4Ep17RAWkqza6qJXugDY=\",\"Payload\":{\"body\":\"eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiaGFzaGVkcmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiIxMDMyOTI2MTRmNmRlZTRkZTdlZDUzM2ZjMmZmZGU2MGY3OTI5OTM5YTFmZTE1ODg5Mzk3NTcxZmQ3NmFlYjEwIn19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FVUNJUUM2OWZNSWw5MFVBSFJoRXdDMi9lYXJ5TkMwYTlvc3IwSkN1c2o3K2M5ejV3SWdKZEJUdGhPWVdVQm44aTBHWW9zN2d0UlJiQXgvbElXd081dkMyMGdkQzNNPSIsInB1YmxpY0tleSI6eyJjb250ZW50IjoiTFMwdExTMUNSVWRKVGlCUVZVSk1TVU1nUzBWWkxTMHRMUzBLVFVacmQwVjNXVWhMYjFwSmVtb3dRMEZSV1VsTGIxcEplbW93UkVGUlkwUlJaMEZGWldaRUsxaFlUbkp3WVVWc1NIaEdVbXBvVEhoSGVFZEJReTg0Y1FwblUwOU5TRE13VEVoeGVXbFdVVlZQTUZOcFQzQnFWSFpKUmtOT2JXWnJlamRhVDNSWlIwbDVPVzkwU0doeWVtOHpNbmw1V1ZBemF6Sm5QVDBLTFMwdExTMUZUa1FnVUZWQ1RFbERJRXRGV1MwdExTMHRDZz09In19fX0=\",\"integratedTime\":1707833345,\"logIndex\":71110514,\"logID\":\"c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d\"}}}}]\n```  \n\n\n\n## Useful References\n\n- [kubebuilder docs](https://book.kubebuilder.io/)\n- [programming kubernetes book](https://www.oreilly.com/library/view/programming-kubernetes/9781492047094/)\n- [kubernetes programming with go book](https://link.springer.com/book/10.1007/978-1-4842-9026-2)\n- [chaos engineering book](https://www.oreilly.com/library/view/chaos-engineering/9781492043850/)  \n\n\n## License\n\nThis operator is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackzoo%2Fkhaos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstackzoo%2Fkhaos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackzoo%2Fkhaos/lists"}