{"id":20178538,"url":"https://github.com/nvtkaszpir/dh-chaos","last_synced_at":"2025-05-07T01:35:22.574Z","repository":{"id":50186155,"uuid":"206944532","full_name":"nvtkaszpir/dh-chaos","owner":"nvtkaszpir","description":"Distributed Hackaton - Chaos Engineering","archived":false,"fork":false,"pushed_at":"2022-12-08T06:07:50.000Z","size":192,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-23T23:00:59.911Z","etag":null,"topics":["azure","chaos-toolkit","cloud","cluster","helm","k8s","kubernetes","pyenv","virtualenv"],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nvtkaszpir.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-09-07T09:16:46.000Z","updated_at":"2021-03-26T06:21:27.000Z","dependencies_parsed_at":"2023-01-24T14:40:16.049Z","dependency_job_id":null,"html_url":"https://github.com/nvtkaszpir/dh-chaos","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nvtkaszpir%2Fdh-chaos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nvtkaszpir%2Fdh-chaos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nvtkaszpir%2Fdh-chaos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nvtkaszpir%2Fdh-chaos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nvtkaszpir","download_url":"https://codeload.github.com/nvtkaszpir/dh-chaos/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224553184,"owners_count":17330418,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","chaos-toolkit","cloud","cluster","helm","k8s","kubernetes","pyenv","virtualenv"],"created_at":"2024-11-14T02:21:31.990Z","updated_at":"2024-11-14T02:21:32.489Z","avatar_url":"https://github.com/nvtkaszpir.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Distributed hackaton - chaos-toolkit\n\nThis was done during Distributed Cloud Native Hackathon 7th Sept 2019.\n[Meetup event info](https://www.meetup.com/Cloud-Native-Kubernetes-Warsaw/events/264307416/)\n\nIt is advised to read this document fully before doing any cluster setups\nor app deployments.\n\n## Objectives\n\nPrimary:\n\n- try to recreate demo scenario from chaos-toolkit about app on k8s with node\n  pool replacement (this was on GKE, so we want to try it on Azure)\n\nSecondary:\n\n- get to know about managed Kubernetes on Microsoft Azure\n  - prior experience almost 0\n- use [chaos-toolkit](https://docs.chaostoolkit.org/) to play around,\n  without prior experience with the tool\n\n## Known limitations\n\n- **this is a sample example not for production use, use at your own risk**.\n- this was tested only on Azure, with simple k8s cluster with\n  smallest instances, 1 core, 4 nodes\n- we used node draining for simulating node loss - after talking in the team\n  and due to time constraints we ruled out calling to Azure API to effectively\n  delete nodes\n- remember to delete cluster afterwards\n\n## Main takeaways\n\n### Objectives achievements\n\n- original primary objective was not achieved - due to time and tech knowledge\n  constraints about Azure in general\n- primary objective was adjusted - we changed it from `replace node` to\n  `drain selected node` - and this was acheived successfully\n- secondary objective achieved, we know Azure managed k8s and\n  chaos-toolkit better\n\n### Azure\n\n- Microsoft Azure provides managed kubernetes, but to use autoscaling node\n  pools requires additional subscription features which are in Preview mode\n  - we were unable to activate it, so we switched to simple node draining scenario\n- Due to how DNS (sic!) was set in web spawned console in Azure\n  (console icon in top right corner) we were unable to reach k8s cluster\n  with kubectl, also we had no time to adjust the env which was console using\n  (such as adjusting DNS and so on and walled-garden approach of that vm/\n  container/whatever) due to time constrains and lack of experience\n- Due to above we went full YOLO mode `¯\\_(ツ)_/¯` - we decided to compress\n  `~/.kube/` on the vm/container, and fetch it to the laptops -\n  this is a short living cluster and it was wiped after 12h\n- Using storage class `azurefile` was much faster than normal disks in Azure\n- total cloud cost 4.45 EUR - 12h of compute/storage with k8s clusters,\n  could be a bit less due to the fact we never touched Azure before.\n\n### Apps and management\n\n- Helm chart for `stable/wordpress` has very slow readiness probes\n\n### Chaos toolkit\n\n- REALLY READ chaos-toolkit tutorials before using it (derp mode on)\n- `chaos-toolkit` looks pretty interesting in automating tests, especially\n  for nightly builds\n- Currently chaos-toolkit provides a lot of testing extensions especially for\n  AWS, while GCP is severely lacking (actually just add/remove node pools in GKE)\n- Currently chaos-toolkit looks mature in it's core but needs additional\n  extendability\n\n## Preparing infrastructure\n\n### Creating kubernetes cluster\n\nCreate kubernetes cluster with 4 nodes, in Azure it should take about 30min.\n\nExport KUBECONFIG variable so that you manage only specific cluster.\n\n```bash\nexport KUBECONFIG=\"$(pwd)/.kube/config\"\n```\n\nIf using Azure and playing in shell console in the cloud just [fetch cluster credentials](https://docs.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-get-credentials).\n\n### Add azurefile storage provider\n\nIf you are using Azure k8s then we need read-write-many StorageClass to allow\nmultiple pods to write to storage:\n\n```bash\nkubectl apply -f provision/kubernetes/azure-pvc-roles.yaml\nkubectl apply -f provision/kubernetes/azure-file-sc.yaml\n```\n\nFor other storage/cloud providers you may need to adjust it accordingly.\n\n### Install helm\n\nInstall helm [v2.14.3](https://github.com/helm/helm/releases/tag/v2.14.3)\non your control host.\n\nThen we need to apply RBAC on cluster and deploy helm:\n\n```bash\nkubectl apply -f provision/kubernetes/helm-tiller.rbac.yaml\n\nhelm init --service-account tiller --wait\n\n```\n\nSince now your kubernetes cluster should be ready for app deployment.\n\n## Deploying app\n\nWe will use official helm chart for wordpress with our minor customizations\nwithin `wordpress-azurefile.yaml` file:\n\n```bash\nhelm install --name wp-02 stable/wordpress -f wordpress-azurefile.yaml\n```\n\nIt takes about few minutes due to the way storage is attached to k8s nodes.\n\n### Look at the app\n\n```bash\nkubectl port-forward service/wp-02-wordpress 80:80 8081:80\nkubectl get endpoints\nkubectl get svc\n```\n\nNotice that Azure managed kubernetes may be exposing app publicly to the Internet.\n\n### Apply labels on the nodes\n\nWe will use label `drain-me` as a selector when doing one of the tests later.\nWe just want to manage specific nodes in this scenario, and leave node with\ndatabase intact.\n\n```bash\nkubectl get nodes\nNAME                       STATUS   ROLES   AGE   VERSION\naks-agentpool-32137755-0   Ready    agent   33m   v1.13.10\naks-agentpool-32137755-1   Ready    agent   33m   v1.13.10\naks-agentpool-32137755-2   Ready    agent   33m   v1.13.10\naks-agentpool-32137755-3   Ready    agent   33m   v1.13.10\n```\n\nSee where database was deployed (kubectl describe will show it),\nin our case it was node with suffix 2. So we add labels accordingly:\n\n```bash\nkubectl label nodes aks-agentpool-32137755-2 app=mysql --overwrite\n\nkubectl label nodes aks-agentpool-32137755-0 app=drain-me --overwrite\nkubectl label nodes aks-agentpool-32137755-1 app=drain-me --overwrite\nkubectl label nodes aks-agentpool-32137755-3 app=drain-me --overwrite\n\n```\n\n## Using chaos toolkit\n\n### Install local dependencies for chaos toolkit\n\nWe will use [pyenv](https://github.com/pyenv/pyenv) with [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv)\nto install desired python version and then to create virtualenv for the project:\n\n```bash\npyenv virtualenv chaos\npyenv activate chaos\n\npip install -r requirements.txt\n\n```\n\n### Read tutorial\n\nWhen the cluster is getting ready it is good time to read about\n[chaos-toolkit tutorial](https://docs.chaostoolkit.org/reference/tutorial/).\n\nThis way next sections will be less cryptic ;-)\n\n### Run chaos discover\n\nRemember to export `KUBECONFIG`.\n\nFirst, run chaos discovery to fetch available options for kubernetes:\n\n```bash\nchaos discover chaostoolkit-kubernetes --no-install\n```\n\nSee `discovery.json`\n\n### Look at the experiment\n\nLook into `experiment.json`  - we defined there:\n\n- hypothesis that specific microservice is up and healthy, where healthy\n  means it has proper number of desired replicas\n- testing methods to validate above:\n  - kill 2 pods and wait 90s\n  - drain 1 random node and wait 90s\n\n### Run chaos experiment\n\nWe know that in Azure and non given cluster the specific wordpress pods are\nin Ready state in about 70s, so we will use 90s as a base time span for tests.\n\nRun chaos experiment which kills 2 pods (out of 3) and waits 90s:\n\n```bash\nchaos run --journal-path journal.json experiment.json\n```\n\nPlay around, with each run `chaos run experiment.json` and see what happens:\n\n- change replicas to 2\n- in `experiment.json` change `pauses after` value to 10.\n\nNotice to uncordon nodes after running experiment.\n\nAdd your own experiments basing on [chaosk8s](https://docs.chaostoolkit.org/drivers/kubernetes/).\n\n### Generate report\n\nGenerate example report:\n\n```bash\nchaos report --export-format=html journal.json report.html\n```\n\nPro tip - there are more formats, such as: asciidoc, beamer, commonmark,\ncontext, docbook, docx, dokuwiki, dzslides, epub, epub3, fb2, haddock,\nhtml, html5, icml, json, latex, man, markdown, markdown_github,\nmarkdown_mmd, markdown_phpextra, markdown_strict, mediawiki, native,\nodt, opendocument, opml, org, pdf, plain, revealjs, rst, rtf, s5,\nslideous, slidy, texinfo, textile\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvtkaszpir%2Fdh-chaos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnvtkaszpir%2Fdh-chaos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvtkaszpir%2Fdh-chaos/lists"}