{"id":7565064,"url":"https://github.com/sameerkasi200x/docker-chaos-engineering","last_synced_at":"2025-07-09T20:32:00.666Z","repository":{"id":232121336,"uuid":"132852596","full_name":"sameerkasi200x/docker-chaos-engineering","owner":"sameerkasi200x","description":"This repository has some examples of how you can perform chaos injection with docker based setup","archived":false,"fork":false,"pushed_at":"2018-05-13T07:00:32.000Z","size":177,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-20T16:40:02.515Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sameerkasi200x.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-10T05:26:51.000Z","updated_at":"2023-01-23T20:54:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"37bce07e-cbae-467a-aacf-732bf882642d","html_url":"https://github.com/sameerkasi200x/docker-chaos-engineering","commit_stats":null,"previous_names":["sameerkasi200x/docker-chaos-engineering"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sameerkasi200x/docker-chaos-engineering","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sameerkasi200x%2Fdocker-chaos-engineering","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sameerkasi200x%2Fdocker-chaos-engineering/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sameerkasi200x%2Fdocker-chaos-engineering/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sameerkasi200x%2Fdocker-chaos-engineering/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sameerkasi200x","download_url":"https://codeload.github.com/sameerkasi200x/docker-chaos-engineering/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sameerkasi200x%2Fdocker-chaos-engineering/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264502653,"owners_count":23618667,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-04-08T03:59:11.361Z","updated_at":"2025-07-09T20:32:00.658Z","avatar_url":"https://github.com/sameerkasi200x.png","language":"HTML","funding_links":[],"categories":["3. Fault Injection"],"sub_categories":["Containers \u0026 Orchestrators"],"readme":"# Chaos Engineering with Docker EE\n \n## Why Chaos Engineering?\n\nEven before we get into the definition of Chaos Engineering or why it has become important, let's take a look at traditional approach. Most of the applications and configuration would be put under stress testing to find out the breakage point. This primarily helped to assure the operations team that the provisioned capacity is enough for the anticipated workload. The tests was relatively (if not fairly) simple to do. But with time there are couple of things that has changed:\n \n 1. System have become more and more complex now\n 2. Workloads can change abruptly and scaling up and down is a necessity now\n \nAlso, there is a philosophical shift happenning the way IT operations used to think -\n \n 1. Servers are disposable - Earlier the basic deployment units (in most cases physical or virtual servers) were treated like \"Pets\" and the [configuration changes would lead to a snowflake](https://martinfowler.com/bliki/SnowflakeServer.html). Now with configuration management tools servers are disposable like \"cattles\" and can be resurrected from scratch if there is a configuration change aka [Pheonix Servers](https://martinfowler.com/bliki/PhoenixServer.html).\n\n 2. Failure have been accepted as business as usual, outages are not. I am not trying to force you to accept system failures, but most of the IT operations today acknowledges that things would go wrong. Simply put, one needs to be prepared for it.\n\n 3. Because of the explosion of internet, services are not limited by geographies any more. Workloads are not predictible any more and they are bound to go beyond the breakage point of one servers, it is just a matter of time and chance. \n\n 4. Complexity of applications has increased multi-fold. Today applications are not just three tier deployments. A web page rednered might be working with 10s or in some cases 100s of micro-services in the backend. Only way test the resiliency of the system is by injecting random issues on purpose. \n \nThis all lead the IT Operation leads to be convinced that the best way to be prepared for an outage is to simulate one. If you are not convinced yet, perhaps you want to read a bit about the study of [how much loss the business can suffer because of infrastructure outage](https://www.zdnet.com/article/cloud-computing-heres-how-much-a-huge-outage-could-cost-you/).\n\n\n## How do you go about it?\nSo what should be your strategy? I believe the easiest way is to introduce unit testing and integration testing for infrastructure and architecture components too, just like application code. so for any kind of High Availability or Disaster Recovery approach you have implemented, you should have a test case. e.g. if you are having a cluster with 2 nodes, your test case could be be shoot down one of the node. Yes, you read it right. I am suggesting that you should take down a node. There is no other way for you to test high availability but to simulate failure. Similarly you can test scalability but injecting slowness and network congetion. \n\nThere are many popular examples and inspirations for Chaos Injection. Most popular one are:\n 1. Generic guidelines are available on [Principles of Chaos Engineering](http://principlesofchaos.org)\n 2. Netflix's [Chaos Monkey](https://github.com/Netflix/chaosmonkey) to do various kind of chaos injection e.g. introduce slowness in the network, kill EC2 instances, detach the network or disks from EC2 instances\n 2. Netflix's [Chaos Kong](https://medium.com/netflix-techblog/chaos-engineering-upgraded-878d341f15fa) though is not open sourced yet but a nice inspiration and aspiration for anyone embarking on chaos engineering within their enterprise.\n 3. [Facebook's Project Storm](https://siliconangle.com/blog/2016/08/31/meet-project-storm-facebooks-swat-team-for-disaster-proofing-data-centers/)\n\nThose who practice chaos engineering by trying to break themselves, have been rewarded well in times of outages. Best example is how [Netflix weathered the storm by preparing for the worst](https://www.techrepublic.com/article/aws-outage-how-netflix-weathered-the-storm-by-preparing-for-the-worst/).\n\n## How does that translate in the container's world?\n\nIn today's date a lot of new applications and services are being deployed as containers. If you are starting up with Chaos Engineering in Docker, there are many different mechanisms and tools available at your disposal.\n\nBefore we get into tools, let's look at some of the basic features of Docker which should be helpful to you.\n\n### 1. Docker Service \n \nIt is often better to deploy your application as a Swarm Service instead of deploying them as native container. In case you are using Kubernetes, it is better to deploy your request as a sevice. Both the definitions are declarative and define the desired state of service. This is really helpful in maintaining the uptime of your application as the service would always try to maintain the availability of service.\n\n#### Example: ####\nIn this example, I am going to use a Dockerfile to build a new image and then I will be using it to deploy a new service. The example is executed against a Docker UCP cluster from a client node (with docker cli and UCP Client Bundle).\n\nSetup a docker build file [```Dockerfile-nohc```](https://raw.githubusercontent.com/sameerkasi200x/docker-chaos-engineering/master/code/Dockerfile-nohc):\n\n    FROM nginx:latest\n    RUN apt-get -qq update\n    COPY index.html /usr/share/nginx/html\n    EXPOSE 80 443\n    CMD [\"nginx\", \"-g\", \"daemon off;\"]\n\nBuild your image Image\n\n    sh-4.2$ docker image build -t $dtr_url/development/tweet_to_us:demoMay -f Dockerfile-nohc .\n    Sending build context to Docker daemon  4.096kB\n    ip-10-100-2-106: Step 1/4 : FROM nginx:latest\n    ip-10-100-2-106:  ---\u003e ae513a47849c\n    ip-10-100-2-106: Step 2/4 : COPY index.html /usr/share/nginx/html\n    ip-10-100-2-106:  ---\u003e Using cache\n    ip-10-100-2-106:  ---\u003e b97207424f3a\n    ip-10-100-2-106: Step 3/4 : EXPOSE 80 443\n    ip-10-100-2-106:  ---\u003e Using cache\n    ip-10-100-2-106:  ---\u003e bfe4f59a2094\n    ip-10-100-2-106: Step 4/4 : CMD nginx -g daemon off;\n    ip-10-100-2-106:  ---\u003e Using cache\n    ip-10-100-2-106:  ---\u003e cb79c6283bb5\n    ip-10-100-2-106: Successfully built cb79c6283bb5\n    ip-10-100-2-106: Successfully tagged dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay\n\n\nNow we need to push you image to a repository (DTR or Dockerhub), so that it is available to all nodes:\n\n\tsh-4.2$ docker image push $dtr_url/development/tweet_to_us:demoMay\n\tThe push refers to a repository [dtr.ashnikdemo.com:12443/development/tweet_to_us]\n\tc75bed55c5fa: Pushed\n\t7ab428981537: Mounted from development/tweet-to-us\n\t82b81d779f83: Mounted from development/tweet-to-us\n\td626a8ad97a1: Mounted from development/tweet-to-us\n\tdemoMay: digest: sha256:08090c853df56ceee495fb95537ac9f2c81cf8718e5fc76c513ba1d8e7d145f0 size: 1155\n\nNow we will start a service using this image:\n\n\tsh-4.2$ docker service create -d --name=twet-app --mode=replicated --replicas=2 --publish 8080:80  dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay\n\tpq6eojqprru4ctw0ib0lwfmj6\n\nThis request asks the Swarm cluster to setup the service with ```--mode=replicated``` and ```--replicas=2``` i.e. Swarm would try to maintain two tasks for this service at any point of time, unless requested otherwise by the user. You can inspect the tasks running for the service with ```docker service ps``` command:\n\n\tsh-4.2$ docker service ps twet-app\n\tID                  NAME                IMAGE                                                      NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS\n\tzzq1jgolcc2o        twet-app.1          dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay   ip-10-100-2-67      Running             Running 3 minutes ago\n\tzlkf4ejuxus8        twet-app.2          dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay   ip-10-100-2-93      Running             Running 3 minutes ago\n \nAs you can see there are two tasks running and these tasks would be setup with VIP which will do load-balancing among the two containers/tasks.\n\n\tsh-4.2$ docker service inspect --format='{{.Endpoint}}'  twet-app\n\t{{vip [{ tcp 80 8080 ingress}]} [{ tcp 80 8080 ingress}] [{f80zlxoy56y20ql48o3v9aiwo 10.255.0.225/16}]}\n\n\nLet's try to kill one of the underlying containers and see if Swarm is able to maintain the declarative state we had requested:\n\n\tsh-4.2$ docker container ls | grep -i twet-app\n\t603c7f8940fe        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay   \"nginx -g 'daemon ...\"   7 minutes ago       Up 7 minutes        80/tcp, 443/tcp                                           ip-10-100-2-67/twet-app.1.zzq1jgolcc2oyucexn4j9u9pq\n\t54aa164ea509        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay   \"nginx -g 'daemon ...\"   7 minutes ago       Up 7 minutes        80/tcp, 443/tcp                                           ip-10-100-2-93/twet-app.2.zlkf4ejuxus851onp4i2t143p\n\tsh-4.2$\n\tsh-4.2$ docker container kill 603c7f8940fe\n\t603c7f8940fe\n\tsh-4.2$\n\tsh-4.2$\n\tsh-4.2$ docker service ps twet-app\n\tID                  NAME                IMAGE                                                      NODE                DESIRED STATE       CURRENT STATE           ERROR                         PORTS\n\tsp4hz64oytu0        twet-app.1          dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay   ip-10-100-2-67      Running             Running 2 seconds ago\n\tzzq1jgolcc2o         \\_ twet-app.1      dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay   ip-10-100-2-67      Shutdown            Failed 7 seconds ago    \"task: non-zero exit (137)\"\n\tzlkf4ejuxus8        twet-app.2          dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay   ip-10-100-2-93      Running             Running 8 minutes ago\n\tsh-4.2$\n\nAs you can see the container ```603c7f8940fe``` was used by one of the tasks of our service ```twet-app``` and once we kill the container, Swarm tries to maintain the state by starting another task. \n\n\n**Note:** Pushing image to repository is needed when you are running with distributed setup. As you can see above in the build was done on one of the nodes from the Swarm cluster```ip-10-100-2-106``` and image would be only available on only one node. Hence if we were to run service without pushing the image to a repository, there is good chance that the tasks would get started on the same node (```ip-10-100-2-106```) i.e. the only node that has access to the image or different nodes would get different images (left by different image builds). Swarm does a good job of reminding us about this. Here is an example if I tried to run the servie without pushing the image:\n\n\tsh-4.2$ docker service create -d --name=twet-app --mode=replicated --replicas=2 --publish 8080:80  dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay\n\timage dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay could not be accessed on a registry to record\n\tits digest. Each node will access dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay independently,\n\tpossibly leading to different nodes running different\n\tversions of the image.\n\n\tt46gb1wi3tc7xs2j08egzcut1\n\n### 2. Health Checks\nDocker allows you to use healthcheck to keep a tab on the health of running containers. The healthcheck can be either baked into you image during the build process using ```HEALTHCHECK``` direction in Dockerfile or during runtime using --healthcheck option with ```docker service create``` or ```docker container run```\n\n\nTo quote the [docker documentation](https://docs.docker.com/engine/reference/builder/#healthcheck)\n\n\u003e The ```HEALTHCHECK``` instruction tells Docker how to test a container to check that it is still working. This can detect cases such as a web server that is stuck in an infinite loop and unable to handle new connections, even though the server process is still running.\n\n**Note:** The ```HEALTHCHECK``` feature was added in Docker 1.12.\n\n#### Build time example of HEALTHCHECK ####\nTo make use of this feature we will add a new command to our [```Dockerfile```](https://raw.githubusercontent.com/sameerkasi200x/docker-chaos-engineering/master/code/Dockerfile) now\n\n\tHEALTHCHECK --interval=30s --timeout=3s --retries=2 \\ \n\t    CMD  python /usr/share/nginx/html/healthcheck.py || exit 1\n\nThis means that the healthcheck command ```python /usr/share/nginx/html/healthcheck.py``` will be run for the first time after ```30s``` i.e. 30 seconds after starting up the tasks. The healthcheck will be run with an ```interval``` of every 30s after that. The healthcheck would ```timeout``` in ```3s``` and upon failure of ```2 retries``` the container will be declared unhealthy.\n\n\nWe will have to add a few new files to support HEALTHCHECK \n+ [```healthcheck.py```](https://raw.githubusercontent.com/sameerkasi200x/docker-chaos-engineering/master/code/healthcheck.py) - our own little piece of code to check the health of container.\n+ [```healthcheck.html```](https://raw.githubusercontent.com/sameerkasi200x/docker-chaos-engineering/master/code/healthcheck.html)\n\nNow we will build and push the image\n\n\tsh-4.2# docker image build --no-cache -t $dtr_url/development/tweet_to_us:demoMay_Healthcheck -f Dockerfile .\n\tSending build context to Docker daemon   7.68kB\n\tStep 1/7 : FROM nginx:latest\n\t ---\u003e b175e7467d66\n\tStep 2/7 : RUN apt-get -qq update\n\t ---\u003e Running in 152a3156632c\n\t ---\u003e 2a6be94d9a04\n\tRemoving intermediate container 152a3156632c\n\tStep 3/7 : RUN apt-get -qq --allow-downgrades --allow-remove-essential --allow-change-held-packages install python \u003e /dev/null\n\t ---\u003e Running in 56a5b9141aaf\n\tdebconf: delaying package configuration, since apt-utils is not installed\n\t ---\u003e 99605506e79f\n\tRemoving intermediate container 56a5b9141aaf\n\tStep 4/7 : COPY healthcheck.html healthcheck.py index.html /usr/share/nginx/html/\n\t ---\u003e b1b93b73d0fa\n\tRemoving intermediate container dab1d03a75e4\n\tStep 5/7 : EXPOSE 80 443\n\t ---\u003e Running in 50b63022a6c3\n\t ---\u003e 4297f32f769b\n\tRemoving intermediate container 50b63022a6c3\n\tStep 6/7 : HEALTHCHECK --interval=30s --timeout=3s --retries=2 CMD python /usr/share/nginx/html/healthcheck.py || exit 1\n\t ---\u003e Running in 1a1a9cd1f139\n\t ---\u003e 042010177008\n\tRemoving intermediate container 1a1a9cd1f139\n\tStep 7/7 : CMD nginx -g daemon off;\n\t ---\u003e Running in 767a8098f177\n\t ---\u003e 9153fcd78222\n\tRemoving intermediate container 767a8098f177\n\tSuccessfully built 9153fcd78222\n\tSuccessfully tagged dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck\n\t\n\tsh-4.2# docker image push dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck\n\tThe push refers to a repository [dtr.ashnikdemo.com:12443/development/tweet_to_us]\n\ta60e00b623bb: Pushed\n\t2e83bcd5bc8d: Pushed\n\t5134599f00a1: Pushed\n\t77e23640b533: Pushed\n\t757d7bb101da: Pushed\n\t3358360aedad: Pushed\n\tdemoMay_Healthcheck: digest: sha256:a4fb4fd2733e37ae7282148ccb497aac4c2fc18a74aa8e950271ffd648b07da8 size: 1579\n\n\nNow once we deploy the service, initially the health status would be ```starting``` until the first healthcheck is initiated\n\n\tsh-4.2$ docker service rm twet-app\n\ttwet-app\n\t\n\tsh-4.2$ docker service create -d --name=twet-app --mode=replicated --replicas=2 --publish 8080:80  dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck\n\tlbfmu7vxa1i6arfmpstzq3rer\n\n\tsh-4.2$ docker container ls | grep -i twet\n\t1feb5ed8e0b6        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   23 seconds ago      Up 23 seconds (health: starting)   80/tcp, 443/tcp                                           ip-10-100-2-93/twet-app.2.rkh6gofzfru83wjqcyzq2mdcl\n\t6ccb4d691fe9        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   24 seconds ago      Up 23 seconds (health: starting)   80/tcp, 443/tcp                                           ip-10-100-2-67/twet-app.1.urb4v2vttrlsvcz11wfnj6yh2\n\nAfter the first healthechk, the healthcheck status would be ```healthy```\n\n\tsh-4.2$ docker container ls | grep -i twet\n\t1feb5ed8e0b6        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   About a minute ago   Up About a minute (healthy)   80/tcp, 443/tcp                                           ip-10-100-2-93/twet-app.2.rkh6gofzfru83wjqcyzq2mdcl\n\t6ccb4d691fe9        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   About a minute ago   Up About a minute (healthy)   80/tcp, 443/tcp                                           ip-10-100-2-67/twet-app.1.urb4v2vttrlsvcz11wfnj6yh2 \n\n\n\n#### Testing Healthcheck and self-healing ####\nNow let's try to force a distruption by connecting to one of the containers and changing the content of ```healthcheck.html```\n\n\tsh-4.2$ docker container exec -it 1feb5ed8e0b6 bash\n\troot@1feb5ed8e0b6:/# echo test \u003e /usr/share/nginx/html/healthcheck.html\n\troot@1feb5ed8e0b6:/# exit\n\n\nSoon (in about 1 minute given our ```interval```, ```timeout``` and ```retries``` configuration in the Dockerfile), the container will be reported unhealthy and replaced with a new container to run the task\n\n\n\tsh-4.2$ docker container ls | grep -i twet\n\t1feb5ed8e0b6        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   3 minutes ago       Up 13 minutes (unhealthy)   80/tcp, 443/tcp                                           ip-10-100-2-93/twet-app.2.xskxfd9n6e39wlghpm0k7tphr\n\t6ccb4d691fe9        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   13 minutes ago      Up 13 minutes (healthy)    80/tcp, 443/tcp                                           ip-10-100-2-67/twet-app.1.urb4v2vttrlsvcz11wfnj6yh2\n\n\tsh-4.2$ docker container ls -a | grep -i twet\n\tefc5b969ef8c        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   42 seconds ago      Up 36 seconds (healthy)          80/tcp, 443/tcp                                                                             ip-10-100-2-93/twet-app.2.xskxfd9n6e39wlghpm0k7tphr\n\t1feb5ed8e0b6        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   11 minutes ago      Exited (0) 41 seconds ago                                                                                                    ip-10-100-2-93/twet-app.2.rkh6gofzfru83wjqcyzq2mdcl\n\t6ccb4d691fe9        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   11 minutes ago      Up 11 minutes (healthy)          80/tcp, 443/tcp                                                                             ip-10-100-2-67/twet-app.1.urb4v2vttrlsvcz11wfnj6yh2 \n\n\n#### Runtime definition of Healthcheck ####\nYou can also override the command to check health, its frequency and retries while creating the service\n\n\tsh-4.2$ docker service rm twet-app\n\ttwet-app\n\n\tsh-4.2$ docker service create -d --name=twet-app \\\n\t  --mode=replicated --replicas=2 --publish 8080:80 \\\n\t  --health-cmd \"python /usr/share/nginx/html/healthcheck.py || exit 1\" \\\n\t  --health-interval 10s \\\n\t  --health-retries 2 \\\n\t  --health-timeout 30ms \\\n\t  dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck\n\n\tmj4lel34whrvupscq8sjt7g5m\n\n#### Disable healthcheck ####\n\nIn runtime while creating a service, you can disable the healtcheck with ```--no-healthcheck``` option. That will supress any healthcheck which has been defined in the base image\n\n\tsh-4.2$ docker service create -d --name=twet-app \\\n\t        --mode=replicated --replicas=2 --publish 8080:80 \\\n\t        --no-healthcheck \\\n\t        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck\n\t\n\tnxkv39smzq5k0o9tgwmc74t2c\n\n\nIf the base container you are going to use has a ```HEALTHCHECK``` defined, it can also disable the healthchek during build time using ```HEALTHCHECK NONE``` \n\n#### Checking the status ####\nYou can use ```docker container inspect``` command to further review the state of your containers and details healthchekc command output:\n\ne.g. in case of timeout error:\n\n\tsh-4.2$ docker container inspect --format='{{json .State.Health}}' a9486dc964af\n\t{\"Status\":\"starting\",\"FailingStreak\":1,\"Log\":[{\"Start\":\"2018-05-12T17:18:01.698087531Z\",\"End\":\"2018-05-12T17:18:01.728282187Z\",\"ExitCode\":-1,\"Output\":\"Health check exceeded timeout (30ms)\"}]}\n\nin case of failures:\n\n\tsh-4.2$ docker container inspect --format='{{json .State.Health}}' 62ad34709fb8\n\t{\"Status\":\"healthy\",\"FailingStreak\":1,\"Log\":[{\"Start\":\"2018-05-12T17:28:33.714393794Z\",\"End\":\"2018-05-12T17:28:33.793534206Z\",\"ExitCode\":0,\"Output\":\"\"},{\"Start\":\"2018-05-12T17:28:53.793900452Z\",\"End\":\"2018-05-12T17:28:53.871217425Z\",\"ExitCode\":1,\"Output\":\"The content of the healthcheck did not match. Expected Content-\\\"healthy\\\", we got: test\\n\"}]}\n\n\tsh-4.2$ docker container inspect --format='{{json .State.Health}}' 62ad34709fb8\n\t{\"Status\":\"unhealthy\",\"FailingStreak\":2,\"Log\":[{\"Start\":\"2018-05-12T17:28:33.714393794Z\",\"End\":\"2018-05-12T17:28:33.793534206Z\",\"ExitCode\":0,\"Output\":\"\"},{\"Start\":\"2018-05-12T17:28:53.793900452Z\",\"End\":\"2018-05-12T17:28:53.871217425Z\",\"ExitCode\":1,\"Output\":\"The content of the healthcheck did not match. Expected Content-\\\"healthy\\\", we got: test\\n\"},{\"Start\":\"2018-05-12T17:29:13.871399894Z\",\"End\":\"2018-05-12T17:29:13.948097443Z\",\"ExitCode\":1,\"Output\":\"The content of the healthcheck did not match. Expected Content-\\\"healthy\\\", we got: test\\n\"}]}\n\nin case of no failures\n\n\tsh-4.2$ docker container inspect --format='{{json .State.Health}}' 181d566f6aa9\n\t{\"Status\":\"healthy\",\"FailingStreak\":0,\"Log\":[{\"Start\":\"2018-05-12T17:25:06.184822447Z\",\"End\":\"2018-05-12T17:25:06.262241844Z\",\"ExitCode\":0,\"Output\":\"\"},{\"Start\":\"2018-05-12T17:25:26.262408086Z\",\"End\":\"2018-05-12T17:25:26.338883823Z\",\"ExitCode\":0,\"Output\":\"\"},{\"Start\":\"2018-05-12T17:25:46.339143953Z\",\"End\":\"2018-05-12T17:25:46.416973058Z\",\"ExitCode\":0,\"Output\":\"\"},{\"Start\":\"2018-05-12T17:26:06.417170336Z\",\"End\":\"2018-05-12T17:26:06.495295881Z\",\"ExitCode\":0,\"Output\":\"\"},{\"Start\":\"2018-05-12T17:26:26.495482044Z\",\"End\":\"2018-05-12T17:26:26.572278146Z\",\"ExitCode\":0,\"Output\":\"\"}]}\n\n\n\t\nNote: The output will contain a friendly message if one is printed by your healthcheck command.\n\n\n### 3. Tooling and Automation \nNow that we have covered the basic building blocks of chaos engineering with Docker, let's try to take a look at some tools. Pumba is a fairly new but quite promising tool for chaos orchestration. Best thing is it works well with a Swarm cluster, you just need to point it to the manager node. We can easily get it to work with Docker UCP Client Bundle. \n\n\n\n#### Example\n\nFirst we need to setup an isolated network where we will setup our application and test it out\n\t docker network create -d overlay tweet-app-net\n\nNow let's setup a service using healthcheck from the previous examples\n\t\n\tdocker service create -d --name=twet-app --network tweet-app-net \\\n\t  --mode=replicated --replicas=2 --publish 8080:80 \\\n\t  --health-cmd \"python /usr/share/nginx/html/healthcheck.py || exit 1\" \\\n\t  --health-interval 20s \\\n\t  --health-retries 2 \\\n\t  --health-timeout 200ms \\\n\t  dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck\n\nLet's ensure that the service has been started properly with requested number of replicas which are healthy\n\n\tsh-4.2$ docker container ls | grep -i twet\n\t75b2bf6f219d        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   27 seconds ago       Up 21 seconds (healthy)   80/tcp, 443/tcp                                           ip-10-100-2-93/twet-app.1.im7f7qm2xh6fk6uqla462qzia\n\t393355d083fb        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   About a minute ago   Up 59 seconds (healthy)   80/tcp, 443/tcp                                           ip-10-100-2-67/twet-app.2.6uueh28nxj7btpfzffeq40f6b\n\nNow let's use pumba to randomly kill some containers under the service \n\n\texport SVC_NAME=twet-app\n    pumba --random kill  $(docker service ps --no-trunc \\\n       --filter \"desired-state=Running\"  \\\n        ${SVC_NAME} | awk ' {if (NR!=1) {print $2\".\"$1} } ')\n\n\nYou will an output confirming that the container has been killed\n\n\tsh-4.2$     pumba --random kill  $(docker service ps --no-trunc \\\n\t       --filter \"desired-state=Running\"  \\\n\t        ${SVC_NAME} | awk ' {if (NR!=1) {print $2\".\"$1} } ')\n\tINFO[0000] Kill containers\n\tINFO[0003] Killing /twet-app.2.6uueh28nxj7btpfzffeq40f6b (393355d083fbd33d8247e6cf9dcdb36046000764547db776b405bb4c37ef7438) with signal SIGKILL\n\nYou will notice that as soon as the container is killed, the swarm manager would try to restore the state back to desired state i.e. with 2 healthy replica\n\n\tsh-4.2$ docker container ls | grep -i twet\n\t75b2bf6f219d        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   42 seconds ago      Up 36 seconds (healthy)   80/tcp, 443/tcp                                           ip-10-100-2-93/twet-app.1.im7f7qm2xh6fk6uqla462qzia\n\tsh-4.2$ docker container ls | grep -i twet\n\t75b2bf6f219d        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   44 seconds ago      Up 38 seconds (healthy)   80/tcp, 443/tcp                                           ip-10-100-2-93/twet-app.1.im7f7qm2xh6fk6uqla462qzia\n\tsh-4.2$\n\tsh-4.2$ docker container ls | grep -i twet\n\tdfb8b7ebc559        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   7 seconds ago       Up 1 seconds (health: starting)   80/tcp, 443/tcp                                           ip-10-100-2-67/twet-app.2.7505c070tcyk14wudbdbufy3t\n\t75b2bf6f219d        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   47 seconds ago      Up 41 seconds (healthy)           80/tcp, 443/tcp                                           ip-10-100-2-93/twet-app.1.im7f7qm2xh6fk6uqla462qzia\n\n\nYou can also try to stop or remove a container various commands provided by ```pumba```.\n\nYou can also use ```--interval``` option to run the command at a regular interval to perform stress testing. e.g. to run the same kill command every 10minutes\n\n\texport SVC_NAME=twet-app\n    pumba --random --interval 10m kill  $(docker service ps --no-trunc \\\n       --filter \"desired-state=Running\"  \\\n        ${SVC_NAME} | awk ' {if (NR!=1) {print $2\".\"$1} } ')\n\n#### Network delay\nLet's first take example of a simple setup with a single node.\n\n\tdocker swarm init\n\n\nSetup the service by running this command aginst the single manager node of your newly initiated Swarm Cluster\n\n\tsh-4.2# docker service create -d --name=twet-app --network tweet-app-net   --mode=replicated --replicas=2 --publish 8080:80   --health-cmd \"python /usr/share/nginx/html/healthcheck.py || exit 1\"   --health-interval 10s   --health-retries 2   --health-timeout 100ms   dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck\n\timage dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck could not be accessed on a registry to record\n\tits digest. Each node will access dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck independently,\n\tpossibly leading to different nodes running different\n\tversions of the image.\n\t\n\tpb7teb13m2oczlp8rub0wjkdo\n\nFire a pumba command to introduce delays\n\n\tsh-4.2# pumba --random netem --interface lo --duration 60s    --tc-image gaiadocker/iproute2 delay    --time 10 jitter 100    --distribution normal $(docker service ps --no-trunc \\\n\t   --filter \"desired-state=Running\"  \\\n\t    ${SVC_NAME} | awk ' {if (NR!=1) {print $2\".\"$1} } ')\n\tINFO[0000] netem: delay for containers\n\tINFO[0000] Running netem command '[delay 10ms 10ms 20.00]' on container 2eb65f467edc585e586feee01d3eba36c301bb3830a9e163e81aa4edf8d5f36c for 1m0s\n\tINFO[0000] Start netem for container 2eb65f467edc585e586feee01d3eba36c301bb3830a9e163e81aa4edf8d5f36c on 'lo' with command '[delay 10ms 10ms 20.00]'\n\n\nMonitor the status for containers running the of for the service:\n\n\tsh-4.2# docker container ls | grep  twet-app\n\t2eb65f467edc        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   About a minute ago   Up About a minute (healthy)   80/tcp, 443/tcp     twet-app.2.eyxe2yo9fs928b6x2oa3q26m0\n\t031b9ec08b50        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   11 minutes ago       Up 11 minutes (healthy)       80/tcp, 443/tcp     twet-app.1.k6wfqvzyn5p7ka60witz92msv\n\nYou will notice that becuase of the network delays introduced by pumba, the containers are failing the healthcheck:\n\n\tsh-4.2# docker container ls | grep  twet-app\n\t2eb65f467edc        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   About a minute ago   Up About a minute (unhealthy)   80/tcp, 443/tcp     twet-app.2.eyxe2yo9fs928b6x2oa3q26m0\n\t031b9ec08b50        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   11 minutes ago       Up 11 minutes (healthy)         80/tcp, 443/tcp     twet-app.1.k6wfqvzyn5p7ka60witz92msv\n\nSoon the unhealthy container would be removed:\n\n\tsh-4.2# docker container ls | grep  twet-app\n\t031b9ec08b50        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   11 minutes ago      Up 11 minutes (healthy)   80/tcp, 443/tcp     twet-app.1.k6wfqvzyn5p7ka60witz92msv\n\nAnd it will be replaced with a new container:\n\n\tsh-4.2# docker container ls | grep  twet-app\n\t2c1453b4d680        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   5 seconds ago       Up Less than a second (health: starting)   80/tcp, 443/tcp     twet-app.2.oc900fzo7a7v1kgz2kt45h1pr\n\t031b9ec08b50        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   11 minutes ago      Up 11 minutes (healthy)                    80/tcp, 443/tcp     twet-app.1.k6wfqvzyn5p7ka60witz92msv\n\nAs soon as the healthcheck is executed, it will turn into a healthy one:\n\n\tsh-4.2# docker container ls | grep  twet-app\n\t2c1453b4d680        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   20 seconds ago      Up 15 seconds (healthy)   80/tcp, 443/tcp     twet-app.2.oc900fzo7a7v1kgz2kt45h1pr\n\t031b9ec08b50        dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   \"nginx -g 'daemon ...\"   11 minutes ago      Up 11 minutes (healthy)   80/tcp, 443/tcp     twet-app.1.k6wfqvzyn5p7ka60witz92msv\n\tsh-4.2#\n\nWhile the container is being replaced, you will notice that pumba command would fail (as the container it attached to has been lost)  \n\n\tINFO[0060] Stopping netem on container 2eb65f467edc585e586feee01d3eba36c301bb3830a9e163e81aa4edf8d5f36c\n\tINFO[0060] Stop netem for container 2eb65f467edc585e586feee01d3eba36c301bb3830a9e163e81aa4edf8d5f36c on 'lo'\n\tERRO[0060] Error response from daemon: cannot join network of a non running container: 2eb65f467edc585e586feee01d3eba36c301bb3830a9e163e81aa4edf8d5f36c\n\tERRO[0060] Error response from daemon: cannot join network of a non running container: 2eb65f467edc585e586feee01d3eba36c301bb3830a9e163e81aa4edf8d5f36c\n\n\nAs you can see, pumba was able to introduce network delay and ```HEALTHCHECK``` in the image or ```--health-cmd``` at service level helped us to restart the images which were slowing. Well, at this time this is the most that Pumba and Swarm can do. I am hoping in times to come, Swarm service healthcheck would allow us to define auto-scale policies too.\n\nNow, if we are running against a UCP setup or any \"true\" swarm cluster which has worker and manager nodes, pumba netem command would not work when you fire it from a client. This is unlike the kill command (or most of the other pumba commands), which do work against a Swarm cluster. I came up with a simple solution to work around it.\n\n#### Pumba in a container\nWell you can run pubma in a container as the example says on it's [github page](https://github.com/alexei-led/pumba).\n\n\n\u003e ```# once in a 10 seconds, try to kill (with `SIGTERM` signal) all containers named **hp(something)**```\n\u003e\n\u003e ``` # on same Docker host, where Pumba container is running```\n\u003e\n\u003e ```$ docker run -d -v /var/run/docker.sock:/var/run/docker.sock gaiaadm/pumba pumba --interval 10s kill --signal SIGTERM ^hp```\n\nThis means that we can create, a service that runs on each node in your Swarm cluster and executes pumba netem command. We need to change the ```entrypoint``` of the service and mount ```/var/run/docker.sock``` of the local node to container so that pumba can have access to docker deamon on each node. \n\nThe pumba command should essentially look for containers that belong to your service only so you need to pass a list of containers to entrypoint pumba command.\n\n\texport container_list=$(docker service ps --no-trunc --filter \"desired-state=Running\" ${SVC_NAME} | awk ' {if (NR!=1) {print $2\".\"$1} } ')\n\nThe command should try to inject delay only in specific interface i.e. the one used by ```HEALTHCHECK```.\n\n\texport netem_interface=lo\n\nNow let's run our pumba netem service\n\n\tdocker service create -d --restart-condition none --mode global --name pumba-netem-delay \\\n\t   --mount type=bind,source=/var/run/docker.sock,destination=/var/run/docker.sock \\\n\t   --entrypoint \"pumba --random netem --interface ${netem_interface} --duration 60s \\\n\t   --tc-image gaiadocker/iproute2 delay \\\n\t   --time 10 jitter 100 \\\n\t   --distribution normal ${container_list}\" \\\n\t   gaiaadm/pumba \n\nThe effect will be same as the previous example we run on one node Swarm Cluster.  \n\nIf you are scripting this, then introduce a delay and then cleanup the swarm service:\n\n\tsleep 60\n\tdocker service rm pumba-netem-delay\n\n#### Simulate Packet loss\nTo be added\n\n\n#### The bold test - Node failure\nOne of the reasont to run your containers in a Swarm cluster is to ensure fault tolderance to node failrues. Let's try to simulate node failure and see how docker UCP manager handles it.\n\nLet's first list various tasks of our application:\n\n\tdocker service ps twet-app\n\nOutput would something like below, giving you details of the number of tasks, their id and node on which they are running:\n\n\tID                  NAME                IMAGE                                                                  NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS\n\ts8iib1ue7nrd        twet-app.1          dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   ip-10-100-2-67      Running             Running 42 minutes ago\n\toiafp1o6klxx        twet-app.2          dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   ip-10-100-2-93      Running             Running 42 minutes ago\n\n\nFor the purpose of our testing let's try to fail one of the nodes, let's say ```ip-10-100-2-67```.\n\n\n\nSince I am running in AWS, I will find out the instance id of the server and restart. We can use ```docker node ls``` before and after restart, to note the node status\n\n\n\tsh-4.2$ docker node ls\n\tID                            HOSTNAME                                         STATUS              AVAILABILITY        MANAGER STATUS\n\t3du1xn000h3jz3t2fcx9lcvdl     ip-10-100-2-38                                   Ready               Active\n\tag12n6ejw7ztf0yqpsao4208u     ip-10-100-2-115                                  Ready               Active\n\tawql5xr67h0jmxjllzfohqqy2 *   ip-10-100-2-15                                   Ready               Active              Reachable\n\te2soqi2u67nfnoxop8mgfvm7a     ip-10-100-2-169                                  Ready               Active              Reachable\n\ti2fjh10bx31ij6i3q2jvzwjco     ip-10-100-2-40                                   Ready               Active              Leader\n\tlpi6z3np5vp83vatmh51d3i59     ip-10-100-2-106                                  Ready               Active\n\tm4j4g27conj199uciw98k5h1b     ip-10-100-2-67                                   Ready               Active\n\tmzwnamamkze7yagqa602tmd71     ip-10-100-2-93                                   Ready               Active\n\to3z2xxqo90mm4dlnpq632zorj     ip-10-100-2-66                                   Ready               Active\n\tyczj5bg55l37xfkugkwwyc5ji     ip-10-100-2-70.ap-southeast-1.compute.internal   Ready               Active\n\n\tsh-4.2$ aws ec2 describe-instances --filters \"Name=network-interface.private-dns-name,Values=ip-10-100-2-67.ap-southeast-1.compute.internal\" | grep -i InstanceId\n\t                    \"InstanceId\": \"i-0db2edf9253157f97\",\n\n\tsh-4.2$ aws ec2 reboot-instances --instance-ids  i-0db2edf9253157f97\n\n\tsh-4.2$ docker node ls\n\tID                            HOSTNAME                                         STATUS              AVAILABILITY        MANAGER STATUS\n\t3du1xn000h3jz3t2fcx9lcvdl     ip-10-100-2-38                                   Ready               Active\n\tag12n6ejw7ztf0yqpsao4208u     ip-10-100-2-115                                  Ready               Active\n\tawql5xr67h0jmxjllzfohqqy2     ip-10-100-2-15                                   Ready               Active              Reachable\n\te2soqi2u67nfnoxop8mgfvm7a *   ip-10-100-2-169                                  Ready               Active              Reachable\n\ti2fjh10bx31ij6i3q2jvzwjco     ip-10-100-2-40                                   Ready               Active              Leader\n\tlpi6z3np5vp83vatmh51d3i59     ip-10-100-2-106                                  Ready               Active\n\tm4j4g27conj199uciw98k5h1b     ip-10-100-2-67                                   Down                Active\n\tmzwnamamkze7yagqa602tmd71     ip-10-100-2-93                                   Ready               Active\n\to3z2xxqo90mm4dlnpq632zorj     ip-10-100-2-66                                   Ready               Active\n\tyczj5bg55l37xfkugkwwyc5ji     ip-10-100-2-70.ap-southeast-1.compute.internal   Ready               Active\n\tsh-4.2$\n\n\nAs you can see the node became unavailable once the reboot was executed\n\nIn order to maintain the desired state of service with 2 replica, Swarm manager would start a new container on one of the surviving nodes\n\n\tsh-4.2$ docker service ps twet-app\n\tID                  NAME                IMAGE                                                                  NODE                        DESIRED STATE       CURRENT STATE               ERROR               PORTS\n\t7e2icqhk0n54        twet-app.1          dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   ip-10-100-2-93              Running             Running 12 seconds ago\n\ts8iib1ue7nrd         \\_ twet-app.1      dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   m4j4g27conj199uciw98k5h1b   Shutdown            Running 50 seconds ago\n\toiafp1o6klxx        twet-app.2          dtr.ashnikdemo.com:12443/development/tweet_to_us:demoMay_Healthcheck   ip-10-100-2-93              Running\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsameerkasi200x%2Fdocker-chaos-engineering","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsameerkasi200x%2Fdocker-chaos-engineering","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsameerkasi200x%2Fdocker-chaos-engineering/lists"}