{"id":22475432,"url":"https://github.com/philips/real-world-kubernetes","last_synced_at":"2025-04-14T03:35:36.142Z","repository":{"id":66550507,"uuid":"45959955","full_name":"philips/real-world-kubernetes","owner":"philips","description":"disaster recovery, security, and high-availability setups for kubernetes tutorials","archived":false,"fork":false,"pushed_at":"2016-09-01T12:47:12.000Z","size":54,"stargazers_count":149,"open_issues_count":0,"forks_count":26,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-27T17:26:45.485Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/philips.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-11-11T04:36:35.000Z","updated_at":"2023-06-28T07:25:50.000Z","dependencies_parsed_at":"2023-02-28T13:16:18.127Z","dependency_job_id":null,"html_url":"https://github.com/philips/real-world-kubernetes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips%2Freal-world-kubernetes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips%2Freal-world-kubernetes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips%2Freal-world-kubernetes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips%2Freal-world-kubernetes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/philips","download_url":"https://codeload.github.com/philips/real-world-kubernetes/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248817282,"owners_count":21166216,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-06T13:16:30.269Z","updated_at":"2025-04-14T03:35:36.123Z","avatar_url":"https://github.com/philips.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\nWe are going to turn on three CoreOS VMs under vagrant and set them under various configurations to show off different failure domains of Kubernetes and how to handle them in production.\n\n```\ngit clone https://github.com/coreos/coreos-vagrant\ncd coreos-vagrant\ngit clone https://github.com/philips/real-world-kubernetes\nsed -e 's%num_instances=1%num_instances=3%g' \u003c config.rb.sample \u003e config.rb\n```\n\nNOTE: please use the latest version of CoreOS alpha box\n\n```\nvagrant box update --box coreos-alpha\nvagrant box update --box coreos-alpha --provider vmware_fusion\n```\n\nNow lets startup the hosts\n\n```\nvagrant up\nvagrant status\n```\n\nAnd configure ssh to talk to the new vagrant hosts correctly:\n\n```\nvagrant ssh-config \u003e ssh-config\nalias ssh=\"ssh -F ssh-config\"\nalias scp=\"scp -F ssh-config\"\n```\n\nThis should show three healthy CoreOS hosts launched.\n\n# etcd clustering\n\nFor etcd we are going to scale the cluster from a single machine up to a three machine cluster. Then we will fail a machine and show everything is still working.\n\n## Single Machine\n\nSetup an etcd cluster with a single machine on core-01. This is as easy as starting the etcd2 service on CoreOS.\n\n```\nvagrant up\nvagrant ssh-config \u003e ssh-config\nvagrant ssh core-01\nsudo systemctl start etcd2\nsystemctl status etcd2\n```\n\nConfirm that you can write into etcd now:\n\n```\netcdctl set kubernetes rocks\netcdctl get kubernetes\n```\n\nNow, we can confirm the cluster configuration with the `etcdctl member list` subcommand.\n\n```\netcdctl member list\n```\n\nBy default etcd will listen on localhost and not advertise a public address. We need to fix this before adding additional members. First, tell the cluster the members new address. The IP is the default IP for core-01 in coreos-vagrant.\n\n```\netcdctl member update ce2a822cea30bfca http://172.17.8.101:2380\nUpdated member with ID ce2a822cea30bfca in cluster\n```\n\nLet's reconfigure etcd to listen on public ports to get it ready to cluster. \n\n```\nsudo su \nsudo mkdir /etc/systemd/system/etcd2.service.d/\ncat  \u003c\u003cEOM \u003e /etc/systemd/system/etcd2.service.d/10-listen.conf\n[Service]\nEnvironment=ETCD_NAME=core-01\nEnvironment=ETCD_ADVERTISE_CLIENT_URLS=http://172.17.8.101:2379\nEnvironment=ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379\nEnvironment=ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380\nEOM\n```\n\nAll thats left is to restart etcd and the reconfiguration should be complete.\n\n```\nsudo systemctl daemon-reload\nsudo systemctl restart etcd2\netcdctl get kubernetes\n```\n\n## Add core-02 to the Cluster\n\nNow that core-01 is ready for clustering lets add our first additional cluster member, core-02.\n\n```\nvagrant ssh core-02\netcdctl --peers http://172.17.8.101:2379 set /foobar baz\n```\n\nLogin to core-02 and lets add it to the cluster:\n\n```\netcdctl --peers http://172.17.8.101:2379 member add core-02 http://172.17.8.102:2380\n```\n\nThe above command will dump out a bunch of initial configuration information. Next, we will put that configuration information into the systemd unit file for this member:\n\n```\nsudo su\nmkdir /etc/systemd/system/etcd2.service.d/\ncat  \u003c\u003cEOM \u003e /etc/systemd/system/etcd2.service.d/10-listen.conf\n[Service]\nEnvironment=ETCD_NAME=core-02\nEnvironment=ETCD_INITIAL_CLUSTER=core-01=http://172.17.8.101:2380,core-02=http://172.17.8.102:2380\nEnvironment=ETCD_INITIAL_CLUSTER_STATE=existing\n\nEnvironment=ETCD_ADVERTISE_CLIENT_URLS=http://172.17.8.102:2379\nEnvironment=ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379\nEnvironment=ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380\nEOM\n```\n\n```\nsudo systemctl daemon-reload\nsudo systemctl restart etcd2\netcdctl member list\n```\n\nNow, at this point the cluster is in an unsafe configuration. If either machine fails etcd will stop working.\n\n```\nsudo systemctl stop etcd2\nexit\nvagrant ssh core-01\nsudo etcdctl set kubernetes bad\nsudo etcdctl get kubernetes\nexit\nvagrant ssh core-02\nsudo systemctl start etcd2\nsudo etcdctl set kubernetes awesome\n```\n\n## Add core-03 to the Cluster\n\nTo get out of this unsafe configuration lets add a third member. After adding this third member this cluster will be able to survive single machine failures.\n\n```\nvagrant ssh core-03\netcdctl --peers http://172.17.8.101:2379 member add core-03 http://172.17.8.103:2380\n```\n\n```\nsudo su\nmkdir /etc/systemd/system/etcd2.service.d/\ncat  \u003c\u003cEOM \u003e /etc/systemd/system/etcd2.service.d/10-listen.conf\n[Service]\nEnvironment=ETCD_NAME=core-03\nEnvironment=ETCD_INITIAL_CLUSTER=core-01=http://172.17.8.101:2380,core-02=http://172.17.8.102:2380,core-03=http://172.17.8.103:2380\nEnvironment=ETCD_INITIAL_CLUSTER_STATE=existing\n\nEnvironment=ETCD_ADVERTISE_CLIENT_URLS=http://172.17.8.103:2379\nEnvironment=ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379\nEnvironment=ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380\nEOM\n\n```\n\n```\nsudo systemctl daemon-reload\nsudo systemctl restart etcd2\netcdctl member list\n```\n\n## Surviving Machine Failure\n\nNow we can have a single machine fail, like core-01, and have the cluster configure to set and retrieve values.\n\n```\nvagrant destroy core-01\nvagrant ssh core-02\netcdctl set foobar asdf\n```\n\n## Automatic Bootstrapping\n\nThis exercise was designed to get you comfortable with etcd bringup and reconfiguration. In environments where you have deterministic IP address you can use [static cluster bringup](https://coreos.com/etcd/docs/latest/clustering.html#static). In environments with dynamic IPs you can use an [etcd discovery](https://coreos.com/etcd/docs/latest/clustering.html#discovery).\n\n## Cleanup\n\nFor the next exercise we are only going to use a single member etcd cluster. Lets destroy the machines and bring up clean hosts:\n\n```\nvagrant destroy -f core-01\nvagrant destroy -f core-02\nvagrant destroy -f core-03\n```\n\n# Disaster Recovery of etcd\n\nIn most all environments etcd will be replicated. But, etcd is generally holding onto critical data so you should plan for backups and disaster recovery. This example will cover restoring etcd from backup.\n\n## Start a Cluster and Destroy\n\nBring up the cluster of three machines\n\n```\nvagrant up\nvagrant ssh-config \u003e ssh-config\n```\n\nStartup a single machine etcd cluster on core-01 and launch a process that will write a key now every 5 seconds with the current date.\n\n```\nssh core-02\nsystemctl start etcd2\nsudo systemd-run /bin/sh -c 'while true; do  etcdctl set now \"$(date)\"; sleep 5; done'\nexit\n```\n\nBackup the etcd cluster state to a tar file and save it on the local filesystem. In a production cluster this could be done with a tool like rclone in a container to save it to an object store or another server.\n\n```\nssh core-02 sudo tar cfz - /var/lib/etcd2 \u003e backup.tar.gz\nssh core-02 etcdctl get now\nvagrant destroy -f core-02\nvagrant up core-02\nvagrant ssh-config \u003e ssh-config\n```\n\n## Restore from Backup\n\nFirst, lets restore the data from the etcd member to the new host location:\n\n```\nscp backup.tar.gz core-01:\nssh core-01\ntar xzvf backup.tar.gz\nsudo su\nmv var/lib/etcd2/member /var/lib/etcd2/\nchown -R etcd /var/lib/etcd2\n```\n\nNext we need to tell etcd to start but to only use the data, not the cluster configuration. We do this by setting a flag called FORCE_NEW_CLUSTER. This is something like \"single user mode\" on a Linux host.\n\n```\nmkdir -p /run/systemd/system/etcd2.service.d\ncat  \u003c\u003cEOM \u003e /run/systemd/system/etcd2.service.d/10-new-cluster.conf\n[Service]\nEnvironment=ETCD_FORCE_NEW_CLUSTER=1\nEOM\nsystemctl daemon-reload\nsystemctl restart etcd2\n```\n\nTo ensure we don't accidently reset the cluster configuration in the future, remove the force new cluster option and flush it from systemd.\n\n```\nrm /run/systemd/system/etcd2.service.d/10-new-cluster.conf\nsystemctl daemon-reload\n```\n\nNow, we should have our database fully recovered with the application data intact. From here we can rebuild the cluster using the methods from the first section.\n\n```\netcdctl member list\netcdctl get now\n```\n\n## Cleanup\n\n```\nvagrant destroy -f core-01\nvagrant destroy -f core-02\nvagrant destroy -f core-03\n```\n\n# Securing etcd\n\nNow that we have good practice with cluster operations of etcd under network partition, adding/removing members, and backups lets add transport security to the machine that will act as our etcd machine: core-01.\n\n```\nvagrant up\nvagrant ssh-config \u003e ssh-config\n```\n\n## Generate Certificate Authority\n\nFirst, lets generate a certificate authority and some certificates signed by that authority. You can take a look at the makefile but it essentially using the [cfssl](https://github.com/cloudflare/cfssl) tool to generate a CA and an etcd cert signed by that CA.\n\n```\npushd real-world-kubernetes/tls-setup\nmake install-cfssl\nmake\npopd\n```\n\nNow drop the certs onto the host:\n\n```\nscp -r real-world-kubernetes/tls-setup/certs core-01:\n```\n\n## Use CA with etcd\n\nInstall the newly generated certificates onto the host. In a real-world environment this would be done with a cloud-config or installed on first boot.\n\n```\nssh core-01\nsudo su\nmkdir /etc/etcd\nmv certs/etcd* /etc/etcd\nchown -R etcd: /etc/etcd\ncp certs/ca.pem /etc/ssl/certs/\n/usr/bin/c_rehash\nexit\nexit\n```\n\nFinally, prepare etcd to use a certificate and key file that are dropped onto the host.\n\n```\nssh core-01\nsudo su\nmkdir /etc/systemd/system/etcd2.service.d/\ncat  \u003c\u003cEOM \u003e /etc/systemd/system/etcd2.service.d/10-listen.conf\n[Service]\nEnvironment=ETCD_NAME=core-01\nEnvironment=ETCD_ADVERTISE_CLIENT_URLS=https://core-01:2379\nEnvironment=ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379\nEnvironment=ETCD_CERT_FILE=/etc/etcd/etcd.pem\nEnvironment=ETCD_KEY_FILE=/etc/etcd/etcd-key.pem\nEOM\nsystemctl daemon-reload\nsystemctl restart etcd2\nexit\nexit\n```\n\n## Test with etcdctl\n\nWith everything in place we should be able to set a key over a secure connection:\n\n```\nssh core-01\netcdctl --peers https://core-01:2379 --ca-file certs/ca.pem set kubernetes is-ready\n```\n\n# Running Kubernetes API Server\n\nNow that we fully understand etcd and how to operate it securely and in clusters lets bringup a Kubernetes API server.\n\n\n```\nCONTROLLER=core-01\n```\n\nGet the basic configuration files in place on the server.\n\n```\nscp -r real-world-kubernetes/k8s-setup ${CONTROLLER}:\nscp -r real-world-kubernetes/k8s-srv-setup ${CONTROLLER}:\n```\n\nThen copy them over to the right locations and restart the kubelet to have it bootstrap the API server.\n\n\n```\nssh ${CONTROLLER}\nsudo su\nmkdir -p /etc/kubernetes/ssl/\ncp certs/ca.pem /etc/kubernetes/ssl/\ncp certs/apiserver* /etc/kubernetes/ssl/\ncp k8s-setup/kubelet.service /etc/systemd/system/kubelet.service\nmkdir -p /etc/kubernetes/manifests/\ncp k8s-setup/kube-*.yaml /etc/kubernetes/manifests/\nmkdir -p /srv/kubernetes/manifests\ncp k8s-srv-setup/*.yaml /srv/kubernetes/manifests\nsystemctl daemon-reload\nsystemctl restart kubelet.service\nsystemctl enable kubelet\nexit\nexit\n```\n\n## Test API\n\nAt this point the API should be up and available. But, we need a DNS entry to point at; so lets set that up first on our workstations. NOTE: this IP will change based on host configuration. \n\n```\nexport CORE_01_IP=$(cat ssh-config | grep HostName | awk '{print $2}' | head -n1)\nsudo -E /bin/sh -c 'echo \"${CORE_01_IP} core-01\" \u003e\u003e /etc/hosts'\n```\n\nWith DNS configured we can try `kubectl` with our pre-made configuration file:\n\n```\nexport KUBECONFIG=real-world-kubernetes/kubeconfig\nkubectl get pods\n```\n\nIf all goes well we shoudl get an empty list of pods! Now, lets add some worker nodes.\n\n# Kubernetes API Server Under etcd Failure\n\n## Temporary Partition\n\nLets start a really boring job on the cluster that just sleeps forever. This goes through just fine:\n\n```\nkubectl run pause --image=gcr.io/google_containers/pause\n```\n\nNext, we will stop etcd simulating a partition:\n\n```\nssh core-01 sudo systemctl stop etcd2\n```\n\nAttempting to do any API call to the server is going to fail blocked on etcd. This behavior is identical to if you had a web service and stopped its SQL database.\n\n```\nkubectl describe rc pause\n```\n\nLets start up etcd and get things going:\n\n```\nssh core-01 sudo systemctl start etcd2\n``` \n\nAfter a few seconds the API server should start responding and we should be able to get the status of our replication controller:\n\n```\nkubectl describe rc pause\n```\n\n## Data-loss and Restore\n\nLets run a really boring application in this cluster with no nodes:\n\n```\nkubectl run pause --image=gcr.io/google_containers/pause\n```\n\nAnd take a quick backup of etcd:\n\n```\nssh core-01 sudo tar cfz - /var/lib/etcd2 \u003e backup.tar.gz\n```\n\n```\nkubectl scale rc pause --replicas=5\nkubectl describe rc pause\n```\n\n```\nssh core-01 sudo systemctl stop etcd2\n```\n\n```\nssh core-01\nsudo su\nmkdir tmp\nmv /etc/kubernetes/manifests/kube-* tmp/\nrm -Rf /var/lib/etcd2/*\nexit\ndocker ps\nexit\n```\n\n```\nscp backup.tar.gz core-01:\nssh core-01\ntar xzvf backup.tar.gz\nsudo su\nmv var/lib/etcd2/member /var/lib/etcd2/\nchown -R etcd /var/lib/etcd2\nsystemctl start etcd2.service\nmv tmp/* /etc/kubernetes/manifests\nexit\netcdctl --peers https://core-01:2379 --ca-file certs/ca.pem set kubernetes is-ready\nexit\n```\n\n```\nkubectl describe rc pause\nkubectl scale rc pause --replicas=1\n```\n\n# Kubernetes Workers \n\n## Setup Workers\n\nLets setup core-02 and core-03 as the worker machines.\n\n```\nWORKER=core-02\n```\n\n```\nscp -r real-world-kubernetes/worker-setup ${WORKER}:\nscp -r real-world-kubernetes/tls-setup/certs ${WORKER}:\nssh ${WORKER}\nsudo su\nmkdir -p /etc/kubernetes/ssl/ /etc/kubernetes/manifests\ncp certs/ca.pem /etc/kubernetes/ssl/\ncp certs/worker* /etc/kubernetes/ssl/\ncp worker-setup/kubelet.service /etc/systemd/system/kubelet.service\ncp worker-setup/kube-*.yaml /etc/kubernetes/manifests/\ncp worker-setup/worker-kubeconfig.yaml /etc/kubernetes\nsystemctl daemon-reload\nsystemctl restart kubelet.service\nsystemctl enable kubelet\nexit\nexit\n```\n\nNow re-run the above after changing the worker variable to setup core-03:\n\n```\nWORKER=core-03\n```\n\nAt this point we should see two machines listed in the set of machines\n\n```\nkubectl get nodes\n```\n\n## Individual Worker Failure\n\n\n```\nkubectl describe rc pause\nkubectl scale rc pause --replicas=10\n```\n\n```\nvagrant halt core-01\n```\n\nNow after about one minute we should notice everything has been moved off of core-03. Why? It is because of a thing called the \n\n```\nkubectl describe node core-03 core-02\n```\n\n# High Availability of API Server\n\nThis is easy, the API server is trivially horizontally scalable.\n\nSet the controller to core-02 and re-run the controller provisioning steps from before against this host.\n\n```\nCONTROLLER=core-02\n```\n\nNow test out that the kubernetes services are running.\n\n```\nkubectl -s https://core-02 get pods\n```\n\n# High-Availability of Scheduler/Controller-Manager\n\nThere is a service called the pod master that moves things in and out of the kubelet's manifest directory based on compare and swap operations in etcd.\n\n\nLet's force a master election of the scheduler by removing all of the control pieces, including the pod master out of the host.\n\n```\nssh core-01\nsudo su\nmkdir tmp\nmv /etc/kubernetes/manifests/* tmp\n```\n\nAfter a few seconds we should see that the scheduler and controller manager get master elected over to \n\n```\nssh core-02\nsudo su\nls -la /etc/kubernetes/manifests\n```\n\nWith this mechanism in place and an HA etcd cluster you can rest easy knowing the control plane won't go down in the face of single machine failure.\n\n## Cleanup\n\n# Upgrading Control Cluster\n\nNow upgrading under the control plane between minor versions is something you might realistically do. However, these sorts of scenarios aren't really tested.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilips%2Freal-world-kubernetes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphilips%2Freal-world-kubernetes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilips%2Freal-world-kubernetes/lists"}