{"id":22475455,"url":"https://github.com/philips/2016-linuxcon-na-coreos-a-tutorial-on-hyperscale-infrastructure","last_synced_at":"2025-08-02T11:32:33.619Z","repository":{"id":66550381,"uuid":"66172169","full_name":"philips/2016-LinuxCon-NA-CoreOS-A-Tutorial-on-Hyperscale-Infrastructure","owner":"philips","description":null,"archived":false,"fork":false,"pushed_at":"2016-08-22T21:11:09.000Z","size":73,"stargazers_count":14,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-04-15T13:33:31.490Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/philips.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-08-20T23:17:35.000Z","updated_at":"2024-06-19T08:07:20.024Z","dependencies_parsed_at":"2023-02-28T13:16:18.849Z","dependency_job_id":null,"html_url":"https://github.com/philips/2016-LinuxCon-NA-CoreOS-A-Tutorial-on-Hyperscale-Infrastructure","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/philips/2016-LinuxCon-NA-CoreOS-A-Tutorial-on-Hyperscale-Infrastructure","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips%2F2016-LinuxCon-NA-CoreOS-A-Tutorial-on-Hyperscale-Infrastructure","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips%2F2016-LinuxCon-NA-CoreOS-A-Tutorial-on-Hyperscale-Infrastructure/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips%2F2016-LinuxCon-NA-CoreOS-A-Tutorial-on-Hyperscale-Infrastructure/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips%2F2016-LinuxCon-NA-CoreOS-A-Tutorial-on-Hyperscale-Infrastructure/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/philips","download_url":"https://codeload.github.com/philips/2016-LinuxCon-NA-CoreOS-A-Tutorial-on-Hyperscale-Infrastructure/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips%2F2016-LinuxCon-NA-CoreOS-A-Tutorial-on-Hyperscale-Infrastructure/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268378965,"owners_count":24240907,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-06T13:17:16.899Z","updated_at":"2025-08-02T11:32:33.611Z","avatar_url":"https://github.com/philips.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\nThis tutorial will demo a number of technologies. The requirements for completing each section will be introduced in the section.\n\nThis presentation was given at LinuxCon 2016 in Toronto. There was a live tutorial given and the video is not yet available. However, there are [slides](https://speakerdeck.com/philips/coreos-a-tutorial-on-hyperscale-infrastructure).\n\n## etcd Basics\n\n**Pre-requisites**\n\n- [etcd and etcdctl](https://github.com/coreos/etcd/releases/tag/v3.0.6) for your platform\n\nFirst, run `etcd` in a terminal window.\n\n```\n./etcd\n...\n```\n\nStoring and retrieving values is done simply using the put/get subcommands.\n\n```\nexport ETCDCTL_API=3\n./etcdctl put foo bar\n```\n\n```\n./etcdctl get foo\n```\n\nWith the `-w` flag additional information can be found. Notice the \"revision\"? With etcd all keys are revisioned and you can use this revision number to get old values of keys, setup multi-key transactions, and view all changes since a certain time.\n\nUsing two differnt put calls we will create a couple of known revisions for the key `foo`.\n\n```\n./etcdctl put foo bar -w json\n{\"header\":{\"cluster_id\":17237436991929493444,\"member_id\":9372538179322589801,\"revision\":9,\"raft_term\":2}}\n```\n\n```\n./etcdctl put foo toronto -w json\n{\"header\":{\"cluster_id\":17237436991929493444,\"member_id\":9372538179322589801,\"revision\":11,\"raft_term\":2}}\n```\n\nWith the revision number etcd can \"time-travel\" and look at old values of `foo`:\n\n```\n./etcdctl get foo --rev 9\nfoo\nbar\n```\n\n```\n./etcdctl get foo --rev 11\nfoo\ntoronto\n```\n\n```\n./etcdctl get foo -w json\n{\"header\":{\"cluster_id\":17237436991929493444,\"member_id\":9372538179322589801,\"revision\":11,\"raft_term\":2},\"kvs\":[{\"key\":\"Zm9v\",\"create_revision\":2,\"mod_revision\":11,\"version\":10,\"value\":\"dG9yb250bw==\"}],\"count\":1}\n```\n\n## etcd Clustering\n\n**Pre-requisites**\n\n- A working [Go environment](https://golang.org/doc/install)\n- Follow the upstream guide to [setup a local cluster](https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/local_cluster.md#local-multi-member-cluster)\n\nAfter setting up the pre-requisites a three node etcd cluster will be running. \n\nMembers of the cluster can be listed like this:\n\n```\n./etcdctl member list\n8211f1d0f64f3269, started, infra1, http://127.0.0.1:12380, http://127.0.0.1:2379\n91bc3c398fb3c146, started, infra2, http://127.0.0.1:22380, http://127.0.0.1:22379\nfd422379fda50e48, started, infra3, http://127.0.0.1:32380, http://127.0.0.1:32379\n```\n \n## Building an Application\n\n**Pre-Requisites**\n\n- A working local Docker client (`brew install docker`)\n- A VM to run Docker, recommend [minikube](https://github.com/kubernetes/minikube/releases)\n\n```\ngit clone https://github.com/philips/2016-LinuxCon-NA-CoreOS-A-Tutorial-on-Hyperscale-Infrastructure\ncd guestbook/v1\neval $(minikube docker-env)\nVERSION=v1 REGISTRY=quay.io/philips make build\nVERSION=v1 REGISTRY=quay.io/philips make push\n```\n\n## Running a Single Container\n\n**Pre-Requisites**\n- Any CoreOS virtual machine and SSH session\n\nNow that we have built the container it is easy to run on a virtual machine with rkt:\n\n```\nrkt fetch docker://quay.io/philips/guestbook:v1 --insecure-options=image\n```\n\nNow, one thing to note is that rkt does not have a daemon. So, we really on your system init system to monitor the process. To do that quickly under systemd do something like this:\n\n```\nsystemd-run rkt fetch docker://quay.io/philips/guestbook:v1 --insecure-options=image\n```\n\nOr with docker:\n\n```\ndocker run quay.io/philips/guestbook:v1 \n```\n\n## Debugging with Toolbox\n\n**Pre-Requisites**\n- Any CoreOS virtual machine and SSH session\n\nWith fewer pieces of software on the host what happens to debugging tools? CoreOS provides a quick solution with `toolbox`:\n\n```\ntoolbox\n```\n\nThe environment can be customized to run any container whether that is Debian, Ubuntu, Fedora, Arch, etc.\n\n\n## Kubernetes Basics\n\n**Pre-Requisites**\n\n- An [AWS account](http://aws.amazon.com/) and [AWS cli](https://aws.amazon.com/cli/)\n  - An [AWS keypair for us-west-2](https://us-west-2.console.aws.amazon.com/ec2/v2/home?region=us-west-2#KeyPairs:sort=keyName)\n- [kube-aws](https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html) installed and in your path\n- [kubectl 1.3.4+](https://coreos.com/kubernetes/docs/latest/configure-kubectl.html) installed and in your path\n- Follow the [Kubernetes + CoreOS + AWS docs](https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html). \n\nWith a working Kubernetes cluster it is possible to proxy through to localhost for development without having to worry about auth:\n\n```\nkubectl proxy\n```\n\nFrom there the API becomes very accessible using well-known tools like curl:\n\n```\ncurl 127.0.0.1:8001/api/v1/services\n```\n\nThe equivalent of this rest API call is:\n\n```\nkubectl describes services\n```\n\n## Kubernetes App Deployments\n\nFirst, lets run the app that was built earlier and pushed to quay.io.\n\n```\nkubectl run guestbook --image quay.io/philips/guestbook:v1 -l app=guestbook\n```\n\nConfirm that the application is running by selecting all things that have `app=guestbook`:\n\n```\nkubectl get pods -l app=guestbook\nNAME                         READY     STATUS    RESTARTS   AGE\nguestbook-2893398214-x04rm   1/1       Running   0          4m\n```\n\nNeat! Now, connect to our application by selecting that pod process and forwarding the port.\n\n```\nkubectl port-forward $(kubectl get pods -l app=guestbook -o template --template=\"{{range.items}}{{.metadata.name}}{{end}}\") 3000:3000\n```\n\nVisit: http://localhost:3000\n\nSuccess! Kubernetes is now able to manage running a container! Cleanup:\n\n```\nkubectl delete deployment guestbook\n```\n\n## Kubernetes App Failures\n\nSetup the application to run again using the kubectl run subcommand.\n\n```\nkubectl run guestbook --image quay.io/philips/guestbook:v1 -l app=guestbook\nkubectl get pods -l app=guestbook\n```\n\nKubernetes will allow the app instance to be killed.\n\n```\nkubectl delete guestbook-2893398214-ikc58\n```\n\nBut, it will drive the cluster state torwards a single running instance. Within a few seconds a replacement is launched:\n\n```\nkubectl get pods -l app=guestbook\n``` \n\nThe reason that the single pod was replaced is because the deployment, which we will discuss later, that is driving the guestbook application \n\n```\nkubectl describe deployment guestbook\n```\n\n## Kubernetes App Scaling\n\n```\nkubectl scale deployment guestbook --replicas=3\n```\n\n- Third party controllers for complex applications\n\n## Kubernetes Services\n\nEarlier we used port forwarding to confirm the application was running. This is fun, but this isn't terribly useful as no one outside of the cluster can reach our application. Delete the deployment and lets try exposing a port:\n\n```\nkubectl delete deployment guestbook\nkubectl run guestbook --image quay.io/philips/guestbook:v1 -l app=guestbook --expose --port 3000\n```\n\nNow the service will have a cluster IP that is routable to other nodes on the cluster.\n\n```\nkubectl describe service guestbook\n```\n\nOften, this isn't terribly useful as users workstation's rarely are on the same network/VPC/overlay/etc that the cluster is on. The 10.0.0.0/24 address isn't routable. But, the IPs of the nodes are. And by using a type of service called a \"NodePort\" a port on the nodes will forward to the service.\n\nAssuming we setup our cluster at http://toronto.example.com:31512/ \n\n```\nkubectl edit service guestbook\n```\n\nIt would be much more convienent however if the cluster setup a real load balancer. This can be done by editing the type once more from \"NodePort\" to \"LoadBalancer\"\n\n```\nkubectl edit service guestbook\nkubectl describe service guestbook\n```\n\nThis time the service description will include a \"LoadBalancer Ingress\" that is a real LoadBalancer depending on the environment of the cluster.\n\nNow, cleanup everything:\n\n```\nkubectl delete deployment guestbook\nkubectl delete service guestbook\n```\n\n## More on Services\n\nPort-forward cluster local DNS to your workstation.\n\n```\nkubectl port-forward --namespace=kube-system $( kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o template --template=\"{{range.items}}{{.metadata.name}}{{end}}\") 5300:53\n```\n\nTry and grab the redis-master service powering our website:\n\n```\ndig +vc -p 5300 @127.0.0.1  redis-master.default.svc.cluster.local\nredis-master.default.svc.cluster.local. 30 IN A 10.3.0.25\n```\n\nFor more [network debugging tips see this page](https://github.com/coreos/docs/blob/master/kubernetes/network-troubleshooting.md).\n\n## Using Kubernetes in a Development Workflow\n\n**Pre-Requisites**\n\n- A working [Go environment](https://golang.org/doc/install)\n- A working redis cluster from above\n- goreman or another Procfile runner (`go get github.com/mattn/goreman`) \n\nIt is very useful to be able to hack on code locally while using services running the cluster. Let's walk through the development workflow for the Guestbook service.\n\nFirst, this setup is really naive so it only works when there is only one slave. Scale the slave replica set down to 1.\n\n```\nkubectl scale rs redis-slave --replicas=1\n```\n\nNow, inside of the guestbook subdirectory of this repo there is a Procfile. When ran with `goreman start` it will forward the redis master and slave.\n\n```\ngoreman start\n```\n\nAt this point you now have the live cluster database forwarding locally. This can be confirmed by querying the redis database using the CLI tooling:\n\n```\nredis-cli -h 127.0.0.1 -p 6380 keys '*'\n```\n\nNow hacking on the application is easy, simply go into the v2 directory and run:\n\n```\nREDIS_SLAVE=localhost:6379 REDIS_MASTER=localhost:6380 go run main.go\n```\n\nIts the best of both worlds!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilips%2F2016-linuxcon-na-coreos-a-tutorial-on-hyperscale-infrastructure","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphilips%2F2016-linuxcon-na-coreos-a-tutorial-on-hyperscale-infrastructure","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilips%2F2016-linuxcon-na-coreos-a-tutorial-on-hyperscale-infrastructure/lists"}