{"id":13510440,"url":"https://github.com/luxas/kubeadm-workshop","last_synced_at":"2025-04-04T15:11:13.488Z","repository":{"id":65041540,"uuid":"85397032","full_name":"luxas/kubeadm-workshop","owner":"luxas","description":"Showcasing a bare-metal multi-platform kubeadm setup with persistent storage and monitoring","archived":false,"fork":false,"pushed_at":"2020-05-22T15:58:51.000Z","size":931,"stargazers_count":680,"open_issues_count":30,"forks_count":189,"subscribers_count":38,"default_branch":"master","last_synced_at":"2025-03-28T14:11:12.486Z","etag":null,"topics":["custom-metrics","kubeadm","kubernetes","monitoring","multi-platform","persistent-storage","raspberry-pi","rook","workshop"],"latest_commit_sha":null,"homepage":null,"language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luxas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-03-18T11:42:25.000Z","updated_at":"2025-03-12T16:28:14.000Z","dependencies_parsed_at":"2022-12-25T19:20:18.142Z","dependency_job_id":null,"html_url":"https://github.com/luxas/kubeadm-workshop","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luxas%2Fkubeadm-workshop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luxas%2Fkubeadm-workshop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luxas%2Fkubeadm-workshop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luxas%2Fkubeadm-workshop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luxas","download_url":"https://codeload.github.com/luxas/kubeadm-workshop/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247198463,"owners_count":20900080,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["custom-metrics","kubeadm","kubernetes","monitoring","multi-platform","persistent-storage","raspberry-pi","rook","workshop"],"created_at":"2024-08-01T02:01:39.075Z","updated_at":"2025-04-04T15:11:13.473Z","avatar_url":"https://github.com/luxas.png","language":"Makefile","funding_links":[],"categories":["Makefile","raspberry-pi","Tutorials \u0026 Learning"],"sub_categories":[],"readme":"### Workshop:\n\n## Building a multi-platform Kubernetes cluster on bare metal with `kubeadm`\n\nHi and welcome to this tutorial and demonstration of how to build a bare-metal Kubernetes cluster with kubeadm!\n\nI'm one of the main kubeadm developers and very excited about bare metal as well, \nso I thought showing some of the things you can do with Kubernetes/kubeadm would be a great fit!\n\nThis workshop is a part of my talk at KubeCon Berlin: [Autoscaling a Multi-Platform Kubernetes Cluster Built with kubeadm [I] - Lucas Käldström - YouTube](https://youtu.be/ZdzKQwMjg2w)\n\nMy slides for the presentation are here: http://slides.com/lucask/kubecon-berlin\n\n### Highligts\n\n* Showcases what you can do on bare-metal, even behind a firewall with no public IP address\n* Demonstrates usage of cutting-edge technologies like Persistent Storage running on-cluster, Autoscaling based on Custom Metrics and Aggregated API Servers\n\nWhat's more, the Kubernetes yaml manifests included in this repository are multi-architecture and works on ARM, both 32- and 64-bit!\n\nMy own setup at home consists of this hardware:\n - 2x Up Board, 4 cores @ 1.44 GHz, 2 GB RAM, 1 GbE, 16 GB eMMc, amd64, [Link](http://up-shop.org/up-boards/2-up-board-2gb-16-gb-emmc-memory.html)\n - 2x Odroid C2, 4 cores @ 1.5 GHz, 2 GB RAM, 1 GbE, 16 GB eMMc, arm64, [Link](http://www.hardkernel.com/main/products/prdt_info.php)\n - 3x Raspberry Pi, 4 cores @ 1.2 GHz, 1 GB RAM, 100 MbE, 16 GB SD Card, arm/arm64, [Link](https://www.raspberrypi.org/products/raspberry-pi-3-model-b/)\n\n![Picture of the cluster](pictures/cluster.jpg)\n\nSo, no more smalltalk then, let's dive right in!\n\n### Contents\n\nThis workshop is divided into these parts:\n\n* Installing kubeadm on all the machines you want in your cluster\n* Setting up your Kubernetes master\n* Setting up the worker nodes\n* Deploying the Pod networking layer\n* Deploying the Dashboard and Heapster\n* Deploying an Ingress Controller for exposing HTTP services\n* Deploying a persistent storage layer on top of Kubernetes with Rook\n* Deploying InfluxDB and Grafana for storing and visualizing CPU and memory metrics\n* Deploying a extension API Server for extending the Kubernetes API\n* Deploying the Prometheus Operator for monitoring Pods in the cluster\n* Deploying a sample custom metrics API Server\n* Deploying and autoscaling a sample node.js application based on custom metrics\n\n### Installing kubeadm on all the machines you want in your cluster\n\n\u003e WARNING: This workshop uses alpha technologies in order to be on the edge and Kubernetes can't be upgraded.\n\u003e This means the features used and demonstrated here might work differently in v1.7 and backwards-compability isn't guaranteed in any way\n\n**Note:** The first part that describes how to install kubeadm is just copied from the [official kubeadm documentation](https://kubernetes.io/docs/getting-started-guides/kubeadm/)\n\n**Note:** It's expected that you have basic knowledge about how Kubernetes and kubeadm work, because quite advanced concepts are covered in this workshop.\n\n**Note:** This guide has been tested on Ubuntu Xenial, Yakkety and Zesty\n\nYou can install kubeadm easily this way:\n\n```bash\ncurl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\ncat \u003c\u003cEOF \u003e /etc/apt/sources.list.d/kubernetes.list\ndeb http://apt.kubernetes.io/ kubernetes-xenial main\nEOF\napt-get update\napt-get install -y docker.io kubeadm\n```\n\nYou should do this on all machines you're planning to include in your cluster, and these commands are exactly the same regardless on which architecture you are on.\n\n### Setting up your Kubernetes master\n\nSSH into your master node, and switch to the `root` account of the machine or use `sudo` everywhere below.\n\nAs mentioned earlier, experimental features of different kinds will be used in this tutorial to show off the latest and greatest features in Kubernetes.\n\nkubeadm for example, can take options from a configuration file in order to be customized easily.\nBut the API exposed right now is _not_ stable, and under heavy development. So this will definitely change (to the better) in time for v1.7.\n\nThe configuration file we'll use here looks like this in `kubeadm.yaml`:\n\n```yaml\nkind: MasterConfiguration\napiVersion: kubeadm.k8s.io/v1alpha1\ncontrollerManagerExtraArgs:\n  horizontal-pod-autoscaler-use-rest-clients: \"true\"\n  horizontal-pod-autoscaler-sync-period: \"10s\"\n  node-monitor-grace-period: \"10s\"\napiServerExtraArgs:\n  runtime-config: \"api/all=true\"\nkubernetesVersion: \"stable-1.8\"\n```\n\nA brief walkthrough what the statements mean:\n - `horizontal-pod-autoscaler-use-rest-clients: \"true\"` tells the controller manager to look for the [custom metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/custom-metrics-api.md)\n\nYou can now go ahead and initialize the master node with this command (assuming you're `root`, append `sudo` if not):\n\n```console\n$ kubeadm init --config kubeadm.yaml\n```\n\nMake sure you got kubeadm v1.8.0-beta.1 or higher and docker 1.12.x.\nIn order to control your cluster securely, you need to specify the `KUBECONFIG` variable to `kubectl` knows where to look for the admin credentials.\nHere is an example how to do it as a regular user.\n\n```bash\nsudo cp /etc/kubernetes/admin.conf $HOME/\nsudo chown $(id -u):$(id -g) $HOME/admin.conf\nexport KUBECONFIG=$HOME/admin.conf\n```\n\n#### Make the `kube-proxy` DaemonSet multi-platform\n\nSince `kube-proxy` runs in a DaemonSet, it will be scheduled on all nodes. By default, an image with the architecture that `kubeadm init` is run on\nis used in the DaemonSet, so if you ran `kubeadm init` on an `arm64` image, the `kube-proxy` image with be `gcr.io/google_containers/kube-proxy-arm64`.\n\nTo make it possible to add nodes with other architectures we have to switch the image to a manifest list like this. First, make the DaemonSet, rolling-upgradeable\nand then change the image to a manifest list.\n\n```console\n$ kubectl -n kube-system set image daemonset/kube-proxy kube-proxy=luxas/kube-proxy:v1.8.0-beta.1\n```\n\nWith those two commands, `kube-proxy` will come up successfully on whatever node you bring to your cluster.\n\n#### Deploying the Pod networking layer\n\nThe networking layer in Kubernetes is extensible, and you may pick the networking solution that fits you the best.\nI've tested this with Weave Net, but it should work with any other compliant provider.\n\nHere's how to use Weave Net as the networking provider the really easy way:\n\n```console\n$ kubectl apply -f https://git.io/weave-kube-1.6\n```\n\n**OR** you can run these two commands if you want to encrypt the communication between nodes:\n\n```console\n$ kubectl create secret -n kube-system generic weave-passwd --from-literal=weave-passwd=$(hexdump -n 16 -e '4/4 \"%08x\" 1 \"\\n\"' /dev/random)\n$ kubectl apply -n kube-system -f \"https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\\n')\u0026password-secret=weave-passwd\"\n```\n\n### Setting up the worker nodes\n\n`kubeadm init` above will print out a `kubeadm join` command for you to paste for joining the other nodes in your cluster to the master.\n\n**Note:** Make sure you join all nodes before you arch-taint the nodes (if you do)!\n\n```console\n$ kubeadm join --token \u003ctoken\u003e \u003cmaster-ip\u003e:\u003cmaster-port\u003e\n```\n\n#### Taints and tolerations\n\n[`Taints and Tolerations`](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/taint-toleration-dedicated.md)\nis a concept of dedicated nodes. Simply put, if you taint a node with a key/value pair and the effect `NoSchedule`, it will reject all Pods\nthat don't have the same key/value set in the `Tolerations` field of the `PodSpec`.\n\nBy default, the master is tainted with the `node-role.kubernetes.io=\"\"` key/value pair which will make it only allow the `kube-dns` Deployment,\nthe `kube-proxy` DaemonSet and most often the CNI network provider's DaemonSet, because they have the toleration.\n\nIn case you only have one node available for testing and want to run normal workloads on the master as well (allow all workloads on the master),\nrun this command:\n\n```console\n$ kubectl taint nodes --all node-role.kubernetes.io/master-\n```\n\nIn order to make the default architecture `amd64`, and you know you might deploy workloads that aren't multi-platform, it's best to taint the\n\"special\" nodes of an other architecture and explicitely tolerate ARM (32- and 64-bit) on the workloads that support it.\n\nYou can taint your arm and arm64 nodes with these commands:\n\n```console\n$ kubectl taint node \u003carm nodes\u003e beta.kubernetes.io/arch=arm:NoSchedule\n$ kubectl taint node \u003carm64 nodes\u003e beta.kubernetes.io/arch=arm64:NoSchedule\n```\n\n### Deploying the Dashboard and Heapster\n\nI really like visualizing the cluster resources in the [Kubernetes Dashboard](https://github.com/kubernetes/dashboard) (although I'm mostly a CLI guy).\n\nYou can install the dashboard with this command:\n\n```console\n$ curl -sSL https://git.io/kube-dashboard | sed \"s|image:.*|image: luxas/kubernetes-dashboard:v1.6.3|\" | kubectl apply -f -\nserviceaccount \"dashboard\" created\nclusterrolebinding \"dashboard-admin\" created\ndeployment \"kubernetes-dashboard\" created\nservice \"kubernetes-dashboard\" created\n```\n\nYou probably want some monitoring as well, if you install [Heapster](https://github.com/kubernetes/heapster) you can easily keep track of the CPU and\nmemory usage in your cluster. Those stats will also be shown in the dashboard!\n\n```console\n$ kubectl apply -f demos/monitoring/heapster.yaml\nserviceaccount \"heapster\" created\nclusterrolebinding \"heapster\" created\ndeployment \"heapster\" created\nservice \"heapster\" created\n```\n\nYou should now see some Services in the `kube-system` namespace:\n\n```console\n$ kubectl -n kube-system get svc\nNAME                   CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE\nheapster               10.104.142.79   \u003cnone\u003e        80/TCP          5s\nkube-dns               10.96.0.10      \u003cnone\u003e        53/UDP,53/TCP   42s\nkubernetes-dashboard   10.97.73.205    \u003cnone\u003e        80/TCP          11s\n```\n\nAfter `heapster` is up and running (check with `kubectl -n kube-system get pods`), you should be able to see the \nCPU and memory usage of the nodes in the cluster and for individual Pods:\n\n```console\n$ kubectl top nodes\nNAME        CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%\ntest-node   131m         1%        9130Mi          30%\n```\n\n### Deploying an Ingress Controller for exposing HTTP services\n\nNow that you have created the dashboard and heapster Deployments and Services, how can you access them?\n\nOne solution might be making your Services of the NodePort type, but that's not a good long-term solution.\n\nInstead, there is the Ingress object in Kubernetes that let's you create rules for how Services in your cluster should be exposed to the world.\nBefore one can create Ingress rules, you need a Ingress Controller that watches for rules, applies them and forwards requests as specified.\n\nOne Ingress Controller provider is [Traefik](traefik.io), and I'm using that one here.\n\nIn this demo I go a step further. Normally in order to expose your app you have locally to the internet requires that one of your machines has a public Internet\naddress. We can workaround this very smoothly in a Kubernetes cluster by letting [Ngrok](ngrok.io) forward requests from a public subdomain of `ngrok.io` to the\nTraefik Ingress Controller that's running in our cluster.\n\nUsing ngrok here is perfect for hybrid clusters where you have no control over the network you're connected to... you just have internet access.\nAlso, this method is can be used in nearly any environment and will behave the same. But for production deployments (which we aren't dealing with here),\nyou should of course expose a real loadbalancer node with a public IP.\n\n```console\n$ kubectl apply -f demos/loadbalancing/traefik-common.yaml\nclusterrole \"traefik-ingress-controller\" created\nserviceaccount \"traefik-ingress-controller\" created\nclusterrolebinding \"traefik-ingress-controller\" created\nconfigmap \"traefik-cfg\" created\n\n$ kubectl apply -f demos/loadbalancing/traefik-ngrok.yaml\ndeployment \"traefik-ingress-controller\" created\nservice \"traefik-ingress-controller\" created\nservice \"traefik-web\" created\nconfigmap \"ngrok-cfg\" created\ndeployment \"ngrok\" created\nservice \"ngrok\" created\n\n$ curl -sSL $(kubectl -n kube-system get svc ngrok -o template --template \"{{.spec.clusterIP}}\")/api/tunnels | jq  \".tunnels[].public_url\" | sed 's/\"//g;/http:/d'\nhttps://foobarxyz.ngrok.io\n```\n\nYou can now try to access the ngrok URL that got outputted by the above command. It first ask you for a password, then return 404 due to the absence of Ingress\nrules.\n\n![Authenticate to Traefik](pictures/basicauth.png)\n\n![404 with no Ingress rules](pictures/404-traefik.png)\n\nLet's change that by creating an Ingress rule!\n\n#### Exposing the Dashboard via the Ingress Controller\n\nWe want to expose the dashboard to our newly-created public URL, under the `/dashboard` path.\n\nThat's easily achievable using this command:\n\n```console\n$ kubectl apply -f demos/dashboard/ingress.yaml\ningress \"kubernetes-dashboard\" created\n```\n\nThe Traefik Ingress Controller is set up to require basic auth before one can access the services.\n\nI've set the username to `kubernetes` and the password to `rocks!`. You can obviously change this if you want by editing the `traefik-common.yaml` before deploying\nthe Ingress Controller.\n\nWhen you've signed in to `https://{ngrok url}/dashboard/` (note the `/` in the end, it's required), you'll see a dashboard like this:\n\n![The Kubernetes Dashboard](pictures/dashboard.png)\n\n### Deploying a persistent storage layer on top of Kubernetes with Rook\n\nStateless services are cool, but deploying stateful applications on your Kubernetes cluster is even more fun.\n\nFor that you need somewhere to store persistent data, and that's not easy to achieve on bare metal. [Rook](https://github.com/rook/rook) is a promising project\naiming to solve this by building a Kubernetes integration layer upon the battle-tested Ceph storage solution.\n\nRook is using `ThirdPartyResources` for knowing how to set up your storage solution, and has an [operator](https://github.com/rook/rook/tree/master/cmd/rook-operator)\nthat is listening for these TPRs.\n\nHere is how to create a default Rook cluster by deploying the operator, a controller that will listen for PersistentVolumeClaims that need binding, a Rook Cluster\nThirdPartyResource and finally a StorageClass.\n\n```console\n$ kubectl apply -f https://raw.githubusercontent.com/rook/rook/release-0.5/cluster/examples/kubernetes/rook-operator.yaml\nclusterrole \"rook-operator\" created\nserviceaccount \"rook-operator\" created\nclusterrolebinding \"rook-operator\" created\ndeployment \"rook-operator\" created\n\n$ kubectl apply -f https://raw.githubusercontent.com/rook/rook/release-0.5/cluster/examples/kubernetes/rook-cluster.yaml\ncluster \"my-rook\" created\n\n$ kubectl apply -f https://raw.githubusercontent.com/rook/rook/release-0.5/cluster/examples/kubernetes/rook-storageclass.yaml\npool \"replicapool\" created\nstorageclass \"rook-block\" created\n\n$ # Repeat this step for all namespaces you want to deploy PersistentVolumes with Rook in\n$ kubectl get secret rook-rook-user -oyaml | sed \"/resourceVer/d;/uid/d;/self/d;/creat/d;/namespace/d\" | kubectl -n kube-system apply -f -\nsecret \"rook-rook-user\" created\n\n$ # In order to make Rook the default Storage Provider by making the `rook-block` Storage Class the default, run this:\n$ kubectl patch storageclass rook-block -p '{\"metadata\":{\"annotations\": {\"storageclass.kubernetes.io/is-default-class\": \"true\"}}}'\nstorageclass \"rook-block\" patched\n\n$ apt-get update \u0026\u0026 apt-get install ceph-common -y\n```\n\nOne limitation with v0.3.0 is that you can't control to which namespaces the rook authentication Secret should be deployed, so if you want to create\n`PersistentVolumes` in an other namespace than `default`, run the above `kubectl` command.\n\n### Deploying InfluxDB and Grafana for storing and visualizing CPU and memory metrics\n\nNow that we have got persistent storage in our cluster, we can deploy some stateful services. For example, we can store monitoring data aggregated by Heapster\nin an InfluxDB database and visualize that data with a Grafana dashboard.\n\nYou must do this if you want to gather CPU/memory data from Heapster for a longer time, by default heapster just saves data from the latest couple of minutes.\n\n```console\n$ kubectl apply -f demos/monitoring/influx-grafana.yaml\npersistentvolumeclaim \"grafana-pv-claim\" created\npersistentvolumeclaim \"influxdb-pv-claim\" created\ndeployment \"monitoring-grafana\" created\nservice \"monitoring-grafana\" created\ndeployment \"monitoring-influxdb\" created\nservice \"monitoring-influxdb\" created\ningress \"monitoring-grafana\" created\n```\n\nNote that an Ingress rule was created for Grafana automatically. You can access your Grafana instance at the `https://{ngrok url}/grafana/` URL.\n\n![Grafana dashboard](pictures/grafana.png)\n\n### Sample API Server\n\nThe core API Server is great, but what about if you want to write your own, extended API server that contains more high-level features that build on top of Kubernetes\nbut still be able to control those high-level features from kubectl? This is now possible using the API Aggregation feature that will make it into beta in v1.7\n\nFirst, let's check which API groups are available normally:\n\n```console\n$ kubectl api-versions\napiregistration.k8s.io/v1beta1\napps/v1beta1\nauthentication.k8s.io/v1\nauthentication.k8s.io/v1beta1\nauthorization.k8s.io/v1\nauthorization.k8s.io/v1beta1\nautoscaling/v1\nautoscaling/v2alpha1\nbatch/v1\nbatch/v2alpha1\ncertificates.k8s.io/v1beta1\nextensions/v1beta1\npolicy/v1beta1\nrbac.authorization.k8s.io/v1alpha1\nrbac.authorization.k8s.io/v1beta1\nrook.io/v1beta1\nsettings.k8s.io/v1alpha1\nstorage.k8s.io/v1\nstorage.k8s.io/v1beta1\nv1\n```\n\nIt's pretty straightforward to write your own API server now with the break-out of [`k8s.io/apiserver`](https://github.com/kubernetes/apiserver).\nThe `sig-api-machinery` team has also given us a sample implementation: [`k8s.io/sample-apiserver`](https://github.com/kubernetes/sample-apiserver).\n\nThe sample API Server called wardle, contains one API group: `wardle.k8s.io/v1alpha1` and one API resource in that group: `Flunder`\nThis guide shows how easy it will be to extend the Kubernetes API in the future.\n\nThe sample API Server saves its data to a separate etcd instance running in-cluster. Notice the PersistentVolume that is created for etcd for that purpose.\nNote that in the future, the etcd Operator should probably be used for running etcd instead of running it manually like now.\n\n```console\n$ kubectl apply -f demos/sample-apiserver/wardle.yaml\nnamespace \"wardle\" created\npersistentvolumeclaim \"etcd-pv-claim\" created\nserviceaccount \"apiserver\" created\nclusterrolebinding \"wardle:system:auth-delegator\" created\nrolebinding \"wardle-auth-reader\" created\ndeployment \"wardle-apiserver\" created\nservice \"api\" created\napiservice \"v1alpha1.wardle.k8s.io\" created\n\n$ kubectl get secret rook-rook-user -oyaml | sed \"/resourceVer/d;/uid/d;/self/d;/creat/d;/namespace/d\" | kubectl -n wardle apply -f -\nsecret \"rook-rook-user\" created\n```\n\nAfter a few minutes, when the extended API server is up and running, `kubectl` will auto-discover that API group and it will be possible to\ncreate, list and delete Flunder objects just as any other API object.\n\n```console\n$ kubectl api-versions\napiregistration.k8s.io/v1beta1\napps/v1beta1\nauthentication.k8s.io/v1\nauthentication.k8s.io/v1beta1\nauthorization.k8s.io/v1\nauthorization.k8s.io/v1beta1\nautoscaling/v1\nautoscaling/v2alpha1\nbatch/v1\nbatch/v2alpha1\ncertificates.k8s.io/v1beta1\nextensions/v1beta1\npolicy/v1beta1\nrbac.authorization.k8s.io/v1alpha1\nrbac.authorization.k8s.io/v1beta1\nrook.io/v1beta1\nsettings.k8s.io/v1alpha1\nstorage.k8s.io/v1\nstorage.k8s.io/v1beta1\nv1\n***wardle.k8s.io/v1alpha1***\n\n$ # There is no foobarbaz resource, but the flunders resource does now exist\n$ kubectl get foobarbaz\nthe server doesn't have a resource type \"foobarbaz\"\n\n$ kubectl get flunders\nNo resources found.\n\n$ kubectl apply -f demos/sample-apiserver/my-flunder.yaml\nflunder \"my-first-flunder\" created\n```\n\nIf you want to make sure this is real, you can check the etcd database running in-cluster with this command:\n\n```console\n$ kubectl -n wardle exec -it $(kubectl -n wardle get po -l app=wardle-apiserver -otemplate --template \"{{ (index .items 0).metadata.name}}\") -c etcd /bin/sh -- -c \"ETCDCTL_API=3 etcdctl get /registry/wardle.kubernetes.io/registry/wardle.kubernetes.io/wardle.k8s.io/flunders/my-first-flunder\" | grep -v /registry/wardle | jq .\n{\n  \"kind\": \"Flunder\",\n  \"apiVersion\": \"wardle.k8s.io/v1alpha1\",\n  \"metadata\": {\n    \"name\": \"my-first-flunder\",\n    \"uid\": \"bef75e16-2c5b-11e7-999c-1602732a5d02\",\n    \"creationTimestamp\": \"2017-04-28T21:43:41Z\",\n    \"labels\": {\n      \"sample-label\": \"true\"\n    },\n    \"annotations\": {\n      \"kubectl.kubernetes.io/last-applied-configuration\": \"{\\\"apiVersion\\\":\\\"wardle.k8s.io/v1alpha1\\\",\\\"kind\\\":\\\"Flunder\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"labels\\\":{\\\"sample-label\\\":\\\"true\\\"},\\\"name\\\":\\\"my-first-flunder\\\",\\\"namespace\\\":\\\"default\\\"}}\\n\"\n    }\n  },\n  \"spec\": {},\n  \"status\": {}\n}\n```\n\nConclusion, the Flunder object we created was saved in the separate etcd instance!\n\n### Deploying the Prometheus Operator for monitoring Services in the cluster\n\n[Prometheus](prometheus.io) is a great monitoring solution, and combining it with Kubernetes makes it even more awesome.\n\nThese commands will first deploy the [Prometheus operator](https://github.com/coreos/prometheus-operator) as well as one Prometheus instance by creating a `Prometheus`\nThirdPartyResource.\n\nA lightweight nodejs application is deployed as well, which exports the `http_requests_total` metric at `/metrics`.\nA `ServiceMonitor` ThirdPartyResource is created that match the sample metrics app by the `app=sample-metrics-app` label.\n\nThe ServiceMonitor will make the Prometheus instance scrape metrics from the sample metrics web app.\n\nYou can access the Prometheus web UI via the NodePort or the internal Service.\n\n```console\n$ kubectl apply -f demos/monitoring/prometheus-operator.yaml\nclusterrole \"prometheus-operator\" created\nserviceaccount \"prometheus-operator\" created\nclusterrolebinding \"prometheus-operator\" created\ndeployment \"prometheus-operator\" created\n\n$ kubectl apply -f demos/monitoring/sample-prometheus-instance.yaml\nclusterrole \"prometheus\" created\nserviceaccount \"prometheus\" created\nclusterrolebinding \"prometheus\" created\nprometheus \"sample-metrics-prom\" created\nservice \"sample-metrics-prom\" created\n\n$ kubectl get svc\nNAME                  CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE\nkubernetes            10.96.0.1       \u003cnone\u003e        443/TCP          30m\nprometheus-operated   None            \u003cnone\u003e        9090/TCP         4m\nsample-metrics-prom   10.108.71.184   \u003cnodes\u003e       9090:30999/TCP   4m\n```\n\n### Deploying a custom metrics API Server and a sample app\n\nIn v1.6, the Horizontal Pod Autoscaler controller can now consume custom metrics for autoscaling.\nFor this to work, one needs to have enabled the `autoscaling/v2alpha1` API group which makes it possible\nto create Horizontal Pod Autoscaler resources of the new version.\n\nAlso, one must have API aggregation enabled (which is the case in this demo) and a extension API Server that\nprovides the `custom-metrics.metrics.k8s.io/v1alpha1` API group/version.\n\nThere won't be an \"official\" one-size-fits all custom metrics API server, instead there will be a boilerplate\npeople can use as the base for creating custom monitoring solutions.\n\nI've built an example custom metrics server that queries a Prometheus instance for metrics data and exposing them\nin the custom metrics Kubernetes API. You can think of this custom metrics server as a shim/conversation layer between\nPrometheus data and the Horizontal Pod Autoscaling API for Kubernetes.\n\nHere is a diagram over how this works on a high level:\n\n![Custom Metrics Architecture](pictures/custom-metrics-architecture.png)\n\nYou can also read the full custom metrics API proposal [here](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/custom-metrics-api.md)\n\n```console\n$ kubectl apply -f demos/monitoring/custom-metrics.yaml\nnamespace \"custom-metrics\" created\nserviceaccount \"custom-metrics-apiserver\" created\nclusterrolebinding \"custom-metrics:system:auth-delegator\" created\nrolebinding \"custom-metrics-auth-reader\" created\nclusterrole \"custom-metrics-read\" created\nclusterrolebinding \"custom-metrics-read\" created\ndeployment \"custom-metrics-apiserver\" created\nservice \"api\" created\napiservice \"v1alpha1.custom-metrics.metrics.k8s.io\" created\nclusterrole \"custom-metrics-server-resources\" created\nclusterrolebinding \"hpa-controller-custom-metrics\" created\n```\n\nIf you want to be able to `curl` the custom metrics API server easily (i.e. allow anyone to access the Custom Metrics API), you can\nrun this `kubectl` command:\n\n```console\n$ kubectl create clusterrolebinding allowall-cm --clusterrole custom-metrics-server-resources --user system:anonymous\nclusterrolebinding \"allowall-cm\" created\n```\n\n```console\n$ kubectl apply -f demos/monitoring/sample-metrics-app.yaml\ndeployment \"sample-metrics-app\" created\nservice \"sample-metrics-app\" created\nservicemonitor \"sample-metrics-app\" created\nhorizontalpodautoscaler \"sample-metrics-app-hpa\" created\ningress \"sample-metrics-app\" created\n```\n\nNow that we have our sample app, we should generate some load against it!\nIf you don't have [rakyll's](https://github.com/rakyll) excellent [hey](https://github.com/rakyll/hey) load generator already, you can install it this way:\n\n```console\n$ # Install hey\n$ docker run -it -v /usr/local/bin:/go/bin golang:1.8 go get github.com/rakyll/hey\n\n$ export APP_ENDPOINT=$(kubectl get svc sample-metrics-app -o template --template {{.spec.clusterIP}}); echo ${APP_ENDPOINT}\n$ hey -n 50000 -c 1000 http://${APP_ENDPOINT}\n```\n\nThen you can go and check out the Custom Metrics API, it should notice that a lot of requests have been served recently.\n\n```console\n$ curl -sSLk https://10.96.0.1/apis/custom-metrics.metrics.k8s.io/v1alpha1/namespaces/default/services/sample-metrics-app/http_requests\n{\n  \"kind\": \"MetricValueList\",\n  \"apiVersion\": \"custom-metrics.metrics.k8s.io/v1alpha1\",\n  \"metadata\": {\n    \"selfLink\": \"/apis/custom-metrics.metrics.k8s.io/v1alpha1/namespaces/default/services/sample-metrics-app/http_requests\"\n  },\n  \"items\": [\n    {\n      \"describedObject\": {\n        \"kind\": \"Service\",\n        \"name\": \"sample-metrics-app\",\n        \"apiVersion\": \"/__internal\"\n      },\n      \"metricName\": \"http_requests\",\n      \"timestamp\": \"2017-06-30T20:56:34Z\",\n      \"value\": \"501484m\"\n    }\n  ]\n}\n```\n\nYou can query custom metrics for individual Pods as well: \n\n```console\n$ kubectl get po\nNAME                                  READY     STATUS    RESTARTS   AGE\nprometheus-operator-815607840-zknhk   1/1       Running   0          38m\nprometheus-sample-metrics-prom-0      2/2       Running   0          33m\nrook-operator-3393217773-sglsv        1/1       Running   0          28m\nsample-metrics-app-3083280453-3hbd8   1/1       Running   0          33m\nsample-metrics-app-3083280453-fbds8   1/1       Running   0          1m\n\n\n$ curl -sSLk https://10.96.0.1/apis/custom-metrics.metrics.k8s.io/v1alpha1/namespaces/default/pods/sample-metrics-app-3083280453-3hbd8/http_requests\n{\n  \"kind\": \"MetricValueList\",\n  \"apiVersion\": \"custom-metrics.metrics.k8s.io/v1alpha1\",\n  \"metadata\": {\n    \"selfLink\": \"/apis/custom-metrics.metrics.k8s.io/v1alpha1/namespaces/default/pods/sample-metrics-app-3083280453-3hbd8/http_requests\"\n  },\n  \"items\": [\n    {\n      \"describedObject\": {\n        \"kind\": \"Pod\",\n        \"name\": \"sample-metrics-app-3083280453-3hbd8\",\n        \"apiVersion\": \"/__internal\"\n      },\n      \"metricName\": \"http_requests\",\n      \"timestamp\": \"2017-06-30T21:00:46Z\",\n      \"value\": \"433m\"\n    }\n  ]\n}\n```\n\n#### Install `helm`\n\n[Helm](https://github.com/kubernetes/helm) is a package manager for applications running on top of Kubernetes.\nYou can read more about Helm in the official repository, for now we're just gonna install it.\n\nBelow you'll see the famous `curl | bash` pattern for installing an application, and yes, I know it's discouraged.\nBut I'm doing it this way here to keep the tutorial short and concise, hardening the helm installation is left as an excercise to the user.\n\nThen we're running `helm init` that will deploy its server side component and set up local cache at `~/.helm`. Make sure the `KUBECONFIG`\nenvironment variable is set to point to the kubeconfig file for your kubeadm cluster.\n\n```console\n$ curl -sSL https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash\n$ helm init\n```\n\n`tiller` is the server-side component of the Helm ecosystem, and handles installation, upgrades and more.\nBy default in version v2.3.x, `helm init` installs tiller without any RBAC privileges. This means tiller won't be able to install any apps unless\nwe give it some RBAC permissions. However, knowing what `tiller` is gonna install for the user in beforehand is very hard, so the only way to make it\nwork in all cases is to give it full privileges (root access) to the cluster. We're doing this by binding the `tiller` ServiceAccount here to the \nvery powerful `cluster-admin` ClusterRole.\n\n```console\n$ kubectl -n kube-system create serviceaccount tiller\n$ kubectl -n kube-system patch deploy tiller-deploy -p '{\"spec\":{\"template\":{\"spec\":{\"serviceAccountName\":\"tiller\"}}}}'\n$ kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount kube-system:tiller\n$ kubectl -n kube-system set image deploy/tiller-deploy tiller=luxas/tiller:v2.5.1\n```\n\n#### Deploying the Service Catalog\n\nThe [Service Catalog](https://github.com/kubernetes-incubator/service-catalog) Kubernetes project is super-interesting and promising.\n\nIf you're interested in the concept, watch these two KubeCon talks:\n - Steward, the Kubernetes-Native Service Broker [A] - Gabe Monroy, Deis: [Youtube video](https://youtu.be/PNPVDKrbgsE?list=PLj6h78yzYM2PAavlbv0iZkod4IVh_iGqV)\n - The Open Service Broker API and the Kubernetes Service Catalog [B] - Paul Morie \u0026 Chip Childers: [Youtube video](https://youtu.be/p35hOAAsxrQ?list=PLj6h78yzYM2PAavlbv0iZkod4IVh_iGqV)\n\nAnyway, here's how to install the Service Catalog on your `kubeadm` cluster:\n\n```console\n$ git clone https://github.com/luxas/service-catalog -b workshop\n$ # First install the Service Catalog API Server and Controller Manager and then a sample Broker\n$ helm install service-catalog/charts/catalog --name catalog --namespace catalog\n$ helm install service-catalog/charts/ups-broker --name ups-broker --namespace ups-broker\n```\n\nI highly recommend this [Service Catalog Walkthough](https://github.com/kubernetes-incubator/service-catalog/blob/master/docs/walkthrough.md).\n\nTL;DR; Now that our Service Catalog API Server is there, we can `kubectl get` the resources:\n\n```console\n$ kubectl get instances,bindings,serviceclasses,brokers\n...\n```\n\nYou can for example make an Instance and a Binding to the sample `ups-broker` you installed above like this:\n\n```console\n$ kubectl apply -f demos/service-catalog/example.yaml\nnamespace \"test-ns\" created\ninstance \"ups-instance\" created\nbrinding \"ups-binding\" created\n\n$ # Since the binding referenced a new Secret called \"my-secret\", the Service Catalog should now have created it for you:\n$ kubectl -n test-ns get secret my-secret\nTODO\n```\n\n### Manifest list images\n\nAll the source for building the images used in this demo is available under `images/`.\n\nYou simply need to cd into the directory and run `REGISTRY=foo make push`, setting the `REGISTRY`\nvariable to your Docker Hub account for example, where you have push rights.\n\nAll pushed images follow the pattern `REGISTRY/IMAGE-ARCH:VERSION` plus a manifest list of the form\n`REGISTRY/IMAGE:VERSION` that references to the architecture-specific images.\n\nCurrently, images are pushed for `amd64`, `arm` and `arm64`.\n\n### Acknowledgements / More reference\n\nI'd like to thank some people that have been very helpful to me while putting together this workshop.\n\n**David Eads** ([@deads2k](https://github.com/deads2k)) has been very helpful to me and answered my questions about API aggregation, RBAC, etc..\n\n**Solly Ross** ([@DirectXMan12](https://github.com/DirectXMan12)) has worked on the custom metrics API and helped me quickly understand\nthe essential parts of it. He also uploaded a [custom metrics API Server boilerplate](https://github.com/DirectXMan12/custom-metrics-boilerplate)\nwhich I've used as the base for my custom metrics implementation.\n\nAlso, these I want to thank the maintainers of the great projects below. Let's be grateful for all the\nreally nice projects that are open sourced on Github.\n\n**Prometheus Operator by CoreOS**: The Prometheus is an integral part of the custom metrics service in\nthis workshop, it made it super-easy to create managed Prometheus instances with the TPR!\n\n**Prometheus by CNCF**: Some projects are just rock-solid. The Prometheus core is such a project.\nMonitoring made available for everyone, simply.\n\n**Rook by Quantum**: Rook is a very interesting and promising project and I'm excited to see how this\nproject can be brought into something stable and reliable in the future.\n\n**Traefik by Containous**: Traefik is a powerful loadbalancer, and I love the Kubernetes integration it has.\nAlso, with the Prometheus exporter integration in v1.2, it got even cooler.\n\n**Weave by Weaveworks**: Weave is a distributed networking system that plays very well with Kubernetes, it also\nis CNI-compliant, which is a good thing.\n\n\n### Future work / contributing\n\nThis workshop uses my own custom-built images under the `luxas` Docker Hub user.\nThis is only a temporary solution while I carry patches I had to make in order to get it working,\nI will work to upstream these changes eventually though.\n\nFeel free to contribute and help me improve things here and I'd be very thankful ;)\n\nI use the Github tracker for tracking the improvements I want to make to this repository\n\n### License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluxas%2Fkubeadm-workshop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluxas%2Fkubeadm-workshop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluxas%2Fkubeadm-workshop/lists"}