{"id":13452268,"url":"https://github.com/hobby-kube/guide","last_synced_at":"2025-05-14T13:07:15.335Z","repository":{"id":39352035,"uuid":"88660234","full_name":"hobby-kube/guide","owner":"hobby-kube","description":" Kubernetes clusters for the hobbyist.","archived":false,"fork":false,"pushed_at":"2023-09-21T14:37:01.000Z","size":142,"stargazers_count":5628,"open_issues_count":9,"forks_count":261,"subscribers_count":110,"default_branch":"master","last_synced_at":"2025-04-11T06:08:10.302Z","etag":null,"topics":["automation","cluster","cost-effective","devops","digitalocean","guide","hetzner-cloud","kubernetes","scaleway","security","setup","small-scale","terraform"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hobby-kube.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null},"funding":{"github":"pstadler"}},"created_at":"2017-04-18T19:00:09.000Z","updated_at":"2025-04-10T18:08:31.000Z","dependencies_parsed_at":"2023-09-29T08:52:02.577Z","dependency_job_id":null,"html_url":"https://github.com/hobby-kube/guide","commit_stats":{"total_commits":68,"total_committers":13,"mean_commits":5.230769230769231,"dds":"0.17647058823529416","last_synced_commit":"e6c66616a8e081446ea1a74db2a599b8b6072dcd"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hobby-kube%2Fguide","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hobby-kube%2Fguide/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hobby-kube%2Fguide/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hobby-kube%2Fguide/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hobby-kube","download_url":"https://codeload.github.com/hobby-kube/guide/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254149958,"owners_count":22022851,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","cluster","cost-effective","devops","digitalocean","guide","hetzner-cloud","kubernetes","scaleway","security","setup","small-scale","terraform"],"created_at":"2024-07-31T07:01:18.936Z","updated_at":"2025-05-14T13:07:10.320Z","avatar_url":"https://github.com/hobby-kube.png","language":null,"readme":"# Kubernetes clusters for the hobbyist\n\nThis guide answers the question of how to setup and operate a fully functional, secure Kubernetes cluster on a cloud provider such as Hetzner Cloud, DigitalOcean or Scaleway. It explains how to overcome the lack of external ingress controllers, fully isolated secure private networking and persistent distributed block storage.\n\nBe aware, that the following sections might be opinionated. Kubernetes is an evolving, fast paced environment, which means this guide will probably be outdated at times, depending on the author's spare time and individual contributions. Due to this fact contributions are highly appreciated.\n\nThis guide is accompanied by a fully automated cluster setup solution in the shape of well structured, modular [Terraform](https://www.terraform.io/) recipes. Links to contextually related modules are spread throughout the guide, visually highlighted using the ![Terraform](assets/terraform.png) Terraform icon.\n\nIf you find this project helpful, please consider supporting its future development on [GitHub Sponsors](https://github.com/users/pstadler/sponsorship).\n\n## Table of Contents\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n\n- [Cluster size](#cluster-size)\n- [Choosing a cloud provider](#choosing-a-cloud-provider)\n- [Choosing an operating system](#choosing-an-operating-system)\n- [Security](#security)\n  - [Firewall](#firewall)\n  - [Secure private networking](#secure-private-networking)\n  - [WireGuard setup](#wireguard-setup)\n- [Installing Kubernetes](#installing-kubernetes)\n  - [Containerd setup](#containerd-setup)\n  - [Etcd setup](#etcd-setup)\n  - [Kubernetes setup](#kubernetes-setup)\n    - [Initializing the master node](#initializing-the-master-node)\n    - [Joining the cluster nodes](#joining-the-cluster-nodes)\n- [Access and operations](#access-and-operations)\n  - [Role-Based Access Control](#role-based-access-control)\n  - [Deploying services](#deploying-services)\n- [Bringing traffic to the cluster](#bringing-traffic-to-the-cluster)\n  - [Ingress controller setup](#ingress-controller-setup)\n  - [DNS records](#dns-records)\n  - [Obtaining SSL/TLS certificates](#obtaining-ssltls-certificates)\n  - [Deploying the Kubernetes Dashboard](#deploying-the-kubernetes-dashboard)\n- [Distributed block storage](#distributed-block-storage)\n  - [Persistent volumes](#persistent-volumes)\n  - [Choosing a solution](#choosing-a-solution)\n  - [Deploying Rook](#deploying-rook)\n  - [Consuming storage](#consuming-storage)\n- [Where to go from here](#where-to-go-from-here)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n## Cluster size\n\nThe professional hobbyist cluster operators aim for resilience—a system's ability to withstand and recover from failure. On the other hand, they usually have a limited amount of funds they can or want to spend on a basic cluster. It's therefore crucial to find a good balance between resilience and cost.\n\nAfter experimenting with various setups and configurations a good reference point is, that a basic cluster can be operated on as little as two virtual hosts with 1GB memory each. At this point it's worth mentioning that Kubernetes does not include *swap memory* in its calculations and will evict pods pretty brutally when reaching memory limits ([reference](https://github.com/kubernetes/kubernetes/issues/7294)). As opposed to memory, raw CPU power doesn't matter that much, although it should be clear that the next Facebook won't be running on two virtual CPU cores.\n\nFor a Kubernetes cluster to be resilient it's recommended that it consists of **at least three hosts**. The main reason behind this is that *etcd*, which itself is an essential part of any Kubernetes setup, is only fault tolerant with a minimum of three cluster members ([reference](https://coreos.com/etcd/docs/latest/v2/admin_guide.html#optimal-cluster-size)).\n\n## Choosing a cloud provider\n\n![Terraform](assets/terraform.png) [`provider/hcloud`](https://github.com/hobby-kube/provisioning/tree/master/provider/hcloud)\n![Terraform](assets/terraform.png) [`provider/digitalocean`](https://github.com/hobby-kube/provisioning/tree/master/provider/digitalocean)\n![Terraform](assets/terraform.png) [`provider/scaleway`](https://github.com/hobby-kube/provisioning/tree/master/provider/scaleway)\n\nAt this point it's time to choose a cloud provider based on a few criteria such as trustworthiness, reliability, pricing and data center location. The very best offer at this time is definitely from [Hetzner Cloud](https://hetzner.cloud/?ref=osR7dA9R4bmz) (referral link, get €20), where one gets a suitable three node cluster up and running for **around €13.50/month** (3x2GB, respectively 3x4GB with arm64 CPUs).\n\n[DigitalOcean](https://m.do.co/c/8bd7e234cf6c) (referral link, get $100) is known for their great support and having data centers around the globe which is definitely a plus. A three node cluster will cost $18/month (3x1GB).\n\n[Scaleway](https://www.scaleway.com/)'s instances start at around €5. A three node cluster will cost around €15/month (3x1GB, with 20GB disk space each).\n\n[Linode](https://www.linode.com/), [Vultr](https://www.vultr.com/) and a couple of other providers with similar offers are other viable options. While they all have their advantages and disadvantages, they should be perfectly fine for hosting a Kubernetes cluster.\n\nWhile pricing for virtual private servers has generally increased in the past years, the rise of arm64 based CPUs has opened doors for less expensive options. This guide results in a setup that can be operated on x86, as well as arm64 based systems.\n\n## Choosing an operating system\n\nWhile Linux comes in many flavors, **Ubuntu** (LTS) is the distribution of choice for hosting our cluster. This may seem opinionated—and it is—but then again, Ubuntu has always been a first class citizen in the Kubernetes ecosystem.\n\nCoreOS would be a great option as well, because of how it embraces the use of containers. On the other hand, not everything we might need in the future is readily available. Some essential packages are likely to be missing at this point, or at least there's no support for running them outside of containers.\n\nThat being said, feel free to use any Linux distribution you like. Just be aware that some of the sections in this guide may differ substantially depending on your chosen operating system.\n\n## Security\n\n\u003e Securing hosts on both public and private interfaces is an absolute necessity.\n\nThis is a tough one. Almost every single guide fails to bring the security topic to the table to the extent it deserves. **One of the biggest misconceptions is that private networks are secure**, but private does not mean secure. In fact, private networks are more often than not shared between many customers in the same data center. This might not be the case with all providers. It's generally good advise to gain absolute certainty, what the actual conditions of a *private* network are.\n\n### Firewall\n\n![Terraform](assets/terraform.png) [`security/ufw`](https://github.com/hobby-kube/provisioning/tree/master/security/ufw)\n\nWhile there are definitely some people out there able to configure *iptables* reliably, the average mortal will cringe when glancing at the syntax of the most basic rules. Luckily, there are more approachable solutions out there. One of those is [UFW](https://help.ubuntu.com/community/UFW), *the uncomplicated firewall*—a human friendly command line interface offering simple abstractions for managing complex *iptables* rules.\n\nAssuming the secure public Kubernetes API runs on port 6443, SSH daemon on 22, plus 80 and 443 for serving web traffic, results in the following basic UFW configuration:\n\n```sh\nufw allow ssh # sshd on port 22, be careful to not get locked out!\nufw allow 6443 # remote, secure Kubernetes API access\nufw allow 80\nufw allow 443\nufw default deny incoming # deny traffic on every other port, on any interface\nufw enable\n```\n\nThis ruleset will get slightly expanded in the upcoming sections.\n\n### Secure private networking\n\nKubernetes cluster members constantly exchange data with each other. A secure network overlay between hosts is not only the simplest, but also the most secure solution for making sure that a third party occupying the same network as our hosts won't be able to eavesdrop on their private traffic. It's a tedious job to secure every single service, as this task usually requires creating and distributing certificates across hosts, managing secrets in one way or another and, last but not least, configuring services to actually use encrypted means of communication. That's why setting up a network overlay using VPN—which itself is a one-time effort requiring very little know how, and which naturally ensures secure inter-host communication for every possible service running now and in the future—is simply the best solution to address this problem.\n\nWhen talking about VPN, there are generally two types of solutions:\n\n- Traditional **VPN** services, running in userland, typically providing a tunnel interface\n- **IPsec**, which is part of the Kernel and enables authentication and encryption on any existing interface\n\nVPN software running in userland has in general a huge negative impact on network throughput as opposed to IPsec, which is much faster. Unfortunately, it's quite a challenge to understand how the latter works. [strongSwan](https://www.strongswan.org/) is certainly one of the more approachable solutions, but setting it up for even the most basic needs is still accompanied by a steep learning curve.\n\n\u003e Complexity is security's worst contender.\n\nA project called [WireGuard](https://www.WireGuard.io/) supplies the best of both worlds at this point. Running as a Kernel module, it not only offers excellent performance, but is dead simple to set up and provides a tunnel interface out of the box. It may be disputed whether running VPN within the Kernel is a good idea, but then again alternatives running in userland such as [tinc](https://www.tinc-vpn.org/) or [fastd](https://projects.universe-factory.net/projects/fastd/wiki) aren't necessarily more secure. However, they are an order of magnitude slower and typically harder to configure.\n\n### WireGuard setup\n\n![Terraform](assets/terraform.png) [`security/wireguard`](https://github.com/hobby-kube/provisioning/tree/master/security/wireguard)\n\nLet's start off by installing WireGuard. Follow the instructions found here: [WireGuard Installation](https://www.wireguard.com/install/).\n\n```sh\napt install wireguard\n```\n\nOnce WireGuard has been installed, it's time to create the configuration files. Each host should connect to its peers to create a secure network overlay via a tunnel interface called wg0. Let's assume the setup consists of three hosts and each one will get a new VPN IP address in the 10.0.1.1/24 range:\n\n| Host  | Private IP address  (ethN) | VPN IP address (wg0) |\n| ----- | -------------------------- | -------------------- |\n| kube1 | 10.8.23.93                 | 10.0.1.1             |\n| kube2 | 10.8.23.94                 | 10.0.1.2             |\n| kube3 | 10.8.23.95                 | 10.0.1.3             |\n\nPlease note that Hetzner Cloud doesn't provide a private network interface, but it's perfectly fine to run WireGuard on the public interface. Just make sure to use the public IP addresses and the public network interface (eth0).\n\nIn this scenario, a configuration file for kube1 would look like this:\n\n```sh\n# /etc/wireguard/wg0.conf\n[Interface]\nAddress = 10.0.1.1\nPrivateKey = \u003cPRIVATE_KEY_KUBE1\u003e\nListenPort = 51820\n\n[Peer]\nPublicKey = \u003cPUBLIC_KEY_KUBE2\u003e\nAllowedIps = 10.0.1.2/32\nEndpoint = 10.8.23.94:51820\n\n[Peer]\nPublicKey = \u003cPUBLIC_KEY_KUBE3\u003e\nAllowedIps = 10.0.1.3/32\nEndpoint = 10.8.23.95:51820\n```\n\nTo simplify the creation of private and public keys, the following command can be used to generate and print the necessary key-pairs:\n\n```sh\nfor i in 1 2 3; do\n  private_key=$(wg genkey)\n  public_key=$(echo $private_key | wg pubkey)\n  echo \"Host $i private key: $private_key\"\n  echo \"Host $i public key:  $public_key\"\ndone\n```\n\nAfter creating a file named `/etc/wireguard/wg0.conf` on each host containing the correct IP addresses and public and private keys, configuration is basically done.\n\nWhat's left is to add the following firewall rules:\n\n```sh\n# open VPN port on private network interface (use eth0 on Hetzner Cloud)\nufw allow in on eth1 to any port 51820\n# allow all traffic on VPN tunnel interface\nufw allow in on wg0\nufw reload\n```\n\nBefore starting WireGuard we need to make sure that ip forwarding and a few other required network settings are enabled:\n\n```sh\necho br_netfilter \u003e /etc/modules-load.d/kubernetes.conf\nmodprobe br_netfilter\n\necho \"net.ipv4.ip_forward=1\" \u003e\u003e /etc/sysctl.conf\necho \"net.bridge.bridge-nf-call-iptables=1\" \u003e\u003e /etc/sysctl.conf\n\nsysctl -p # apply settings from /etc/sysctl.conf\n```\n\nExecuting the command `systemctl start wg-quick@wg0` on each host will start the VPN service and, if everything is configured correctly, the hosts should be able to establish connections between each other. Traffic can now be routed securely using the VPN IP addresses (10.0.1.1–10.0.1.3).\n\nIn order to check whether the connections are established successfully, `wg show` comes in handy:\n\n```sh\n$ wg show\ninterface: wg0\n  public key: 5xKk9...\n  private key: (hidden)\n  listening port: 51820\n\npeer: HBCwy...\n  endpoint: 10.8.23.199:51820\n  allowed ips: 10.0.1.1/32\n  latest handshake: 25 seconds ago\n  transfer: 8.76 GiB received, 25.46 GiB sent\n\npeer: KaRMh...\n  endpoint: 10.8.47.93:51820\n  allowed ips: 10.0.1.3/32\n  latest handshake: 59 seconds ago\n  transfer: 41.86 GiB received, 25.09 GiB sent\n```\n\nLast but not least, run `systemctl enable wg-quick@wg0` to launch the service whenever the system boots.\n\n## Installing Kubernetes\n\n![Terraform](assets/terraform.png) [`service/kubernetes`](https://github.com/hobby-kube/provisioning/tree/master/service/kubernetes)\n\nThere are plenty of ways to set up a Kubernetes cluster from scratch. At this point however, we settle on [kubeadm](https://kubernetes.io/docs/getting-started-guides/kubeadm/). This dramatically simplifies the setup process by automating the creation of certificates, services and configuration files.\n\nBefore getting started with Kubernetes itself, we need to take care of setting up two essential services that are not part of the actual stack, namely **containerd** and **etcd**. We've been using Docker in the past, but containerd is now preferred.\n\n### Containerd setup\n\n[containerd](https://containerd.io/) is robust container runtime. Hints regarding supported versions are available in [the official container runtimes guide](https://kubernetes.io/docs/setup/production-environment/container-runtimes/). Let's install a supported containerd version:\n\n```sh\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmour -o /etc/apt/keyrings/docker.gpg\necho \"deb [signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable\" \\\n  \u003e /etc/apt/sources.list.d/docker.list\n\napt-get install -y containerd.io=1.6.15-1 # Kubernetes 1.26 requires at least containerd v1.6\n```\n\nKubernetes recommends running containerd with the cgroup driver. This can be done by creating a containerd config file and setting the required configuration flag:\n\n```sh\n# write default containerd config\ncontainerd config default \u003e /etc/containerd/config.toml\n# set systemd cgroup flag to true\nsed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml\n\n# enable containerd and restart\nsystemctl enable containerd\nsystemctl restart containerd\n```\n\n### Etcd setup\n\n![Terraform](assets/terraform.png) [`service/etcd`](https://github.com/hobby-kube/provisioning/tree/master/service/etcd)\n\n[etcd](https://coreos.com/etcd/docs/latest/) is a highly-available key value store, which Kubernetes uses for persistent storage of all of its REST API objects. It is therefore a crucial part of the cluster. kubeadm would normally install etcd on a single node. Depending on the number of hosts available, it would be rather stupid not to run etcd in cluster mode. As mentioned earlier, it makes sense to run at least a three node cluster due to the fact that etcd is fault tolerant only from this size on.\n\nEven though etcd is generally available with most package managers, it's recommended to manually install a more recent version:\n\n```sh\nexport ETCD_VERSION=\"v3.5.6\"\nmkdir -p /opt/etcd\ncurl -L https://storage.googleapis.com/etcd/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz \\\n  -o /opt/etcd-${ETCD_VERSION}-linux-amd64.tar.gz\ntar xzvf /opt/etcd-${ETCD_VERSION}-linux-amd64.tar.gz -C /opt/etcd --strip-components=1\n```\n\nIn an insecure environment configuring etcd typically involves creating and distributing certificates across nodes, whereas running it within a secure network makes this process a whole lot easier. There's simply no need to make use of additional security layers as long as the service is bound to an end-to-end secured VPN interface.\n\nThis section is not going to explain etcd configuration in depth, refer to the [official documentation](https://coreos.com/etcd/docs/latest/getting-started-with-etcd.html) instead. All that needs to be done is creating a systemd unit file on each host. Assuming a three node cluster, the configuration for kube1 would look like this:\n\n```sh\n# /etc/systemd/system/etcd.service\n[Unit]\nDescription=etcd\nAfter=network.target wg-quick@wg0.service\n\n[Service]\nType=notify\nExecStart=/opt/etcd/etcd --name kube1 \\\n  --data-dir /var/lib/etcd \\\n  --listen-client-urls \"http://10.0.1.1:2379,http://localhost:2379\" \\\n  --advertise-client-urls \"http://10.0.1.1:2379\" \\\n  --listen-peer-urls \"http://10.0.1.1:2380\" \\\n  --initial-cluster \"kube1=http://10.0.1.1:2380,kube2=http://10.0.1.2:2380,kube3=http://10.0.1.3:2380\" \\\n  --initial-advertise-peer-urls \"http://10.0.1.1:2380\" \\\n  --heartbeat-interval 200 \\\n  --election-timeout 5000\nRestart=always\nRestartSec=5\nTimeoutStartSec=0\nStartLimitInterval=0\n\n[Install]\nWantedBy=multi-user.target\n```\n\nIt's important to understand that each flag starting with `--initial` does only apply during the first launch of a cluster. This means for example, that it's possible to add and remove cluster members at any time without ever changing the value of `--initial-cluster`.\n\nAfter the files have been placed on each host, it's time to start the etcd cluster:\n\n```sh\nsystemctl enable etcd.service # launch etcd during system boot\nsystemctl start etcd.service\n```\n\nExecuting `/opt/etcd/etcdctl member list` should show a list of cluster members. If something went wrong check the logs using  `journalctl -u etcd.service`.\n\n### Kubernetes setup\n\nNow that containerd is configured and etcd is running, it's time to deploy Kubernetes. The first step is to install the required packages on each host:\n\n```sh\n# https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management\n# (`xenial` is correct even for newer Ubuntu versions)\ncurl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg\necho \"deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main\" \\\n  \u003e /etc/apt/sources.list.d/kubernetes.list\n\napt-get update\n\napt-get install -y kubelet=1.26.0-00 kubeadm=1.26.0-00 kubectl=1.26.0-00 # kubernetes-cni package comes as dependency of the others\n# Pin Kubernetes major version since there are breaking changes between releases.\n# For example, Kubernetes 1.26 requires a newer containerd (https://kubernetes.io/blog/2022/11/18/upcoming-changes-in-kubernetes-1-26/#cri-api-removal).\napt-mark hold kubelet kubeadm kubectl kubernetes-cni\n```\n\n#### Initializing the master node\n\nBefore initializing the master node, we need to create a manifest on kube1 which will then be used as configuration in the next step:\n\n```yaml\n# /tmp/master-configuration.yml\napiVersion: kubeadm.k8s.io/v1beta3\nkind: InitConfiguration\nlocalAPIEndpoint:\n  advertiseAddress: 10.0.1.1\n  bindPort: 6443\n---\napiVersion: kubeadm.k8s.io/v1beta3\nkind: ClusterConfiguration\ncertificatesDir: /etc/kubernetes/pki\napiServer:\n  certSANs:\n  - \u003cPUBLIC_IP_KUBE1\u003e\netcd:\n  external:\n    endpoints:\n      - http://10.0.1.1:2379\n      - http://10.0.1.2:2379\n      - http://10.0.1.3:2379\n---\napiVersion: kubelet.config.k8s.io/v1beta1\nkind: KubeletConfiguration\nfailSwapOn: false\ncgroupDriver: systemd\n```\n\nThen we run the following command on kube1:\n\n```sh\nkubeadm init --config /tmp/master-configuration.yml --ignore-preflight-errors=Swap,NumCPU\n```\nAfter the setup is complete, kubeadm prints a token such as `818d5a.8b50eb5477ba4f40`. It's important to write it down, we'll need it in a minute to join the other cluster nodes.\n\nKubernetes is built around openness, so it's up to us to choose and install a suitable pod network. This is required as it enables pods running on different nodes to communicate with each other. One of the [many options](https://kubernetes.io/docs/concepts/cluster-administration/addons/) is [Cilium](https://cilium.io/). It requires little configuration and is considered stable and well-maintained:\n\n```sh\n# create symlink for the current user in order to gain access to the API server with kubectl\n[ -d $HOME/.kube ] || mkdir -p $HOME/.kube\nln -s /etc/kubernetes/admin.conf $HOME/.kube/config\n\n# install Cilium\nCILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)\nCLI_ARCH=\"$(arch | sed 's/x86_64/amd64/; s/aarch64/arm64/')\"\ncurl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}\nsha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum\nsudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin\nrm cilium-linux-${CLI_ARCH}.tar.gz*\n\ncilium install --version 1.14.1 --set ipam.mode=cluster-pool --set ipam.operator.clusterPoolIPv4PodCIDRList=10.96.0.0/16\n\n# allow traffic on the newly created Cilium network interface\nufw allow in on cilium_vxlan\nufw reload\n```\n\nCilium will not readily work with our current cluster configuration because traffic will be routed via the wrong network interface. This can be fixed by running the following command on each host:\n\n```sh\nip route add 10.96.0.0/16 dev $VPN_INTERFACE src $VPN_IP\n\n# on kube1:\nip route add 10.96.0.0/16 dev wg0 src 10.0.1.1\n# on kube2:\nip route add 10.96.0.0/16 dev wg0 src 10.0.1.2\n# on kube3:\nip route add 10.96.0.0/16 dev wg0 src 10.0.1.3\n```\n\nThe added route will not survive a reboot as it is not persistent. To ensure that the route gets added after a reboot, we have to add a *systemd* service unit on each node which will wait for the WireGuard interface to come up and after that adds the route. For kube1 it would look like this:\n\n```sh\n# /etc/systemd/system/overlay-route.service\n[Unit]\nDescription=Overlay network route for WireGuard\nAfter=wg-quick@wg0.service\n\n[Service]\nType=oneshot\nUser=root\nExecStart=/sbin/ip route add 10.96.0.0/16 dev wg0 src 10.0.1.1\n\n[Install]\nWantedBy=multi-user.target\n```\n\nAfter that we have to enable it by running following command:\n\n```sh\nsystemctl enable overlay-route.service\n```\n\nFinally, we can check if everything works:\n\n```sh\ncilium status --wait\n```\n\n#### Joining the cluster nodes\n\nAll that's left is to join the cluster with the other nodes. Run the following command on each host:\n\n```sh\nkubeadm join --token=\"\u003cTOKEN\u003e\" 10.0.1.1:6443 \\\n  --discovery-token-unsafe-skip-ca-verification \\\n  --ignore-preflight-errors=Swap\n```\n\nThat's it, a Kubernetes cluster is ready at our disposal.\n\n## Access and operations\n\n![Terraform](assets/terraform.png) [`service/kubernetes`](https://github.com/hobby-kube/provisioning/tree/master/service/kubernetes)\n\nAs soon as the cluster is running, we want to be able to access the Kubernetes API remotely. This can be done by copying `/etc/kubernetes/admin.conf` from kube1 to your own machine. After [installing kubectl](https://kubernetes.io/docs/tasks/kubectl/install/) locally, execute the following commands:\n\n```sh\n# create local config folder\nmkdir -p ~/.kube\n# backup old config if required\n[ -f ~/.kube/config ] \u0026\u0026 cp ~/.kube/config ~/.kube/config.backup\n# copy config from master node\nscp root@\u003cPUBLIC_IP_KUBE1\u003e:/etc/kubernetes/admin.conf ~/.kube/config\n# change config to use correct IP address\nkubectl config set-cluster kubernetes --server=https://\u003cPUBLIC_IP_KUBE1\u003e:6443\n```\n\nYou're now able to remotely access the Kubernetes API. Running `kubectl get nodes` should show a list of nodes similar to this:\n\n```sh\nNAME    STATUS   ROLES                  AGE     VERSION\nkube1   Ready    control-plane,master   5h11m   v1.21.1\nkube2   Ready    \u003cnone\u003e                 5h11m   v1.21.1\nkube3   Ready    \u003cnone\u003e                 5h11m   v1.21.1\n```\n\n### Role-Based Access Control\n\nAs of version 1.6, kubeadm configures Kubernetes with RBAC enabled. Because our hobby cluster is typically operated by trusted people, we should enable permissive RBAC permissions to be able to deploy any kind of services using any kind of resources. If you're in doubt whether this is secure enough for your use case, please refer to the official [RBAC documentation](https://kubernetes.io/docs/admin/authorization/rbac).\n\n```sh\nkubectl create clusterrolebinding permissive-binding \\\n  --clusterrole=cluster-admin \\\n  --user=admin \\\n  --user=kubelet \\\n  --group=system:serviceaccounts\n```\n\n### Deploying services\n\nServices can now be deployed remotely by calling `kubectl -f apply \u003cFILE\u003e`. It's also possible to apply multiple files by pointing to a folder, for example:\n\n```sh\n$ ls dashboard/\ndeployment.yml  service.yml\n\n$ kubectl apply -f dashboard/\ndeployment \"kubernetes-dashboard\" created\nservice \"kubernetes-dashboard\" created\n```\n\nThis guide will make no further explanations in this regard. Please refer to the official documentation on [kubernetes.io](https://kubernetes.io/).\n\n## Bringing traffic to the cluster\n\nThere are downsides to running Kubernetes outside of well integrated platforms such as AWS or GCE. One of those is the lack of external ingress and load balancing solutions. Fortunately, it's fairly easy to get an NGINX powered ingress controller running inside the cluster, which will enable services to register for receiving public traffic.\n\n### Ingress controller setup\n\nBecause there's no load balancer available with most cloud providers, we have to make sure the NGINX server is always running on the same host, accessible via an IP address that doesn't change. As our master node is pretty much idle at this point, and no ordinary pods will get scheduled on it, we make kube1 our dedicated host for routing public traffic.\n\nWe already opened port 80 and 443 during the initial firewall configuration, now all we have to do is to write a couple of manifests to deploy the NGINX ingress controller on kube1:\n\n- [ingress/00-namespace.yml](https://github.com/hobby-kube/manifests/blob/master/ingress/00-namespace.yml)\n- [ingress/deployment.yml](https://github.com/hobby-kube/manifests/blob/master/ingress/deployment.yml)\n- [ingress/configmap.yml](https://github.com/hobby-kube/manifests/blob/master/ingress/configmap.yml)\n\nOne part requires special attention. In order to make sure NGINX runs on kube1—which is a tainted control-plane node and no pods will normally be scheduled on it—we need to specify a toleration:\n\n```yaml\n# from ingress/deployment.yml\ntolerations:\n- key: node-role.kubernetes.io/control-plane\n  operator: Equal\n  effect: NoSchedule\n```\n\nSpecifying a toleration doesn't make sure that a pod is getting scheduled on any specific node. For this we need to add a node affinity rule. As we have just a single control-plane node, the following specification is enough to schedule a pod on kube1:\n\n```yaml\n# from ingress/deployment.yml\naffinity:\n  nodeAffinity:\n    requiredDuringSchedulingIgnoredDuringExecution:\n      nodeSelectorTerms:\n      - matchExpressions:\n        - key: node-role.kubernetes.io/control-plane\n          operator: Exists\n```\n\nRunning `kubectl apply -f ingress/` will apply all manifests in this folder. First, a namespace called *ingress* is created, followed by the NGINX deployment, plus a default backend to serve 404 pages for undefined domains and routes including the necessary service object. There's no need to define a service object for NGINX itself, because we configure it to use the host network (`hostNetwork: true`), which means that the container is bound to the actual ports on the host, not to some virtual interface within the pod overlay network.\n\nServices are now able to make use of the ingress controller and receive public traffic with a simple manifest:\n\n```yaml\napiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  name: example-ingress\nspec:\n  ingressClassName: \"nginx\"\n  rules:\n  - host: service.example.com\n    http:\n      paths:\n      - path: /\n        backend:\n          serviceName: example-service\n          servicePort: http\n```\n\nThe NGINX ingress controller is quite flexible and supports a whole bunch of [configuration options](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/).\n\n### DNS records\n\n![Terraform](assets/terraform.png) [`dns/cloudflare`](https://github.com/hobby-kube/provisioning/tree/master/dns/cloudflare)\n![Terraform](assets/terraform.png) [`dns/google`](https://github.com/hobby-kube/provisioning/tree/master/dns/google)\n![Terraform](assets/terraform.png) [`dns/aws`](https://github.com/hobby-kube/provisioning/tree/master/dns/aws)\n![Terraform](assets/terraform.png) [`dns/digitalocean`](https://github.com/hobby-kube/provisioning/tree/master/dns/digitalocean)\n\nAt this point we could use a domain name and put some DNS entries into place. To serve web traffic it's enough to create an A record pointing to the public IP address of kube1 plus a wildcard entry to be able to use subdomains:\n\n| Type  | Name          | Value             |\n| ----- | ------------- | ----------------- |\n| A     | example.com   | \u003cPUBLIC_IP_KUBE1\u003e |\n| CNAME | *.example.com | example.com       |\n\nOnce the DNS entries are propagated our example service would be accessible at `http://service.example.com`. If you don't have a domain name at hand, you can always add an entry to your hosts file instead.\n\nAdditionally, it might be a good idea to assign a subdomain to each host, e.g. kube1.example.com. It's way more comfortable to ssh into a host using a domain name instead of an IP address.\n\n### Obtaining SSL/TLS certificates\n\nThanks to [Let’s Encrypt](https://letsencrypt.org/) and a project called [cert-manager](https://github.com/jetstack/cert-manager) it's incredibly easy to obtain free certificates for any domain name pointing at our Kubernetes cluster. Setting this service up takes no time and it plays well with the NGINX ingress controller we deployed earlier. These are the related manifests:\n\n- [ingress/tls/00-cert-manager.yml](https://github.com/hobby-kube/manifests/blob/master/ingress/tls/00-cert-manager.yml)\n- [ingress/tls/cert-issuer.yml](https://github.com/hobby-kube/manifests/blob/master/ingress/tls/cert-issuer.yml)\n\nBefore deploying cert-manager using the manifests above, make sure to replace the email address in `ingress/tls/cert-issuer.yml` with your own.\n\nTo enable certificates for a service, the ingress manifest needs to be slightly extended:\n\n```yaml\napiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  name: example-ingress\n  annotations:\n    cert-manager.io/cluster-issuer: \"letsencrypt\" # enable certificates\nspec:\n  ingressClassName: \"nginx\"\n  tls: # specify domains to fetch certificates for\n  - hosts:\n    - service.example.com\n    secretName: example-service-tls\n  rules:\n  - host: service.example.com\n    http:\n      paths:\n      - path: /\n        backend:\n          serviceName: example-service\n          servicePort: http\n```\n\nAfter applying this manifest, cert-manager will try to obtain a certificate for service.example.com and reload the NGINX configuration to enable TLS. Make sure to check the logs of the cert-manager pod if something goes wrong.\n\nNGINX will automatically redirect clients to HTTPS whenever TLS is enabled. In case you still want to serve traffic on HTTP, add `nginx.ingress.kubernetes.io/ssl-redirect: \"false\"`  to the list of annotations.\n\n### Deploying the Kubernetes Dashboard\n\nNow that everything is in place, we are able to expose services on specific domains and automatically obtain certificates for them. Let's try this out by deploying the [Kubernetes Dashboard](https://github.com/kubernetes/dashboard) with the following manifests:\n\n- [dashboard/deployment.yml](https://github.com/hobby-kube/manifests/blob/master/dashboard/deployment.yml)\n- [dashboard/service.yml](https://github.com/hobby-kube/manifests/blob/master/dashboard/service.yml)\n- [dashboard/ingress.yml](https://github.com/hobby-kube/manifests/blob/master/dashboard/ingress.yml)\n- [dashboard/secret.yml](https://github.com/hobby-kube/manifests/blob/master/dashboard/secret.yml)\n\nOptionally, the following manifests can be used to get resource utilization graphs within the dashboard using [metrics-server](https://github.com/kubernetes-incubator/metrics-server):\n\n- [dashboard/metrics-server/deployment.yml](https://github.com/hobby-kube/manifests/blob/master/dashboard/metrics-server/deployment.yml)\n- [dashboard/metrics-server/service.yml](https://github.com/hobby-kube/manifests/blob/master/dashboard/metrics-server/service.yml)\n- [dashboard/metrics-server/apiservice.yml](https://github.com/hobby-kube/manifests/blob/master/dashboard/metrics-server/apiservice.yml)\n\nWhat's new here is that we enable **basic authentication** to restrict access to the dashboard. The following annotations are supported by the NGINX ingress controller, and may or may not work with other solutions:\n\n```yaml\n# from dashboard/ingress.yml\nannotations:\n  # ...\n  nginx.ingress.kubernetes.io/auth-type: basic\n  nginx.ingress.kubernetes.io/auth-secret: kubernetes-dashboard-auth\n  nginx.ingress.kubernetes.io/auth-realm: \"Authentication Required\"\n\n# dashboard/secret.yml\napiVersion: v1\nkind: Secret\nmetadata:\n  name: kubernetes-dashboard-auth\n  namespace: kube-system\ndata:\n  auth: YWRtaW46JGFwcjEkV3hBNGpmQmkkTHYubS9PdzV5Y1RFMXMxMWNMYmJpLw==\ntype: Opaque\n```\n\nThis example will prompt a visitor to enter their credentials (user: admin / password: test) when accessing the dashboard. Secrets for basic authentication can be created using `htpasswd`, and need to be added to the manifest as a base64 encoded string.\n\n## Distributed block storage\n\nData in containers is ephemeral, as soon as a pod gets stopped, crashes, or is for some reason rescheduled to run on another node, all its data is gone. While this is fine for static applications such as the Kubernetes Dashboard (which obtains its data from persistent sources running outside of the container), persisting data becomes a non-optional requirement as soon as we deploy databases on our cluster.\n\nKubernetes supports various [types of volumes](https://kubernetes.io/docs/concepts/storage/volumes/) that can be attached to pods. Only a few of these match our requirements. There's the `hostPath` type which simply maps a path on the host to a path inside the container, but this won't work because we don't know on which node a pod will be scheduled.\n\n### Persistent volumes\n\nThere's a concept within Kubernetes of how to separate storage management from cluster management. To provide a layer of abstraction around storage there are two types of resources deeply integrated into Kubernetes, *PersistentVolume* and *PersistentVolumeClaim*. When running on well integrated platforms such as GCE, AWS or Azure, it's really easy to attach a persistent volume to a pod by creating a persistent volume claim. Unfortunately, we don't have access to such solutions.\n\nOur cluster consists of multiple nodes and **we need the ability to attach persistent volumes to any pod running on any node**. There are a couple of projects and companies emerging around the idea of providing hyper-converged storage solutions. Some of their services are running as pods within Kubernetes itself, which is certainly the perfect way of managing storage on a small cluster such as ours.\n\n### Choosing a solution\n\nCurrently there are a couple of interesting solutions matching our criteria, but they all have their downsides:\n\n- [Rook.io](https://rook.io/) is an open source project based on Ceph. Even though it's in an early stage, it offers good documentation and is quite flexible.\n- [gluster-kubernetes](https://github.com/gluster/gluster-kubernetes) is an open source project built around GlusterFS and Heketi. Setup seems tedious at this point, requiring some kind of schema to be provided in JSON format.\n- [Portworx](https://portworx.com/) is a commercial project that offers a [free variant](https://github.com/portworx/px-dev) of their proprietary software, providing great documentation and tooling.\n\nRook and Portworx both shine with a simple setup and transparent operations. Rook is our preferred choice because it offers a little more flexibility and is open source in contrast to Portworx, even though the latter wins in simplicity by launching just a single pod per instance.\n\n### Deploying Rook\n\nAs we run only a three node cluster, we're going to deploy Rook on all three of them by adding a control-plane toleration to the Rook cluster definition.\n\nBefore deploying Rook we need to either provide a raw, unformatted block device or specify a directory that will be used for storage on each host. On a typical Ubuntu installation, the volume on which the operating system is installed is called `/dev/vda`. Attaching another volume will be available as  `/dev/vdb`.\n\nMake sure to edit the cluster manifest as shown below and choose the right configuration depending on whether you want to use a directory or a block device available in your environment for storage:\n\n```yaml\n# storage/cluster.yml\n  # ...\n  storage:\n    useAllNodes: true\n    useAllDevices: false\n    storeConfig:\n      databaseSizeMB: 1024\n      journalSizeMB: 1024\n    # Uncomment the following line and replace it with the name of block device used for storage:\n    #deviceFilter: vdb\n    # Uncomment the following lines when using a directory for storage:\n    #directories:\n    #- path: /storage/data\n```\n\nAs mentioned earlier, Rook is using [Ceph](https://ceph.com) under the hood. Run `apt-get install ceph-common` on each host to install the Ceph common utilities. Afterwards, apply the storage manifests in the following order:\n\n- [storage/00-namespace.yml](https://github.com/hobby-kube/manifests/blob/master/storage/00-namespace.yml)\n- [storage/operator.yml](https://github.com/hobby-kube/manifests/blob/master/storage/operator.yml)\n- [storage/cluster.yml](https://github.com/hobby-kube/manifests/blob/master/storage/cluster.yml) (wait for the rook-ceph-mon pods to be deployed `kubectl -n rook get pods` before continuing)\n- [storage/storageclass.yml](https://github.com/hobby-kube/manifests/blob/master/storage/storageclass.yml)\n- [storage/tools.yml](https://github.com/hobby-kube/manifests/blob/master/storage/tools.yml)\n\nIt's worth mentioning that the storageclass manifest contains the configuration for the replication factor:\n\n```yaml\n# storage/storageclass.yml\napiVersion: ceph.rook.io/v1\nkind: CephBlockPool\nmetadata:\n  name: replicapool\n  namespace: rook\nspec:\n  failureDomain: host\n  replicated:\n    size: 2 # replication factor\n---\n# ...\n```\n\nIn order to operate on the storage cluster simply run commands within the Rook tools pod, such as:\n\n```sh\n# show ceph status\nkubectl -n rook exec -it rook-tools -- ceph status\n\n# show volumes\nkubectl -n rook exec -it rook-tools -- rbd list replicapool\n\n# show volume information\nkubectl -n rook exec -it rook-tools -- rbd info replicapool/\u003cvolume\u003e\n```\n\nFurther commands are listed in the [Rook Tools documentation](https://rook.io/docs/rook/master/tools.html).\n\n### Consuming storage\n\nThe storage class we created can be consumed with a persistent volume claim:\n\n```yaml\n# minio/pvc.yml\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: minio-persistent-storage\nspec:\n  storageClassName: rook-block\n  accessModes:\n  - ReadWriteOnce\n  resources:\n    requests:\n      storage: 5Gi\n```\n\nThis will create a volume called *minio-persistent-storage* with 5GB storage capacity.\n\nIn this example we're deploying [Minio](https://minio.io), an Amazon S3 compatible object storage server, to create and mount a persistent volume:\n\n- [minio/deployment.yml](https://github.com/hobby-kube/manifests/blob/master/minio/deployment.yml)\n- [minio/ingress.yml](https://github.com/hobby-kube/manifests/blob/master/minio/ingress.yml)\n- [minio/secret.yml](https://github.com/hobby-kube/manifests/blob/master/minio/secret.yml) (MINIO_ACCESS_KEY: admin / MINIO_SECRET_KEY: admin.minio.secret.key)\n- [minio/service.yml](https://github.com/hobby-kube/manifests/blob/master/minio/service.yml)\n- [minio/pvc.yml](https://github.com/hobby-kube/manifests/blob/master/minio/pvc.yml)\n\nThe volume related configuration is buried in the deployment manifest:\n\n```yaml\n# from minio/deployment.yml\ncontainers:\n- name: minio\n  volumeMounts:\n  - name: data\n    mountPath: /data\n# ...\nvolumes:\n- name: data\n  persistentVolumeClaim:\n    claimName: minio-persistent-storage\n```\n\nThe *minio-persistent-storage* volume will live as long as the persistent volume claim is not deleted (e.g. `kubectl delete -f minio/pvc.yml `). The Minio pod itself can be deleted, updated or rescheduled without data loss.\n\n## Where to go from here\n\nThis was hopefully just the beginning of your journey. There are many more things to explore around Kubernetes. Feel free to leave feedback or raise questions at any time by opening an issue [here](https://github.com/hobby-kube/guide/issues).\n","funding_links":["https://github.com/sponsors/pstadler"],"categories":["Others","Misc","Others (1002)","HarmonyOS","devops","automation","Tools"],"sub_categories":["Windows Manager","Rust"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhobby-kube%2Fguide","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhobby-kube%2Fguide","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhobby-kube%2Fguide/lists"}