{"id":18230413,"url":"https://github.com/benc-uk/pikube","last_synced_at":"2025-04-03T15:30:23.900Z","repository":{"id":75891028,"uuid":"267404469","full_name":"benc-uk/pikube","owner":"benc-uk","description":"Building a Kubernetes cluster on the Raspberry Pi","archived":false,"fork":false,"pushed_at":"2024-02-05T16:39:15.000Z","size":6477,"stargazers_count":17,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-19T05:42:48.720Z","etag":null,"topics":["kubernetes","kubernetes-cluster","raspberry-pi"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/benc-uk.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-05-27T19:08:03.000Z","updated_at":"2025-03-17T23:34:19.000Z","dependencies_parsed_at":"2023-07-11T23:45:24.764Z","dependency_job_id":null,"html_url":"https://github.com/benc-uk/pikube","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benc-uk%2Fpikube","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benc-uk%2Fpikube/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benc-uk%2Fpikube/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benc-uk%2Fpikube/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/benc-uk","download_url":"https://codeload.github.com/benc-uk/pikube/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247027627,"owners_count":20871565,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kubernetes","kubernetes-cluster","raspberry-pi"],"created_at":"2024-11-04T11:04:06.186Z","updated_at":"2025-04-03T15:30:21.902Z","avatar_url":"https://github.com/benc-uk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Project PiKube\n\n_Project PiKube_ is a slightly grandiose title for my personal guide to building your own Kubernetes cluster, using a small number of Raspberry Pis. This is a \"bare metal\" approach and we'll be installing \u0026 setting up Kubernetes from scratch, how exciting! \n\nIf you've never used Kubernetes before, this will probably not be a good starting point, I suggest getting some familiarity before you begin (try kind, k3s or minikube). You will also need to be reasonably Linux/Unix savvy \n\n\u003cimg src=\"assets/pic1.jpg\" style=\"width:400px\"\u003e \u003cimg src=\"assets/pic2.png\" style=\"width:400px\"\u003e\n\nUpdated Feb 2024: This guide was (re)written for Raspberry Pi OS Debian 12 aka Bookworm, and Kubernetes 1.29\n\n# 🚩 Pre-reqs\n\n## Hardware\n\nI am not going to provide a full bill of materials for what you need, but at a minimum have at least two spare Raspberry Pi in order to make this worthwhile. For reference this is what I used:\n\n- 3 x Raspberry Pi 4\n- 3 x microSDHC cards (I used 32GB, I'm sure 16GB would work too)\n- 3 x Short 30cm USB C cables (There's 1000s on Amazon, find one that sells a pack of 3+)\n- 1 x [Multi port USB Power block from Anker](https://www.amazon.co.uk/gp/product/B00VTI8K9K)\n- 2 x [Cluster case / rack](https://thepihut.com/products/cluster-case-for-raspberry-pi)\n\nI used one Raspberry Pi as a master node, and two as worker nodes\n\n## Software\n\n- A SSH client\n- A decent terminal/shell\n- [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) (optional but recommended)\n\n# 🍔 Installing the OS\n\nFor this part rather than a step by step guide, these steps are very condensed as they don't really involve anything specific to this project. If this is your first time with a Raspberry Pi you might want to get familiar with using it and setting it up first.\n\n- Download Raspberry Pi OS 64-bit lite (headless) edition: https://downloads.raspberrypi.com/raspios_lite_arm64/images/\n- Image all SD cards with the official imaging tool https://www.raspberrypi.com/software/\n  - During the imaging process pick the option to customise the OS; enable WiFi, set hostnames, enable SSH with your own SSH pub-key\n  - Pick names for your nodes, I went for `master` `node1` and `node2`\n- Get all nodes booted and on the network\n\n⚠ IMPORTANT: It's **highly** recommended to set static IPs for all nodes, [see the appendix](#appendix-1---static-ip-on-raspberry-pi-os-bookworm)\n\nOther notes:\n\n- Simply joining the Pis to your home WiFi network works very well. Surprisingly there is no need to use ethernet, or set up a dedicated network or hub for the cluster, of course you can if you wish, this is an exercise left to the reader\n- We will do the install of Kubernetes using SSH, so ensure you can SSH into the nodes from your machine either with a password or SSH keys (I would strongly advise SSH keys!)\n- You should not need a keyboard or screen connected to the Pis if you enabled SSH and have them on your network (i.e. configured with WiFi) unless something has gone wrong!\n\n# 💾 Base Configuration \u0026 Setup (all nodes)\n\nNearly every command we will be running will require root access, rather than place sudo literally everywhere, you should switch to root user with `sudo su -` for all of the setup, install and initial configuration\n\nUpdate the OS and install common packages\n\n```sh\napt update \u0026\u0026 apt -y full-upgrade\napt -y install iptables apt-transport-https gnupg2 software-properties-common \\\n  apt-transport-https ca-certificates curl vim git\n```\n\nDisable swap, it's probably not enabled but just in case\n\n```sh\nswapoff -a\nsudo dphys-swapfile swapoff \u0026\u0026 \\\nsudo dphys-swapfile uninstall \u0026\u0026 \\\nsudo systemctl disable dphys-swapfile\n```\n\nEnable iptables as kube-proxy uses it\n\n```sh\nupdate-alternatives --set iptables /usr/sbin/iptables-legacy\nupdate-alternatives --set ip6tables /usr/sbin/ip6tables-legacy\n```\n\nEnable some kernel modules which are needed\n\n```sh\ntee /etc/modules-load.d/k8s.conf \u003c\u003cEOF\noverlay\nbr_netfilter\nEOF\n\nmodprobe overlay\nmodprobe br_netfilter\n```\n\nEdit kernel boot command `/boot/firmware/cmdline.txt` and add the following to the end\n\n```text\ncgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1\n```\n\nConfigure the IP stack with some sysctl tunables\n\n```sh\ntee /etc/sysctl.d/kubernetes.conf \u003c\u003cEOF\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nEOF\n\nsysctl --system\n```\n\n**⚠ IMPORTANT**: Now reboot the system\n\n# 📦 Install Container Runtime (all nodes)\n\nWe will install \u0026 use containerd. These steps should also be run as root.\n\n```sh\ncurl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/trusted.gpg.d/debian.gpg\nadd-apt-repository \"deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/debian $(lsb_release -cs) stable\"\napt update\napt install -y containerd.io\n```\n\nCreate default containerd configuration\n\n```sh\nmkdir -p /etc/containerd\ncontainerd config default | tee /etc/containerd/config.toml\n```\n\n**⚠ IMPORTANT**: Set cgroup driver for runc to use systemd. See [note](https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd-systemd). If you forget this step, Bad Things(TM) happen later and you won't get any helpful error messages to explain why, just a headache.\n\n```sh\nsudo sed -i 's/            SystemdCgroup = false/            SystemdCgroup = true/' /etc/containerd/config.toml\n```\n\nYou can also edit /etc/containerd/config.toml manually to make this change if sed is too hacky for your tastes.\n\nNow enable \u0026 start containerd\n\n```sh\nsystemctl restart containerd\nsystemctl enable containerd\n```\n\n# ☸️ Install core Kubernetes (all nodes)\n\nAdd the Kubernetes package repositories. Note these are the [new k8s.io community repos](https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/).\n\n```sh\nKUBE_VER=\"v1.29\"\n\necho \"deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/$KUBE_VER/deb/ /\" | tee /etc/apt/sources.list.d/kubernetes.list\ncurl -fsSL https://pkgs.k8s.io/core:/stable:/$KUBE_VER/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg\n```\n\nInstall kubelet, kubeadm and kubectl, they are kinda important!\n\n```sh\napt update\napt install -y kubelet kubeadm kubectl\n```\n\nVerify if you're feeling paranoid\n\n```sh\nkubectl version --client\nkubeadm version\n```\n\n# 👑 Install Control Plane (master only)\n\nOk now we're ready to actually start getting Kubernetes set up, first enable the kubelet service on the master node\n\n```sh\nsystemctl enable kubelet\n```\n\nPull the images need by Kubernetes\n\n```sh\nkubeadm config images pull\n```\n\nNow you are ready to initialize the control plane. I highly suggest running a single master node so you don't need to worry about some of the many (many!) advanced options when running `kubeadm init`.  \nNote `10.244.0.0/16` is the pod CIDR that Flannel likes to use by default (see below)\n\n```sh\nkubeadm init --pod-network-cidr=10.244.0.0/16\n```\n\n**⚠ IMPORTANT** Make a note of the join details that are output, this will be shown once the init process is complete\n\nStill as the root user you can use `kubectl` to validate everything is started up. Note the CoreDNS pods will not start until after the next step, so don't worry about them.\n\n```sh\nexport KUBECONFIG=/etc/kubernetes/admin.conf\nkubectl get pods -A\n```\n\nNow you can stop running as root, switch to a regular user (but still on the master node) and run the following to configure kubectl for that user\n\n```sh\nmkdir -p $HOME/.kube\nsudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\nsudo chown $(id -u):$(id -g) $HOME/.kube/config\n```\n\nInstall a pod overlay network add-on, we'll use Flannel as it's simple and does the job, but [many other options are available](https://kubernetes.io/docs/concepts/cluster-administration/addons/)\n\n```sh\nkubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml\n```\n\nValidate and check the CoreDNS \u0026 kube-flannel pods start up\n\n```sh\nkubectl get po -A\n```\n\nIf everything is running you have a working Kubernetes master and control plane! \n\n**Optional but HIGHLY recommended**: You really shouldn't need to be SSH'ed into your cluster in order to do anything, so now is a good time to scp the kube config file to outside the master to allow remote admin (kubectl) from your dev machine or other system\n\n```sh\nscp master:~/.kube/config ~/pikube.config\nexport KUBECONFIG=~/pikube.config\nkubectl get pods -A\n```\n\n# 🛠️ Join Worker Nodes (workers only)\n\nAdding additional worker nodes to the cluster is very easy, and when you finished running the `kubeadm init` on the master you would have been given a command to join other nodes, below is just an example\n\n```sh\nsudo kubeadm join 192.168.0.150:6443 \\\n  --token __CHANGE_ME__ \\\n  --discovery-token-ca-cert-hash __CHANGE_ME__\n```\n\nIf you missed or lost the output after the `kubeadm init` step, then you can SSH onto the master and use the `kubeadm token` command to create a new one.\n\nValidate the node(s) joined the cluster successfully by running `kubectl get node`\n\nIt also helps to label your node(s) as workers, e.g.\n\n```sh\nkubectl label node __CHANGE_ME__ node-role.kubernetes.io/worker=worker\n```\n\nYou can repeat this step for each of your worker nodes\n\n# 💡 Conclusion\n\nOK you should now have a basic but functional Kubernetes cluster, deploy a test web application by running `kubectl apply -f ./samples/test-app.yaml` to verify everything is working.  \n\nThen run `kubectl get pods -o wide` and check the pods are running and they are assigned to your worker nodes\n\nGo to the following URL `http://master:30000/` (if your master node hostname is not `master` then change it) to open the app.  \n\nIf it loads then congratulations you've built a Kubernetes cluster from nothing and deployed an application to it. Go you! celebrate with a coffee/tea/beer/soda/absinthe you've earned it. 🥳\n\n# 🐾 Next Steps\n\nThere's practically an endless number of things you can do next with your cluster, however there are a few common next steps to get it to the next level and more functional. A few of these will require you to have a git clone of this repo, or you can just download the zip if you don't have git.\n\n- Deploy metrics server by running: `kubectl apply -f samples/metrics-server.yaml`\n- [Add an Ingress Controller](./ingress.md)\n- [Deploy Kubernetes dashboard](./dashboard.md)\n- [Enable NFS based storage for persistent volumes](./storage/)\n- Support load balancing with MetalLB - *Coming soon!*\n- [Deploy \u0026 use Prometheus and Grafana](./monitoring.md)\n- If you have a [Display-o-tron 3000](https://shop.pimoroni.com/products/display-o-tron-hat) I have created [a Python script to display all sorts of details about your cluster](./status-dot3k/)\n\n---\n\n## Appendix 1 - Static IP on Raspberry Pi OS Bookworm\n\nThere's a lot of outdated information on setting static IP on the Raspberry Pi, and as of version 'Bookworm' (Oct 2023) the method has changed drastically, and now uses network manager and NOT dhcpcd. Finding this out cost me *hours*, so I've called this step out with a little detail so you don't fall into the same trap.\n\nYou'll need to find the name of the network config for the network interface you are using (either eth0 or wlan0)\n\n```sh\nsudo nmcli c show\n```\n\nTo set the connection to be static, the following commands can be used.  \nObviously modify the `__CHANGE_ME__` to the connection name, and IP addresses below to what you need, and don't just copy, paste \u0026 run!\n\n```sh\nsudo nmcli c mod '__CHANGE_ME__' ipv4.addresses 192.168.0.150/24 ipv4.method manual\nsudo nmcli c mod '__CHANGE_ME__' ipv4.gateway 192.168.0.1\nsudo nmcli c mod '__CHANGE_ME__' ipv4.dns 192.168.0.1\n```\n\nIn my case my three nodes were 192.168.0.150 ~ 152 and my router \u0026 DNS server was on 192.168.0.1\n\nAssuming you are making these changes via SSH, it's simplest to reboot to the IP address change take effect.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenc-uk%2Fpikube","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbenc-uk%2Fpikube","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenc-uk%2Fpikube/lists"}