{"id":22983250,"url":"https://github.com/mikejoh/k8s-on-kvm","last_synced_at":"2026-01-30T15:32:56.873Z","repository":{"id":217947867,"uuid":"610202842","full_name":"mikejoh/k8s-on-kvm","owner":"mikejoh","description":"Deploy small clusters for learning purposes e.g. CKA/CKAD/CKS studies! :books:","archived":false,"fork":false,"pushed_at":"2024-12-03T18:57:44.000Z","size":63,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-06T09:51:55.355Z","etag":null,"topics":["certified-kubernetes-administrator","kubernetes","libvirt","libvirt-provider","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mikejoh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-06T09:54:24.000Z","updated_at":"2025-01-31T07:02:57.000Z","dependencies_parsed_at":"2024-11-06T21:19:12.383Z","dependency_job_id":"a2288fb4-bd35-4e26-bb20-f61fe6afe3fe","html_url":"https://github.com/mikejoh/k8s-on-kvm","commit_stats":null,"previous_names":["mikejoh/k8s-on-kvm"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mikejoh/k8s-on-kvm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikejoh%2Fk8s-on-kvm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikejoh%2Fk8s-on-kvm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikejoh%2Fk8s-on-kvm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikejoh%2Fk8s-on-kvm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mikejoh","download_url":"https://codeload.github.com/mikejoh/k8s-on-kvm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikejoh%2Fk8s-on-kvm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28914905,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T12:13:43.263Z","status":"ssl_error","status_checked_at":"2026-01-30T12:13:22.389Z","response_time":66,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["certified-kubernetes-administrator","kubernetes","libvirt","libvirt-provider","terraform"],"created_at":"2024-12-15T02:37:00.581Z","updated_at":"2026-01-30T15:32:56.853Z","avatar_url":"https://github.com/mikejoh.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kubernetes on KVM\n\nDeploy (small) Kubernetes clusters on KVM, all from scratch! 🚀\n\n## ✅ Pre-requisites\n\n* KVM (see your favourite Linux distribution how-to)\n* OpenTofu (the `terraform` fork)\n\n## 🗒️ Important notes\n\n* See the `scripts/` folder for various utility scripts.\n* You probably want to change `user` and `group` to `libvirt-qemu` and `kvm` respectively in `/etc/libvirt/qemu.conf` to mitigate permission issues on storage pools.\n* All cluster nodes will be running Ubuntu 24.04.\n\n## 🏃 Getting started\n\nIf you run a `ufw` firewall locally you might need to add the following before continuing:\n\n```bash\nufw allow in on virbr1\nufw allow out on virbr1\n```\n\n### Provision cluster nodes\n\nBy default we'll deploy a cluster on three nodes, they will have both the control-plane and worker roles.\n\n_If you're setting a lower resource values on each node then you might need to set: `--ignore-preflight-errors=mem,numcpu` during `kubeadm init`._\n\n1. Change the `k8s.auto.tfvars` to fit your needs!\n2. Run `tofu init`\n3. Run `tofu plan`\n4. Run `tofu apply`\n5. The needed nodes shall be provisioned with everything included for you to start bootstrapping the cluster.\n\n### Accessing the nodes\n\n_Remember to start your VMs after a reboot!_\n\n#### SSH to one node a time\n\nTo find out the IP addresses of the VMs you can run the following using `virsh`:\n\n```bash\nsudo virsh net-dhcp-leases k8s_net\n```\n\n#### SSH using the provided helper script\n\nThis requires that you have [`fzf`](https://github.com/junegunn/fzf) installed.\n\n```bash\nPRIVATE_SSH_KEY=~/.ssh/kvm-k8s scripts/ssh.sh\n```\n\n_Make sure you're using the private key that matches the public key added as part of the cluster node provisioning. We're adding a user called `cloud` by default that has the provided public key as one of the `ssh_authorized_keys`._\n\n#### Using `tmux` and `xpanes` to SSH to all available nodes\n\nDon't forget to inline the private key path below and replace `\u003cPRIVATE_KEY\u003e` before running the command:\n\n```bash\ntmux\n\nsudo virsh net-dhcp-leases k8s_net | tail -n +3 | awk '{print $5 }' | cut -d\"/\" -f1 | xpanes -l ev -c 'ssh -l cloud -i \u003cPRIVATE_KEY\u003e {}'\n```\n\n### Bootstrap the first control-plane node\n\n_Please note that we're skipping the addon `kube-proxy` since in these clusters we want to utilize Cilium as the CNI and with a configuration that replaces the need for `kube-proxy`._\n\n```bash\nCONTROL_PLANE_IP=$(ip addr show ens3 | grep 'inet ' | awk '{print $2}' | cut -d/ -f1)\nsudo kubeadm init --control-plane-endpoint \"$CONTROL_PLANE_IP:6443\" --upload-certs\n```\n\n### Add a worker node\n\nAdd worker nodes to the cluster:\n\n1. Generate the join command on a control-plane node:\n\n```bash\nkubeadm token create --print-join-command\n```\n\n2. Use the generate join command and run that on the worker node.\n\n### Upgrade cluster\n\n_As of writing this the clusters are deployed with the latest available patch release of `v1.30`. The following guide will upgrade the cluster to `v1.31.2`._\n\n1. Add the following line to the `/etc/apt/sources.list.d/kubernetes.list`, note the Kubernetes version:\n\n```bash\necho \"deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /\" \u003e\u003e /etc/apt/sources.list.d/kubernetes.list\n```\n\n2. Run `apt update`.\n\n3. Check which install candidate we have: `apt policy kubeadm`\n\n```bash\nkubeadm:\n  Installed: 1.30.6-1.1\n  Candidate: 1.31.2-1.1\n  Version table:\n *** 1.31.2-1.1 500\n        500 https://pkgs.k8s.io/core:/stable:/v1.31/deb  Packages\n     1.31.1-1.1 500\n        500 https://pkgs.k8s.io/core:/stable:/v1.31/deb  Packages\n     1.31.0-1.1 500\n        500 https://pkgs.k8s.io/core:/stable:/v1.31/deb  Packages\n     1.30.6-1.1 500\n        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages\n     1.30.5-1.1 500\n        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages\n     1.30.4-1.1 500\n        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages\n     1.30.3-1.1 500\n        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages\n     1.30.2-1.1 500\n        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages\n     1.30.1-1.1 500\n        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages\n     1.30.0-1.1 500\n        500 https://pkgs.k8s.io/core:/stable:/v1.30/deb  Packages\n```\n\n4. Install a newer version `kubeadm`:\n\n```bash\napt install kubeadm=1.31.2-1.1\n```\n\n5. Drain and cordon the control-plane node:\n\n```bash\nkubectl drain \u003cnode name\u003e --ignore-daemonsets --delete-emptydir-data --disable-eviction\n```\n\n_Use `--disable-eviction` flag with caution, in this case we'll have a small test cluster which most likely wont accomodate for any Pod Distruption Budgets configured._\n\n6. Check the upgrade `plan` and proceed with upgrade:\n\n```bash\nkubeadm upgrade plan\nkubeadm upgrade apply v1.31.2\n```\n\n```bash\n[upgrade/version] You have chosen to change the cluster version to \"v1.31.2\"\n[upgrade/versions] Cluster version: v1.30.6\n[upgrade/versions] kubeadm version: v1.31.2\n[upgrade] Are you sure you want to proceed? [y/N]: y\n```\n\n7. Upgrade `kubelet` and `kubectl`:\n\n```bash\napt install kubelet=1.31.2-1.1 kubectl=1.31.2-1.1\n```\n\n8. Continue with the rest of the control-plane nodes.\n\nCheck versions:\n\n```bash\nkubectl version\nClient Version: v1.31.2\nKustomize Version: v5.4.2\nServer Version: v1.31.2\n\nkubectl get nodes\nNAME         STATUS   ROLES           AGE     VERSION\nk8s01-cp01   Ready    control-plane   4d21h   v1.31.2\nk8s01-w01    Ready    \u003cnone\u003e          4d20h   v1.30.6\n```\n\n_Please note that for workers you'll only install a newer version of the `kubelet`!_\n\n### Clean up the cluster\n\nRun the clean-up utility script: `scripts/clean_up.sh`, please note that this removes everything related to the VMs and `tofu` state.\n\n## 📚 Add-ons\n\n### Install Cilium\n\n```bash\n./scripts/install_cilium.sh\n```\n\n_Note that we run Cilium with `kubeProxyReplacement=true` with `kube-proxy` running, you could remove all things related to `kube-proxy` [manually](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#quick-start) or skip the `kube-proxy` phase during `kubeadm init`._\n\n### Install Open Policy Agent Gatekeeper\n\n1. Install:\n\n```bash\n./scripts/install_opa.sh\n```\n\n2. Create a constraint template and a constraint:\n\n```bash\nkubectl create -f manifests/opa/image-constraint.yaml\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikejoh%2Fk8s-on-kvm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmikejoh%2Fk8s-on-kvm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikejoh%2Fk8s-on-kvm/lists"}