{"id":16567166,"url":"https://github.com/zloeber/k8s-lab-terraform-libvirt","last_synced_at":"2025-03-05T10:28:30.440Z","repository":{"id":40623793,"uuid":"257393337","full_name":"zloeber/k8s-lab-terraform-libvirt","owner":"zloeber","description":"A Kubernetes lab environment using terraform and libvirt","archived":false,"fork":false,"pushed_at":"2020-05-01T03:14:00.000Z","size":143,"stargazers_count":40,"open_issues_count":0,"forks_count":16,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-15T23:43:10.532Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zloeber.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-20T20:15:24.000Z","updated_at":"2024-09-03T08:39:28.000Z","dependencies_parsed_at":"2022-09-15T09:12:27.534Z","dependency_job_id":null,"html_url":"https://github.com/zloeber/k8s-lab-terraform-libvirt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zloeber%2Fk8s-lab-terraform-libvirt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zloeber%2Fk8s-lab-terraform-libvirt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zloeber%2Fk8s-lab-terraform-libvirt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zloeber%2Fk8s-lab-terraform-libvirt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zloeber","download_url":"https://codeload.github.com/zloeber/k8s-lab-terraform-libvirt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242008883,"owners_count":20056983,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T21:05:49.399Z","updated_at":"2025-03-05T10:28:30.415Z","avatar_url":"https://github.com/zloeber.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Libvirt/Terraform K8s Lab Environment\n\n## Purpose\n\nThis is meant to be a one stop location for studying for the CKA exam where you can bring up a multi-node lab locally with minimal effort and start practicing your Kubernetes kung-fu. This lab will kickstart your nodes up to the point where you need to get them clustered via kubeadm.\n\n## Requirements\n\nThis uses the libvirt provider for terraform which, as far as I can see, only creates prebuilt binaries for Linux. As such, Linux as your base host is a requirement.\n\nI got this all running using Qemu as the backing provider for libvirt. I also used the Ubuntu release version of the libvirt terraform provider (this can be changed in the init.sh script to some other supported Distro though). As such an Ubuntu based host should be used. You will also want to ensure that the folder you cloned this repo to is on a disk with plenty of space and a host that can accommodate the three VMs that will get spun up.\n\nWe need a few base apps and utilities (probably more than this needs to be installed, sorry if the base list of packages isn't 100% correct)\n\n```bash\nsudo apt install libvirt-clients libvirt-daemon qemu tmux genisoimage\n```\n\nYour account must have access to manage libvirt resources as well this should get you started.\n\n```bash\nsudo usermod -a -G libvirt $(whoami)\nnewgrp libvirt\n```\nWe need to manually pull in the most recent libvirt terraform provider plugin and create an ssh key for use in the deployment. You can do all of this easily enough with a quick make task.\n\n```bash\nmake deps\n```\n\n## The Lab\n\nThis lab consists of 3 virtual machines with the following names and specifications:\n\n| Node | vCPUS | RAM | Disk | IP |\n|---|---|---|---|---|\n| k8s-master-1 | 2 | 4GB | 40GB | 172.16.1.11/24 |\n| k8s-worker-1 | 2 | 2GB | 40GB | 172.16.1.21/24 |\n| k8s-worker-2 | 2 | 2GB | 40GB | 172.16.1.22/24 |\n\n```bash\n# If all the dependencies are in place then initialize, plan, and apply the terraform manifest to bring things up.\nmake init plan create\n\n# Destroy just as easily.\nmake destroy\n\n# Access the master node\nmake ssh/master\n\n# or the worker nodes\nmake ssh/worker1\nmake ssh/worker2\n\n# or all the nodes at once with synchronized input\nmake ssh/all\n```\n\nYou can increase the number of master/worker nodes by updating the variables at the top of the main.tf file. The Makefile was only setup to work with the three nodes though as that's all you really need for most basic lab environments.\n\nIf you need to update/change the Kubernetes component versions please update the main.tf variables.\n\n**NOTE:** When you bring the lab up for the first time it may take a few minutes to install docker and become fully ready for a Kubernetes installation.\n\n## Cloud Init\n\nThe cloud_init.cfg template is used to provide an initial configuration for the instances. This includes;\n\n- Setting the local hostname (important for the cluster to work properly)\n- Adding the docker repo and gpg key\n- Adding the kubernetes repo and gpg key\n- Installing some initial required packages common to all nodes\n- Configure the default user (ubuntu) with passwordless sudo rights.\n- Install containerd.io, docker-ce, docker-ce-cli, kubelet, kubeadm, and kubectl (and put kube* marked for hold)\n- Deploy docker service as a daemon\n\nSee the cloud_init.cfg for further information or to customize the deployment further.\n\n## Extras\n\nThere are some additional configuration tasks that you can use to shortcut a cluster setup found in the `tasks/1-deploy-kubernetes.md` documentation.\n\nIf you are brave and are using asdf-vm along with direnv you may also be able to use the included ansible playbook to deploy your cluster even more quickly. This assumes that the direnv Python virtual environment got setup correctly and that asdf is also working. The original playbook (see link in references) was modified to not install docker and does not do any additional deployment beyond the calico CNI setup and node joins via kubeadm. Nor has any other CNI been tested.\n\n```bash\nmake cluster/deploy\nmake kube/export/config\n```\n\nIt wouldn't be too hard to dissect the makefile task to run this playbook manually of course.\n\n## Issues\n\nOn Ubuntu distros SELinux is enforced by qemu even if it is disabled globally, this might cause unexpected Could not open '/var/lib/libvirt/images/\u003cFILE_NAME\u003e': Permission denied errors. Double check that security_driver = \"none\" is uncommented in /etc/libvirt/qemu.conf and issue sudo systemctl restart libvirt-bin to restart the daemon. [source](https://github.com/dmacvicar/terraform-provider-libvirt/issues/546)\n\nI also had to do some things found [here](https://github.com/jedi4ever/veewee/issues/996) to get this resolved,\n\n```bash\nsudo groupadd --system libvirt\nsudo usermod -a -G libvirt $(whoami)\nnewgrp libvirt\nid $(whoami)\nsudo systemctl stop libvirtd.service\nsudo nano /etc/libvirt/qemu.conf\nsudo echo \"user = `whoami`\" \u003e\u003e /etc/libvirt/qemu.conf\nsudo echo \"group = `whoami`\" \u003e\u003e /etc/libvirt/qemu.conf\nsudo systemctl start libvirtd.service\n```\n\n## Exam Tips\n\nHere are a few random tips which may or may not help you on exam day.\n\n**kubectl explain**\n\nUsing `kubectl explain` will show more information on just about everything you may be authoring in YAML manfiests. The format is: `\u003cresource\u003e.\u003cgroup\u003e.\u003cproperty\u003e` for individual properties. Or you can get all available properties by omitting the `.\u003cproperty\u003e`. With some creativity and the `--recursive=true` flag you can get somthing that looks almost like a YAML snippet.\n\n```bash\n# Show all available properties for pod.spec\nkubectl explain pod.spec | grep '\u003c'\n\n# Show all available properties for deployment.spec.template.spec.containers\nkubectl explain deployment.spec.template.spec.containers --recursive=true | grep '\u003c' \n```\n\n## Links\n\n[CKA Practice Excercises](https://github.com/alijahnas/CKA-practice-exercises) (initial source of the terraform manifest and a great study guide!)\n\n[Additional CKA Study Repo 1](https://github.com/walidshaari/Kubernetes-Certified-Administrator)\n\n[Additional CKA Study Repo 2](https://github.com/dgkanatsios/CKAD-exercises)\n\n[A Great CKA Study Guide](http://www.kubernet.io/)\n\n[Very Good Kube Troubleshooting Handbook](https://github.com/feiskyer/kubernetes-handbook/blob/master/en/troubleshooting/index.md)\n\n[Libvirt Terraform Provider](https://github.com/dmacvicar/terraform-provider-libvirt)\n\n[xpanes for tmux](https://github.com/greymd/tmux-xpanes)\n\n[Cloud Init Examples](https://cloudinit.readthedocs.io/en/latest/topics/examples.html)\n\n[kubeadm via Ansible](https://github.com/kairen/kubeadm-ansible)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzloeber%2Fk8s-lab-terraform-libvirt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzloeber%2Fk8s-lab-terraform-libvirt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzloeber%2Fk8s-lab-terraform-libvirt/lists"}