{"id":24189357,"url":"https://github.com/devsunb/k8s-on-macos","last_synced_at":"2025-10-15T23:09:51.838Z","repository":{"id":241767548,"uuid":"799664848","full_name":"devsunb/k8s-on-macos","owner":"devsunb","description":null,"archived":false,"fork":false,"pushed_at":"2024-05-30T06:58:29.000Z","size":58,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-01-13T14:36:28.381Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devsunb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-12T19:59:41.000Z","updated_at":"2024-05-30T06:58:32.000Z","dependencies_parsed_at":"2024-05-30T02:12:38.960Z","dependency_job_id":"ac4a66e1-ee98-48fe-8e33-5d33cca93a7a","html_url":"https://github.com/devsunb/k8s-on-macos","commit_stats":null,"previous_names":["devsunb/k8s-on-macos"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devsunb%2Fk8s-on-macos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devsunb%2Fk8s-on-macos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devsunb%2Fk8s-on-macos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devsunb%2Fk8s-on-macos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devsunb","download_url":"https://codeload.github.com/devsunb/k8s-on-macos/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241601163,"owners_count":19988878,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-13T14:29:25.612Z","updated_at":"2025-10-15T23:09:51.730Z","avatar_url":"https://github.com/devsunb.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kubernetes on macOS (Apple Silicon)\n\nDescribe how to configure a Kubernetes environment for local testing on macOS Apple Silicon.\n\n# Background\n\nAs a DevOps Engineer working with MacBook, I need to configure a local Kubernetes Cluster for testing.\n\nBut VM is required to run docker engine in macOS, and there is too many tools for creating vm and configuring docker/kubernetes environment.\n\nI tried multipass, utm, lima, colima, minikube, kind, k3d and decided to use below\n\n- a fully functional kubernetes cluster (kube-vip, 3 CPs, 3 Workers) based on lima and kubeadm for learning kubernetes architecture.\n- colima docker + minikube kubernetes cluster (1 CP, can add worker node) for daily work, testing multi-node (Affinity, PodDisruptionBudge, cordon, drain, ...), and testing multiple kubernetes versions\n\n# lima\n\nSource: \u003chttps://github.com/louhisuo/k8s-on-macos\u003e\n\n## Versions\n\n- Lima VM 0.22.0 / socket_vmnet 1.1.4 - Virtualization\n- Debian 12 - Node images\n- Kubernetes 1.30.1 - Kubernetes release\n- Cilium 1.15.4 - CNI, L2 LB, L7 LB (Ingress Controller) and L4/L7 LB (Gateway API)\n- Gateway API 1.1 - CRDs supported by Cilium 1.15.4\n- kube-vip 0.8.0 - Kubernetes Control Plane LB\n- metrics-server 0.7.1\n- local-path-provisioner 0.0.26\n\n## Prerequisites\n\n### Required Tools\n\nFollowing tools are required by this project.\n\n- [lima](https://github.com/lima-vm/lima)\n- [socket_vmnet](https://github.com/lima-vm/socket_vmnet/)\n- [kubernetes-cli](https://github.com/kubernetes/kubectl)\n- [helm](https://helm.sh/)\n- [git](https://git-scm.com/)\n- [cilium-cli](https://github.com/cilium/cilium-cli/)\n\n[Homebrew](https://brew.sh/) can be used to install these tools on macOS host.\n\n```sh\nbrew install lima socket_vmnet kubernetes-cli helm git\n```\n\n`cilium-cli` macOS install script can be found on [Cilium Getting Started Doc](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/)\n\n```sh\nCILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)\nCLI_ARCH=amd64\nif [ \"$(uname -m)\" = \"arm64\" ]; then CLI_ARCH=arm64; fi\ncurl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}\nshasum -a 256 -c cilium-darwin-${CLI_ARCH}.tar.gz.sha256sum\nsudo tar xzvfC cilium-darwin-${CLI_ARCH}.tar.gz /usr/local/bin\nrm cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}\n```\n\n### Networking\n\nShared network `192.168.105.0/24` in macOS is used as it allows both Host-VM and VM-VM communication.\nBy default Lima VM uses DHCP range until `192.168.105.254` therefore we use IP address range\nfrom `192.168.105.100` and onward in our Kubernetes setup.\nTo have predictable node IPs for a Kubernetes cluster,\nit is neccessary to [reserve IPs](https://github.com/lima-vm/socket_vmnet#how-to-reserve-dhcp-addresses) to be used from DHCP server in macOS.\n\n#### Kubernetes node IP range\n\nDefine following [/etc/bootptab](./etc/bootptab) file.\n\n```sh\n# WARNING: this will overwrite your bootptab\nsudo cp etc/bootptab /etc/bootptab\n```\n\n| Host     | MAC Address       | IP address      |\n| -------- | ----------------- | --------------- |\n| cp       | 00:00:00:00:00:00 | 192.168.105.100 |\n| cp-1     | 00:00:00:00:00:01 | 192.168.105.101 |\n| cp-2     | 00:00:00:00:00:02 | 192.168.105.102 |\n| cp-3     | 00:00:00:00:00:03 | 192.168.105.103 |\n| worker-1 | 10:00:00:00:00:01 | 192.168.105.201 |\n| worker-2 | 10:00:00:00:00:02 | 192.168.105.202 |\n| worker-3 | 10:00:00:00:00:03 | 192.168.105.203 |\n\nIf DHCP daemon already running, reload it.\n\nBefore first VM start, `system/com.apple.bootpd` service might not exist.\n\n```sh\nsudo /bin/launchctl print system/com.apple.bootpd # if state = running, run below\nsudo /bin/launchctl kickstart -kp system/com.apple.bootpd\n```\n\n#### Kubernetes API server\n\nKubernetes API server is available via VIP address `192.168.105.100`.\n\n#### L2 Aware load balancer, Ingress and Gateway\n\nIP address pool for `Load Balancer` services must be configured to same shared subnet than cluster `node IPs`. Currently L2 Aware LB in Cilium CNI is used and default address pool is configured is `192.168.105.240/28`.\n\nFrom the assigned address pool following IPs are \"reserved\", leaving 12 addresses available for different LB services.\n\n- `192.168.105.241` is assigned to `Ingress` (Cilium Ingress Controller) and\n- `192.168.105.242` is reserved for `Gateway` (Cilium Gateway API). `Gateway` configuration is work in progress.\n\n## Misc\n\n### Exposing services via NodePort to macOS host\n\nIt is possible to expose Kubernetes services via `NodePort` to `macOS` host. Full `NodePort` range `30000-32767` is exposed to `macOS` host from provisioned `Lima VM` machines during machine creation phaase.\n\nActual services with `type: NodePort` will be available on `macOS` host via `node IP` address of any Control Plane or Worker nodes of a cluster (not via VIP address) and assigned `NodePort` value for a service.\n\n## Setup\n\n### Set lima sudoers\n\n```sh\nlimactl sudoers | sudo tee /private/etc/sudoers.d/lima\n```\n\n### Create VMs\n\n```sh\n# if you create lima vm for the first time, lima will download vm image, so wait for the first vm to create before creating the others or you will get a file error.\nlimactl create --set='.networks[].macAddress=\"00:00:00:00:00:01\"' --name cp-1 lima.yaml --tty=false\nlimactl create --set='.networks[].macAddress=\"00:00:00:00:00:02\"' --name cp-2 lima.yaml --tty=false\nlimactl create --set='.networks[].macAddress=\"00:00:00:00:00:03\"' --name cp-3 lima.yaml --tty=false\nlimactl create --set='.networks[].macAddress=\"10:00:00:00:00:01\"' --name worker-1 lima.yaml --tty=false\nlimactl create --set='.networks[].macAddress=\"10:00:00:00:00:02\"' --name worker-2 lima.yaml --tty=false\nlimactl create --set='.networks[].macAddress=\"10:00:00:00:00:03\"' --name worker-3 lima.yaml --tty=false\n\n# if you start lima vm for the first time, lima will create user config file, so wait for the first vm to run before starting the others or you will get a file error.\n# and if you're staring more than one vm, start them at the same time, or start the other vms after the startup is complete.\n# Starting a vm when another vm is in the middle of starting causes a network error.\nlimactl start cp-1\nlimactl start cp-2\nlimactl start cp-3\nlimactl start worker-1\nlimactl start worker-2\nlimactl start worker-3\n\nlimactl shell cp-1\nlimactl shell cp-2\nlimactl shell cp-3\nlimactl shell worker-1\nlimactl shell worker-2\nlimactl shell worker-3\n```\n\nRun in all vms\n\n```sh\nsudo apt update \u0026\u0026 sudo apt upgrade -y\n\n# Network configuration\ncat \u003c\u003cEOF | sudo tee -a /etc/hosts\n192.168.105.101 cp-1\n192.168.105.102 cp-2\n192.168.105.103 cp-3\n192.168.105.201 worker-1\n192.168.105.202 worker-2\n192.168.105.203 worker-3\nEOF\n\ncat \u003c\u003cEOF | sudo tee -a /etc/modules-load.d/k8s.conf\noverlay\nbr_netfilter\nEOF\n\nsudo modprobe overlay\nsudo modprobe br_netfilter\n\ncat \u003c\u003cEOF | sudo tee -a /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-iptables  = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward                 = 1\nEOF\n\nsudo sysctl --system\n\n# Install containerd\nsudo mkdir -p /etc/containerd\ncontainerd config default | sudo tee /etc/containerd/config.toml\nsudo sed -i 's#SystemdCgroup = false#SystemdCgroup = true#' /etc/containerd/config.toml\nsudo sed -i 's#sandbox_image = \"registry.k8s.io/pause:3.8\"#sandbox_image = \"registry.k8s.io/pause:3.9\"#' /etc/containerd/config.toml\nsudo systemctl restart containerd\nsudo systemctl enable containerd\n\n# Install kubernetes\nK8S_REPO=v1.30\nK8S_VERSION=1.30.1\n\nsudo apt install -y apt-transport-https ca-certificates curl gpg ipvsadm jq\n\ncurl -fsSL https://pkgs.k8s.io/core:/stable:/${K8S_REPO}/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg\necho 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/'${K8S_REPO}'/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list\n\nsudo apt update\nsudo apt install -y kubeadm=${K8S_VERSION}-* kubectl=${K8S_VERSION}-* kubelet=${K8S_VERSION}-*\nsudo apt-mark hold kubeadm kubectl kubelet\nsudo systemctl enable --now kubelet\n```\n\n### Initiate Kubernetes Cluster\n\n#### Initiate Kubernetes Control Plane\n\nrun in each cp vms\n\n```sh\nexport KVVERSION=v0.8.0\nexport INTERFACE=lima0\nexport VIP=192.168.105.100\nsudo ctr image pull ghcr.io/kube-vip/kube-vip:$KVVERSION\nsudo ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip manifest pod \\\n    --arp \\\n    --controlplane \\\n    --address $VIP \\\n    --interface $INTERFACE \\\n    --enableLoadBalancer \\\n    --leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml\n\n# Kubernetes \u003e=1.29 kube-vip workaround https://github.com/kube-vip/kube-vip/issues/684#issuecomment-1864855405\nsudo sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml\n\n# run next line only in cp-1 and it should be done before running cp-2, cp-3 join command\nsudo kubeadm init --upload-certs --config manifests/kubeadm/cp-1-init-cfg.yaml\n# run next line only in cp-2\nsudo kubeadm join --config manifests/kubeadm/cp-2-join-cfg.yaml\n# run next line only in cp-3\nsudo kubeadm join --config manifests/kubeadm/cp-3-join-cfg.yaml\n\nsudo sed -i 's#path: /etc/kubernetes/super-admin.conf#path: /etc/kubernetes/admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml\n\nmkdir -p $HOME/.kube\nsudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\nsudo chown $(id -u):$(id -g) $HOME/.kube/config\n```\n\n```sh\n# combine or replace ~/.kube/config in macOS\nlimactl cp cp-1:.kube/config kubeconfig\n```\n\n#### Join worker nodes to Kubernetes cluster\n\nrun in each worker vms\n\n```sh\n# run next line only in worker-1\nsudo kubeadm join --config manifests/kubeadm/worker-1-join-cfg.yaml\n# run next line only in worker-2\nsudo kubeadm join --config manifests/kubeadm/worker-2-join-cfg.yaml\n# run next line only in worker-3\nsudo kubeadm join --config manifests/kubeadm/worker-3-join-cfg.yaml\n```\n\n### Post Cluster Creation Steps\n\n#### Manual approval of kubelet serving certificates\n\nApprove any pending `kubelet-serving` certificate\n\n```sh\nkubectl get csr\nkubectl get csr | grep \"Pending\" | awk '{print $1}' | xargs kubectl certificate approve\n```\n\n#### Install Cilium CNI\n\nInstall Gateway API CRDs supported by Cilium.\n\n\u003chttps://docs.cilium.io/en/latest/network/servicemesh/gateway-api/gateway-api/\u003e\n\n```sh\nkubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml\nkubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml\nkubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml\nkubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml\n# there is config/crd/standard/gateway.networking.k8s.io_grpcroutes.yaml in v1.1.0, but if use it an error occurs when cilium init\nkubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_grpcroutes.yaml\nkubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml\n```\n\nInstall CNI (Cilium) with L2 LB, L7 LB (Ingress Controller) and L4/L7 LB (Gateway API) support enabled.\n\n```sh\nhelm repo add cilium https://helm.cilium.io/\nhelm repo update\nhelm install cilium cilium/cilium --version 1.15.5 --namespace kube-system --values manifests/cilium/values.yaml\n```\n\nConfigure L2 announcements and address pool for L2 aware Load Balancer\n\n```sh\nkubectl apply -f manifests/cilium/l2-lb-cfg.yaml\n```\n\nConfigure Gateway (default)\n\n```sh\nkubectl apply -f manifests/cilium/gtw-cfg.yaml\n```\n\n#### Install Metrics server add-on\n\nInstall metrics server\n\n```sh\nkubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml\n```\n\n#### Install Local path provisioner CSI\n\nInstall local path provisioner\n\n```sh\nkubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.26/deploy/local-path-storage.yaml\n```\n\nApply local path provisioner configuration. There will be a delay after `configmap` is applied before the provisioner picks it up.\n\n```sh\nkubectl apply -f manifests/local-path-provisioner/provisioner-cm.yaml\n```\n\n### Finals checks\n\nCheck from that cluster works as expected\n\n```sh\nkubectl version\ncilium status\nkubectl get --raw='/readyz?verbose'\n```\n\n# UNAVAILABLE: tart\n\nI tried to use tart instead of lima, but tart is **unavailable because it does not seem to support inter-VM connections**\n\nKeep just for the record\n\n## Versions\n\n- Tart Virtualization 2.11.0\n- Debian 12\n- Kubernetes 1.29.4\n- Cilium 1.15.4\n- Gateway API 1.1\n- kube-vip 0.8.0\n- metrics-server 0.7.1\n- local-path-provisioner 0.0.26\n\n## Prerequisites\n\n### Required Tools\n\nFollowing tools are required by this project.\n\n- [tart](https://github.com/cirruslabs/tart/)\n  - [sshpass](https://tart.run/quick-start/#ssh-access)\n- [kubernetes-cli](https://github.com/kubernetes/kubectl)\n- [helm](https://helm.sh/)\n- [git](https://git-scm.com/)\n- [cilium-cli](https://github.com/cilium/cilium-cli/)\n\n[Homebrew](https://brew.sh/) can be used to install these tools on macOS host.\n\n```sh\nbrew install tart cirruslabs/cli/sshpass kubernetes-cli helm git\n```\n\n`cilium-cli` macOS install script can be found on [Cilium Getting Started Doc](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/)\n\n```sh\nCILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)\nCLI_ARCH=amd64\nif [ \"$(uname -m)\" = \"arm64\" ]; then CLI_ARCH=arm64; fi\ncurl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}\nshasum -a 256 -c cilium-darwin-${CLI_ARCH}.tar.gz.sha256sum\nsudo tar xzvfC cilium-darwin-${CLI_ARCH}.tar.gz /usr/local/bin\nrm cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}\n```\n\n### Networking\n\n#### Kubernetes node IP range\n\n| Host     | IP address      |\n| -------- | --------------- |\n| cp       | 192.168.106.100 |\n| cp-1     | 192.168.106.101 |\n| cp-2     | 192.168.106.102 |\n| cp-3     | 192.168.106.103 |\n| worker-1 | 192.168.106.104 |\n| worker-2 | 192.168.106.105 |\n| worker-3 | 192.168.106.106 |\n\n#### Kubernetes API server\n\nKubernetes API server is available via VIP address `192.168.106.100`.\n\n#### L2 Aware load balancer, Ingress and Gateway\n\nIP address pool for `Load Balancer` services must be configured to same shared subnet than cluster `node IPs`. Currently L2 Aware LB in Cilium CNI is used and default address pool is configured is `192.168.106.240/28`.\n\nFrom the assigned address pool following IPs are \"reserved\", leaving 12 addresses available for different LB services.\n\n- `192.168.106.241` is assigned to `Ingress` (Cilium Ingress Controller) and\n- `192.168.106.242` is reserved for `Gateway` (Cilium Gateway API). `Gateway` configuration is work in progress.\n\n## Setup\n\n### Set ssh config\n\n`~/.ssh/config`\n\n`PreferredAuthentications` is required when you use 1password ssh agent\n\n```\nHost 192.168.106.* cp-* worker-*\n  User admin\n  PreferredAuthentications password\n```\n\n### Set /etc/hosts\n\n`/etc/hosts`\n\n```\n192.168.106.101 cp-1\n192.168.106.102 cp-2\n192.168.106.103 cp-3\n192.168.106.104 worker-1\n192.168.106.105 worker-2\n192.168.106.106 worker-3\n```\n\n### Create VMs\n\n```sh\ntart clone ghcr.io/cirruslabs/debian:latest cp-1\ntart clone ghcr.io/cirruslabs/debian:latest cp-2\ntart clone ghcr.io/cirruslabs/debian:latest cp-3\ntart clone ghcr.io/cirruslabs/debian:latest worker-1\ntart clone ghcr.io/cirruslabs/debian:latest worker-2\ntart clone ghcr.io/cirruslabs/debian:latest worker-3\n\n# default: cpu 4, memory 4096, disk-size 20\ntart set worker-1 --disk-size 100\ntart set worker-2 --disk-size 100\ntart set worker-3 --disk-size 100\n\ncat \u003c\u003cEOF | tee ~/.tart/vms/cp-1/config.json\n{\n  \"version\": 1,\n  \"os\": \"linux\",\n  \"arch\": \"arm64\",\n  \"cpuCount\": 2,\n  \"cpuCountMin\": 4,\n  \"memorySize\": 2147483648,\n  \"memorySizeMin\": 4294967296,\n  \"display\": {\n    \"width\": 1024,\n    \"height\": 768\n  },\n  \"macAddress\": \"10:00:00:00:00:01\"\n}\nEOF\ncat \u003c\u003cEOF | tee ~/.tart/vms/cp-2/config.json\n{\n  \"version\": 1,\n  \"os\": \"linux\",\n  \"arch\": \"arm64\",\n  \"cpuCount\": 2,\n  \"cpuCountMin\": 4,\n  \"memorySize\": 2147483648,\n  \"memorySizeMin\": 4294967296,\n  \"display\": {\n    \"width\": 1024,\n    \"height\": 768\n  },\n  \"macAddress\": \"10:00:00:00:00:02\"\n}\nEOF\ncat \u003c\u003cEOF | tee ~/.tart/vms/cp-3/config.json\n{\n  \"version\": 1,\n  \"os\": \"linux\",\n  \"arch\": \"arm64\",\n  \"cpuCount\": 2,\n  \"cpuCountMin\": 4,\n  \"memorySize\": 2147483648,\n  \"memorySizeMin\": 4294967296,\n  \"display\": {\n    \"width\": 1024,\n    \"height\": 768\n  },\n  \"macAddress\": \"10:00:00:00:00:03\"\n}\nEOF\ncat \u003c\u003cEOF | tee ~/.tart/vms/worker-1/config.json\n{\n  \"version\": 1,\n  \"os\": \"linux\",\n  \"arch\": \"arm64\",\n  \"cpuCount\": 4,\n  \"cpuCountMin\": 4,\n  \"memorySize\": 4294967296,\n  \"memorySizeMin\": 4294967296,\n  \"display\": {\n    \"width\": 1024,\n    \"height\": 768\n  },\n  \"macAddress\": \"10:00:00:00:00:04\"\n}\nEOF\ncat \u003c\u003cEOF | tee ~/.tart/vms/worker-2/config.json\n{\n  \"version\": 1,\n  \"os\": \"linux\",\n  \"arch\": \"arm64\",\n  \"cpuCount\": 4,\n  \"cpuCountMin\": 4,\n  \"memorySize\": 4294967296,\n  \"memorySizeMin\": 4294967296,\n  \"display\": {\n    \"width\": 1024,\n    \"height\": 768\n  },\n  \"macAddress\": \"10:00:00:00:00:05\"\n}\nEOF\ncat \u003c\u003cEOF | tee ~/.tart/vms/worker-3/config.json\n{\n  \"version\": 1,\n  \"os\": \"linux\",\n  \"arch\": \"arm64\",\n  \"cpuCount\": 4,\n  \"cpuCountMin\": 4,\n  \"memorySize\": 4294967296,\n  \"memorySizeMin\": 4294967296,\n  \"display\": {\n    \"width\": 1024,\n    \"height\": 768\n  },\n  \"macAddress\": \"10:00:00:00:00:06\"\n}\nEOF\n\nnohup tart run --no-graphics cp-1 \u0026\u003e/dev/null \u0026\nnohup tart run --no-graphics cp-2 \u0026\u003e/dev/null \u0026\nnohup tart run --no-graphics cp-3 \u0026\u003e/dev/null \u0026\nnohup tart run --no-graphics worker-1 \u0026\u003e/dev/null \u0026\nnohup tart run --no-graphics worker-2 \u0026\u003e/dev/null \u0026\nnohup tart run --no-graphics worker-3 \u0026\u003e/dev/null \u0026\n\n# ssh with initial ip\nsshpass -p admin ssh $(tart ip cp-1)\nsshpass -p admin ssh $(tart ip cp-2)\nsshpass -p admin ssh $(tart ip cp-3)\nsshpass -p admin ssh $(tart ip worker-1)\nsshpass -p admin ssh $(tart ip worker-2)\nsshpass -p admin ssh $(tart ip worker-3)\n```\n\nrun in each vm\n\n```sh\nsudo apt update \u0026\u0026 sudo apt upgrade -y \u0026\u0026 sudo apt install -y locales-all\n\n# install kubeadm, kubelet, kubectl\n# https://v1-29.docs.kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/\nsudo apt install -y containerd apt-transport-https gpg\ncurl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg\necho 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list\nsudo apt update\nsudo apt install -y kubelet kubeadm kubectl\nsudo apt-mark hold kubelet kubeadm kubectl\nsudo systemctl enable --now kubelet\n\necho \"network: {config: disabled}\" | sudo tee /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg\nsudo rm /etc/netplan/50-cloud-init.yaml\n\n# use network.ethernets.enp0s1.addresses[0] to match ip table above\ncat \u003c\u003cEOF | sudo tee /etc/netplan/60-static-ip.yaml\nnetwork:\n  version: 2\n  ethernets:\n    enp0s1:\n      dhcp4: false\n      addresses:\n        - 192.168.106.101/24\n      routes:\n        - to: default\n          via: 192.168.106.1\n      nameservers:\n        addresses:\n          - 192.168.106.1\nEOF\ncat \u003c\u003cEOF | sudo tee /etc/netplan/60-static-ip.yaml\nnetwork:\n  version: 2\n  ethernets:\n    enp0s1:\n      dhcp4: false\n      addresses:\n        - 192.168.106.102/24\n      routes:\n        - to: default\n          via: 192.168.106.1\n      nameservers:\n        addresses:\n          - 192.168.106.1\nEOF\ncat \u003c\u003cEOF | sudo tee /etc/netplan/60-static-ip.yaml\nnetwork:\n  version: 2\n  ethernets:\n    enp0s1:\n      dhcp4: false\n      addresses:\n        - 192.168.106.103/24\n      routes:\n        - to: default\n          via: 192.168.106.1\n      nameservers:\n        addresses:\n          - 192.168.106.1\nEOF\ncat \u003c\u003cEOF | sudo tee /etc/netplan/60-static-ip.yaml\nnetwork:\n  version: 2\n  ethernets:\n    enp0s1:\n      dhcp4: false\n      addresses:\n        - 192.168.106.104/24\n      routes:\n        - to: default\n          via: 192.168.106.1\n      nameservers:\n        addresses:\n          - 192.168.106.1\nEOF\ncat \u003c\u003cEOF | sudo tee /etc/netplan/60-static-ip.yaml\nnetwork:\n  version: 2\n  ethernets:\n    enp0s1:\n      dhcp4: false\n      addresses:\n        - 192.168.106.105/24\n      routes:\n        - to: default\n          via: 192.168.106.1\n      nameservers:\n        addresses:\n          - 192.168.106.1\nEOF\ncat \u003c\u003cEOF | sudo tee /etc/netplan/60-static-ip.yaml\nnetwork:\n  version: 2\n  ethernets:\n    enp0s1:\n      dhcp4: false\n      addresses:\n        - 192.168.106.106/24\n      routes:\n        - to: default\n          via: 192.168.106.1\n      nameservers:\n        addresses:\n          - 192.168.106.1\nEOF\n\nsudo chmod 600 /etc/netplan/60-static-ip.yaml\nsudo sync # if you stop vm without run this, file changes may not applied\n\ncat \u003c\u003cEOF | sudo tee -a /etc/hosts\n192.168.106.101 cp-1\n192.168.106.102 cp-2\n192.168.106.103 cp-3\n192.168.106.104 worker-1\n192.168.106.105 worker-2\n192.168.106.106 worker-3\nEOF\n\ncat \u003c\u003cEOF | sudo tee /etc/modules-load.d/k8s.conf\noverlay\nbr_netfilter\nEOF\nsudo modprobe overlay\nsudo modprobe br_netfilter\n\ncat \u003c\u003cEOF | sudo tee /etc/sysctl.d/kubernetes.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nEOF\nsudo sysctl --system\n\nsudo mkdir -p /etc/containerd\ncontainerd config default | sudo tee /etc/containerd/config.toml\nsudo sed -i 's#SystemdCgroup = false#SystemdCgroup = true#' /etc/containerd/config.toml\nsudo sed -i 's#sandbox_image = \"registry.k8s.io/pause:3.8\"#sandbox_image = \"registry.k8s.io/pause:3.9\"#' /etc/containerd/config.toml\nsudo systemctl restart containerd\n```\n\n```sh\ntart stop cp-1\ntart stop cp-2\ntart stop cp-3\ntart stop worker-1\ntart stop worker-2\ntart stop worker-3\n\n# start vm with dir mount\nnohup tart run --no-graphics --dir=\"~/path/to/k8s-on-macos\" cp-1 \u0026\u003e/dev/null \u0026\nnohup tart run --no-graphics --dir=\"~/path/to/k8s-on-macos\" cp-2 \u0026\u003e/dev/null \u0026\nnohup tart run --no-graphics --dir=\"~/path/to/k8s-on-macos\" cp-3 \u0026\u003e/dev/null \u0026\nnohup tart run --no-graphics --dir=\"~/path/to/k8s-on-macos\" worker-1 \u0026\u003e/dev/null \u0026\nnohup tart run --no-graphics --dir=\"~/path/to/k8s-on-macos\" worker-2 \u0026\u003e/dev/null \u0026\nnohup tart run --no-graphics --dir=\"~/path/to/k8s-on-macos\" worker-3 \u0026\u003e/dev/null \u0026\n\n# now you can ssh with static ip\nsshpass -p admin ssh cp-1\nsshpass -p admin ssh cp-2\nsshpass -p admin ssh cp-3\nsshpass -p admin ssh worker-1\nsshpass -p admin ssh worker-2\nsshpass -p admin ssh worker-3\n\n# --- run in each vm\nip addr # check ip\n# mount host ~/dev/k8s to init k8s cluster\nsudo mkdir /mnt/shared\nsudo mount -t virtiofs com.apple.virtio-fs.automount /mnt/shared # if you reboot vm, you shoud re-run this\n# --- run in each vm\n```\n\n### Initiate Kubernetes Cluster\n\n#### Initiate Kubernetes Control Planes\n\n```sh\n# run in cp\nexport KVVERSION=v0.8.0\nexport INTERFACE=enp0s1\nexport VIP=192.168.106.100\nsudo mkdir -p /etc/kubernetes/manifests\nsudo ctr image pull ghcr.io/kube-vip/kube-vip:$KVVERSION\nsudo ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip manifest pod \\\n    --arp \\\n    --controlplane \\\n    --address $VIP \\\n    --interface $INTERFACE \\\n    --enableLoadBalancer \\\n    --leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml\n\n# Kubernetes \u003e=1.29 kube-vip workaround https://github.com/kube-vip/kube-vip/issues/684#issuecomment-1864855405\nsudo sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml\n\n# --- NOT WORKING ---\n# --- run in cp-1 only\nsudo kubeadm init --upload-certs --config /mnt/shared/manifests/kubeadm/cp-1-init-cfg.yaml\n# --- run in cp-1 only\n# --- run in cp-2 only\nsudo kubeadm join --config /mnt/shared/manifests/kubeadm/cp-2-join-cfg.yaml\n# --- run in cp-2 only\n# --- run in cp-3 only\nsudo kubeadm join --config /mnt/shared/manifests/kubeadm/cp-3-join-cfg.yaml\n# --- run in cp-3 only\n```\n\n# minikube\n\nSingle VM to Docker Runtime (colima), minikube with docker\n\nPros over colima\n\n- Multiple worker node\n- Various kubernetes versions\n\nTo use minikube docker driver, docker should be configured.\n\n```sh\ncolima start -c 4 -m 4\n# -c 4: VM with 4 CPUs (default 2)\n# -m 4: VM with 4 GiB Memory (default 2)\n```\n\ndocker context will be automatically configured\n\n```sh\nmk start -n 2 --kubernetes-version 1.30\n# -n 2: 2 nodes (Single CP/Worker and n-1 Workers)\n```\n\n`~/.kube/config` will be automatically modified\n\n# colima\n\nI tried to use colima single node kubernetes cluster base on k3s for daily work but with basic setup, kubectl to colima kubernetes cluster while lima vms running occurs tls error:\n\n```\nUnable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority\n```\n\nSo now I use minikube instead.\n\nPros over minikube\n\n- create/start/stop/delete is simpler and faster\n- LoadBalancer Service is accessible with localhost without `minikube tunnel`\n\n## Setup\n\n```sh\ncolima start -k -c 4 -m 4\n# -k: start single node (k3s in docker) kubernetes cluster\n# -c 4: VM with 4 CPUs (default 2)\n# -m 4: VM with 4 GiB Memory (default 2)\n```\n\ndocker context and `~/.kube/config` will be automatically modified\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevsunb%2Fk8s-on-macos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevsunb%2Fk8s-on-macos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevsunb%2Fk8s-on-macos/lists"}