{"id":20360012,"url":"https://github.com/fred78290/autoscaled-masterkube-vmware","last_synced_at":"2025-07-23T16:11:00.969Z","repository":{"id":65557354,"uuid":"426573166","full_name":"Fred78290/autoscaled-masterkube-vmware","owner":"Fred78290","description":"This projet is an helper to create a kubernetes cluster from scratch with kubeadm on vsphere","archived":false,"fork":false,"pushed_at":"2024-03-05T10:14:42.000Z","size":339,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-22T00:34:56.878Z","etag":null,"topics":["autoscaler","autoscaling","containerd","cri-o","kubeadm","kubernetes","vsphere"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Fred78290.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-11-10T10:10:27.000Z","updated_at":"2024-08-11T15:45:08.000Z","dependencies_parsed_at":"2023-12-20T15:45:28.232Z","dependency_job_id":null,"html_url":"https://github.com/Fred78290/autoscaled-masterkube-vmware","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fred78290%2Fautoscaled-masterkube-vmware","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fred78290%2Fautoscaled-masterkube-vmware/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fred78290%2Fautoscaled-masterkube-vmware/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fred78290%2Fautoscaled-masterkube-vmware/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Fred78290","download_url":"https://codeload.github.com/Fred78290/autoscaled-masterkube-vmware/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248513136,"owners_count":21116764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoscaler","autoscaling","containerd","cri-o","kubeadm","kubernetes","vsphere"],"created_at":"2024-11-14T23:37:10.107Z","updated_at":"2025-04-12T03:33:26.329Z","avatar_url":"https://github.com/Fred78290.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Licence](https://img.shields.io/badge/License-GPL_v2-blue.svg)](https://github.com/Fred78290/autoscaled-masterkube-vmware/blob/master/LICENSE)\n\n# Introduction\n\nThis directory contains everthing to create a single autoscaled cluster or HA cluster with worker node on vSphere infrastructure using **kubeadm** or **k3s**.\n\n## Prerequistes\n\nEnsure that you have sudo right\n\nYou must also install\n\n|**Linux**|**MacOS**|\n| --- | --- |\n|kubectl|kubectl|\n|govc|govc|\n|cfssl|cfssl|\n|jq|jq|\n|yq|yq|\n||gnu-getopt|\n||gsed|\n||gbase64|\n\nFor MacOS it's recommanded to install **homebrew** and install **Prerequistes** from **homebrew**\n\n## Create the masterkube\n\nFirst step is to fill a file named **govc.defs** in the bin directory with the values needed by govc tool\n\n```\nexport GOVC_DATACENTER=\nexport GOVC_DATASTORE=\nexport GOVC_FOLDER=\nexport GOVC_FOLDER=\nexport GOVC_HOST=\nexport GOVC_INSECURE=\"1\"\nexport GOVC_NETWORK=\nexport GOVC_USERNAME=\nexport GOVC_PASSWORD=\nexport GOVC_RESOURCE_POOL=\nexport GOVC_RESOURCE_POOL=\nexport GOVC_URL=\nexport GOVC_VIM_VERSION=\"6.0\"\n\n# If you use cert-manager with a public domain\nexport CERT_EMAIL=\nexport PUBLIC_DOMAIN_NAME=\n\n# If your public domain is hosted on route53 for cert-manager\nexport AWS_ROUTE53_PUBLIC_ZONE_ID=\nexport AWS_ROUTE53_ACCESSKEY=\nexport AWS_ROUTE53_SECRETKEY=\n\n```\n\nThe simply way to create the masterkube is to run [create-masterkube.sh](create-masterkube.sh)\n\nSome needed file are located in:\n\n| Name | Description |\n| --- | --- |\n| `bin` | Essentials scripts to build the master kubernetes node  |\n| `etc/ssl`  | Your CERT for https. Autosigned will be generated if empty  |\n| `template`  | Templates files to deploy pod \u0026 service |\n\nThe first thing done by this script is to create a VM Template Ubuntu-20.04 image containing kubernetes binaries and a container runtime of you choice (docker/containerd/cri-o) with cni plugin (calico/flannel/weave/...). The VM template will be named by default focal-kubernetes-cni-(cni plugin)-(kuberneres version)-(container runtime)-(architecture)\n\nas example: focal-kubernetes-cni-flannel-v1.23.1-containerd-amd64\n\nNext step will be to launch a cloned VM and create a master node. It will also deploy a dashboard at the URL https://masterkube-vmware-dashboard.@your-domain@/\n\nTo connect to the dashboard, copy paste the token from file [cluster/vmware-ca-k8s/dashboard-token](./cluster/vmware-ca-k8s/dashboard-token)\n\nNext step is to deploy a replicaset helloworld. This replicaset use hostnetwork:true to enforce one pod per node.\n\nDuring the process the script will create many files located in\n\n| Name | Description |\n| --- | --- |\n| `cluster/vmware-ca-k8s` | Essentials file to connect to kubernetes with kubeadm join  |\n| `config/vmware-ca-k8s`  | Configuration file generated during the build process  |\n\n**The cluster kubernetes will use metallb as load balancer for services declared LoadBalancer.**\n\n## Command line arguments\n\n| Parameter | Description | Default |\n| --- | --- |--- |\n| `-h\\|--help` | Help  | |\n| `-v\\|--verbose` | Verbose mode  | |\n| `-x\\|--trace` | Trace execution  | |\n| `-r\\|--resume` | Allow to resume interrupted creation of cluster kubernetes  | |\n| `--delete` | Delete cluster and exit  | |\n| `--distribution=\u003cvalue\u003e` | Ubuntu distribution to use ${DISTRO}  | |\n| `--create-image-only`| Create image only and exit ||\n| `--upgrade`| Upgrade existing cluster to upper version of kubernetes ||\n| **Flags to set some location informations** |\n| `--configuration-location=\u003cpath\u003e` | Specify where configuration will be stored | current directory |\n| `--ssl-location=\u003cpath\u003e` | Specify where the etc/ssl dir is stored | ./etc/ssl|\n| `--govc-defs=\u003cpath\u003e` | Override the GOVC definitions  | bin/govc.defs |\n| **Design domain** |\n| `--public-domain=\u003cvalue\u003e` | Specify the public domain to use  | |\n| `--dashboard-hostname=\u003cvalue\u003e` | Specify the hostname for kubernetes dashboard |masterkube-vmware-dashboard|\n| **CERT manager** |\n| `--cert-email=\u003cvalue\u003e` | Specify the mail for lets encrypt  ||\n| `--use-zerossl` | Specify cert-manager to use zerossl.| YES |\n| `--dont-use-zerossl` | Specify cert-manager to use letsencrypt.||\n| `--zerossl-eab-kid=\u003cvalue\u003e` | Specify zerossl eab kid. ||\n| `--zerossl-eab-hmac-secret=\u003cvalue\u003e` | Specify zerossl eab hmac secret. ||\n| `--godaddy-key=\u003cvalue\u003e` | Specify godaddy api key. ||\n| `--godaddy-secret=\u003cvalue\u003e` | Specify godaddy api secret. ||\n| **Route53** |\n| `--route53-zone-id=\u003cvalue\u003e` | Specify the route53 zone id. ||\n| `--route53-access-key=\u003cvalue\u003e` | Specify the route53 aws access key. ||\n| `--route53-secret-key=\u003cvalue\u003e` | Specify the route53 aws secret key. ||\n| **Flag to design the kubernetes cluster** |\n| `--k8s-distribution=[kubeadm, k3s, rke2]` | Which kubernetes distribution to use: kubeadm,k3s,rke2 | kubeadm |\n| `-c\\|--ha-cluster` | Allow to create an HA cluster with 3 control planes | NO |\n| `--worker-nodes=\u003cvalue\u003e` | Specify the number of worker node created in the cluster. | 3 |\n| `--container-runtime=\u003cvalue\u003e` | Specify which OCI runtime to use. [**docker**\\|**containerd**\\|**cri-o**]| containerd |\n| `--max-pods=\u003cvalue\u003e` | Specify the max pods per created VM. | 110 |\n| `-d\\|--default-machine=\u003cvalue\u003e` | Override machine type used for auto scaling | medium |\n| `-k\\|--ssh-private-key=\u003cvalue\u003e`  | Alternate ssh key file |~/.ssh/id_rsa|\n| `-t\\|--transport=\u003cvalue\u003e`  | Override the transport to be used between autoscaler and vmware-autoscaler [**tcp**\\|**linux**] |linux|\n| `--node-group=\u003cvalue\u003e`  | Override the node group name |vmware-ca-k8s|\n| `--cni-plugin=\u003cvalue\u003e`  | Override CNI plugin [**calico**\\|**flannel**\\|**weave**\\|**romana**]|flannel|\n| `-n\\|--cni-version=\u003cvalue\u003e`  | Override CNI plugin version |v1.1.1|\n| `-k\\|--kubernetes-version=\u003cvalue\u003e`  |Which version of kubernetes to use |latest|\n| **Flags in ha mode only** |\n| `-e\\|--create-external-etcd` | Allow to create and use an external HA etcd cluster  | NO |\n| `-u\\|--use-keepalived` | Allow to use keepalived as load balancer else NGINX is used | NGINX |\n| **Flags to set the template vm** |\n| `--target-image=\u003cvalue\u003e` | The VM name created for cloning with kubernetes | focal-kubernetes |\n| `--seed-image=\u003cvalue\u003e` | The VM name used to created the targer image | focal-server-cloudimg-seed |\n| `--seed-user=\u003cvalue\u003e` | The cloud-init user name | ubuntu |\n| `-p\\|--password=\u003cvalue\u003e`  |Define the kubernetes user password |randomized|\n| **Flags to set the template vm** |\n| `--public-address=\u003cvalue\u003e` | The public address to expose kubernetes endpoint [**DHCP**\\|**1.2.3.4**] | DHCP |\n| `--no-dhcp-autoscaled-node=\u003cvalue\u003e` | Autoscaled node don't use DHCP | DHCP |\n| `--vm-private-network=\u003cvalue\u003e` | Override the name of the private network in vsphere | 'Private Network' |\n| `--vm-public-network=\u003cvalue\u003e` | Override the name of the public network in vsphere | 'Public Network' |\n| `--net-address=\u003cvalue\u003e` | Override the IP of the kubernetes control plane node | 192.168.1.20 |\n| `--net-gateway=\u003cvalue\u003e` | The public IP gateway | 10.0.0.1 |\n| `--net-dns=\u003cvalue\u003e` | The public IP dns | 10.0.0.1 |\n| `--net-domain=\u003cvalue\u003e` | The local domain name | example.com |\n| `--metallb-ip-range=\u003cvalue\u003e` | Override the metalb ip range | 10.0.0.100-10.0.0.127 |\n| `--dont-use-dhcp-routes-private` | Tell if we don't use DHCP routes in private network|\n| `--dont-use-dhcp-routes-public` | Tell if we don't use DHCP routes in public network|\n| `--add-route-private=\u003cvalue\u003e` | Add route to private network syntax is --add-route-private=to=X.X.X.X/YY,via=X.X.X.X,metric=100 --add-route-private=to=Y.Y.Y.Y/ZZ,via=X.X.X.X,metric=100|\n| `--add-route-public=\u003cvalue\u003e` | Add route to public network syntax is --add-route-public=to=X.X.X.X/YY,via=X.X.X.X,metric=100 --add-route-public=to=Y.Y.Y.Y/ZZ,via=X.X.X.X,metric=100|\n| **Flags for autoscaler** |\n| `--cloudprovider=\u003cvalue\u003e` | autoscaler flag \u003cgrpc|externalgrpc\u003e.|externalgrpc|\n| `--max-nodes-total=\u003cvalue\u003e` | Maximum number of nodes in all node groups. Cluster autoscaler will not grow the cluster beyond this number. | 9 |\n| `--cores-total=\u003cvalue\u003e` | Minimum and maximum number of cores in cluster, in the format \u003c min \u003e:\u003c max \u003e. Cluster autoscaler will not scale the cluster beyond these numbers. | 0:16 |\n| `--memory-total=\u003cvalue\u003e` | Minimum and maximum number of gigabytes of memory in cluster, in the format \u003c min \u003e:\u003c max \u003e. Cluster autoscaler will not scale the cluster beyond these numbers. | 0:48 |\n| `--max-autoprovisioned-node-group-count=\u003cvalue\u003e` | The maximum number of autoprovisioned groups in the cluster | 1 |\n| `--scale-down-enabled=\u003cvalue\u003e` | Should CA scale down the cluster | true |\n| `--scale-down-delay-after-add=\u003cvalue\u003e` | How long after scale up that scale down evaluation resumes | 1 minutes |\n| `--scale-down-delay-after-delete=\u003cvalue\u003e` | How long after node deletion that scale down evaluation resumes, defaults to scan-interval | 1 minutes |\n| `--scale-down-delay-after-failure=\u003cvalue\u003e` | How long after scale down failure that scale down evaluation resumes | 1 minutes |\n| `--scale-down-unneeded-time=\u003cvalue\u003e` | How long a node should be unneeded before it is eligible for scale down | 1 minutes |\n| `--scale-down-unready-time=\u003cvalue\u003e` | How long an unready node should be unneeded before it is eligible for scale down | 1 minutes |\n| `--unremovable-node-recheck-timeout=\u003cvalue\u003e` | The timeout before we check again a node that couldn't be removed before | 1 minutes |\n\n```bash\ncreate-masterkube.sh \\\n    --verbose \\\n    --ha-cluster \\\n    --nodegroup=\u003cMy Group Name\u003e \\\n    --target-image=\u003cMy VM template Name\u003e \\\n    --seed-image=\u003cMy custom VM Template\u003e \\\n    --seed-user=\u003cMy custom user\u003e \\\n    --vm-private-network=\u003cMy private network\u003e \\\n    --vm-public-network=\u003cMy public network\u003e \\\n    --net-address=\"10.0.4.200\" \\\n    --net-gateway=\"10.0.4.1\" \\\n    --net-dns=\"10.0.4.1\" \\\n    --net-domain=\"acme.com\"\n```\n\n## Raise autoscaling\n\nTo scale up or down the cluster, just play with `kubectl scale`\n\nTo scale fresh masterkube `kubectl scale --replicas=2 deploy/helloworld -n kube-public`\n\n## Delete master kube and worker nodes\n\nTo delete the master kube and associated worker nodes, just run the command [delete-masterkube.sh](./bin/delete-masterkube.sh).\nIf the create process fail for any reason, you can use flag **--force**\n\n## Upgrade existing cluster\n\nTo upgrade cluster with an upper version of kubernetes, just run `create-masterkube.sh --upgrade --kubernetes-version=\u003cupper version\u003e`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffred78290%2Fautoscaled-masterkube-vmware","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffred78290%2Fautoscaled-masterkube-vmware","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffred78290%2Fautoscaled-masterkube-vmware/lists"}