{"id":15750815,"url":"https://github.com/rgl/terraform-rke-vsphere-cloud-provider-example","last_synced_at":"2026-05-05T07:32:51.951Z","repository":{"id":139753347,"uuid":"394414188","full_name":"rgl/terraform-rke-vsphere-cloud-provider-example","owner":"rgl","description":"Dynamically provision Persistent Volumes using the vSphere Cloud Provider in a RKE cluster","archived":false,"fork":false,"pushed_at":"2021-08-13T19:30:38.000Z","size":11,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-31T07:22:31.115Z","etag":null,"topics":["k8s","kubernetes","persistent-storage","persistent-volume","rke","terraform","vsphere"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rgl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-09T19:26:39.000Z","updated_at":"2021-08-15T14:21:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"5b799d4b-14d9-4807-8a98-a99a0b7b03f3","html_url":"https://github.com/rgl/terraform-rke-vsphere-cloud-provider-example","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rgl/terraform-rke-vsphere-cloud-provider-example","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rgl%2Fterraform-rke-vsphere-cloud-provider-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rgl%2Fterraform-rke-vsphere-cloud-provider-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rgl%2Fterraform-rke-vsphere-cloud-provider-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rgl%2Fterraform-rke-vsphere-cloud-provider-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rgl","download_url":"https://codeload.github.com/rgl/terraform-rke-vsphere-cloud-provider-example/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rgl%2Fterraform-rke-vsphere-cloud-provider-example/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32640533,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-04T10:08:07.713Z","status":"online","status_checked_at":"2026-05-05T02:00:06.033Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["k8s","kubernetes","persistent-storage","persistent-volume","rke","terraform","vsphere"],"created_at":"2024-10-04T07:00:18.766Z","updated_at":"2026-05-05T07:32:51.935Z","avatar_url":"https://github.com/rgl.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About\n\nThis shows how to dynamically provision Persistent Volumes using the [vSphere Cloud Provider](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/), [terraform](https://www.terraform.io/), and [the rke provider](https://registry.terraform.io/providers/rancher/rke) in a single-node Kubernetes instance.\n\nThe Persistent Volume will be dynamically created from a Persistent Volume Claim inside a vSphere Datastore.\n\nThe Persistent Volume will be dynamically (de)attached to the Virtual Machine as a SCSI disk.\n\n**NB** There's a big caveat with terraform and vSphere Persistent Volumes: PVs are attached as a Virtual Machine Disks and when you try to use `terraform plan` again, these will appear as being modified outside of the terraform control; `terraform` is therefore [configured to ignore disks changes](https://github.com/hashicorp/terraform-provider-vsphere/issues/1028) to prevent it from modifying the VM configuration.\n\n**NB** This uses the deprecated vSphere Cloud Provider driver (that's what RKE uses out-of-the-box). Newer installations should probably use the [vSphere CSI Driver](https://vsphere-csi-driver.sigs.k8s.io/).\n\n**NB** This uses a VMFS Datastore. It does not uses vSAN.\n\n## Usage (Ubuntu 20.04 host)\n\nInstall the [Ubuntu 20.04 VM template](https://github.com/rgl/ubuntu-vagrant).\n\nInstall `terraform`, `govc` and `kubectl`:\n\n```bash\n# install terraform.\nwget https://releases.hashicorp.com/terraform/1.0.4/terraform_1.0.4_linux_amd64.zip\nunzip terraform_1.0.4_linux_amd64.zip\nsudo install terraform /usr/local/bin\nrm terraform terraform_*_linux_amd64.zip\n# install govc.\nwget https://github.com/vmware/govmomi/releases/download/v0.26.0/govc_Linux_x86_64.tar.gz\ntar xf govc_Linux_x86_64.tar.gz govc\nsudo install govc /usr/local/bin/govc\nrm govc govc_Linux_x86_64.tar.gz\n# install kubectl.\nkubectl_version='1.20.8'\nwget -qO /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg\necho 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list \u003e/dev/null\nsudo apt-get update\nkubectl_package_version=\"$(apt-cache madison kubectl | awk \"/$kubectl_version-/{print \\$3}\")\"\nsudo apt-get install -y \"kubectl=$kubectl_package_version\"\n```\n\nSave your environment details as a script that sets the terraform variables from environment variables, e.g.:\n\n```bash\ncat \u003esecrets.sh \u003c\u003c'EOF'\nexport TF_VAR_vsphere_user='administrator@vsphere.local'\nexport TF_VAR_vsphere_password='password'\nexport TF_VAR_vsphere_server='vsphere.local'\nexport TF_VAR_vsphere_datacenter='Datacenter'\nexport TF_VAR_vsphere_compute_cluster='Cluster'\nexport TF_VAR_vsphere_datastore='Datastore'\nexport TF_VAR_vsphere_network='VM Network'\nexport TF_VAR_prefix='rke_example'\nexport TF_VAR_vsphere_folder=\"examples/$TF_VAR_prefix\"\nexport TF_VAR_vsphere_ubuntu_template='vagrant-templates/ubuntu-20.04-amd64-vsphere'\nexport TF_VAR_controller_count='1'\nexport TF_VAR_worker_count='1'\nexport GOVC_INSECURE='1'\nexport GOVC_URL=\"https://$TF_VAR_vsphere_server/sdk\"\nexport GOVC_USERNAME=\"$TF_VAR_vsphere_user\"\nexport GOVC_PASSWORD=\"$TF_VAR_vsphere_password\"\nEOF\n```\n\n**NB** You could also add these variables definitions into the `terraform.tfvars` file, but I find the environment variables more versatile as they can also be used from other tools, like govc.\n\nLaunch this example:\n\n```bash\nsource secrets.sh\n# see https://github.com/vmware/govmomi/blob/master/govc/USAGE.md\ngovc version\ngovc about\ngovc datacenter.info # list datacenters\ngovc find # find all managed objects\nrm -f *.log kubeconfig.yaml\nterraform init\nterraform plan -out=tfplan\ntime terraform apply tfplan\n# do another plan and you should verify that there are no changes.\nterraform plan -out=tfplan\n```\n\nTest accessing the cluster:\n\n```bash\nterraform output --raw rke_state \u003erke_state.json # might be useful for troubleshooting.\nterraform output --raw kubeconfig \u003ekubeconfig.yaml\nexport KUBECONFIG=$PWD/kubeconfig.yaml\nkubectl get nodes -o wide\n```\n\nTest creating a persistent workload:\n\n```bash\n# create a test StorageClass.\nkubectl apply -f - \u003c\u003cEOF\napiVersion: storage.k8s.io/v1\nkind: StorageClass\nmetadata:\n  name: test\nprovisioner: kubernetes.io/vsphere-volume\nparameters:\n  datastore: $TF_VAR_vsphere_datastore\n  diskformat: thin\n  fstype: ext4\nEOF\nkubectl get storageclass\n# create a test PersistentVolumeClaim.\n# NB the vSphere Cloud Provider will create a folder named \"kubevols\"\n#    inside the vSphere datastore. the actual k8s volumes .vmdk will be\n#    stored as a, e.g., kubernetes-dynamic-pvc-5ed4b014-7db0-425e-97d4-8ff8dd0cd0e1.vmdk file.\nkubectl apply -f - \u003c\u003c'EOF'\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: test\nspec:\n  storageClassName: test\n  accessModes:\n    - ReadWriteOnce\n  resources:\n    requests:\n      storage: 5Gi\nEOF\n# show the PVC status.\n# NB this must not show any error or pending state.\nkubectl describe pvc\n# show the PVC status.\n# NB this must not show any error or pending state.\n# NB it should look something alike:\n#       NAME   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE\n#       test   Bound    pvc-5ed4b014-7db0-425e-97d4-8ff8dd0cd0e1   5Gi        RWO            test           16m\nkubectl get pvc\n# show the corresponding PV (created automatically from the PVC).\nkubectl get pv\n# create the test Pod that uses the test PersistentVolumeClaim created PV.\n# NB this will trigger the attachment of the PV volume as a new VM Hard Disk.\ncat \u003etest-pod.yaml \u003c\u003c'EOF'\napiVersion: v1\nkind: Pod\nmetadata:\n  name: test\nspec:\n  containers:\n    - name: web\n      image: nginx\n      ports:\n        - name: web\n          containerPort: 80\n      volumeMounts:\n        - mountPath: /usr/share/nginx/html\n          name: web\n  volumes:\n    - name: web\n      persistentVolumeClaim:\n        claimName: test\nEOF\nkubectl apply -f test-pod.yaml\n# see the pod status and wait for it be Running.\nkubectl get pods -o wide\n# show the current VM disks. you should see the sdb disk device and the\n# corresponding disk UUID. sdb is the backing device of the pod volume.\nssh \"vagrant@$(kubectl get pods test -o json | jq -r .status.hostIP)\" \\\n  -- lsblk -o KNAME,SIZE,TRAN,FSTYPE,UUID,LABEL,MODEL,SERIAL\n# enter the pod and check the mount volume.\nkubectl exec -it test -- /bin/bash\n# show the mounts. you should see sdb mounted at /usr/share/nginx/html.\nmount | grep nginx\n# create the index.html file.\ncat \u003e/usr/share/nginx/html/index.html \u003c\u003c'EOF'\nThis is served from a Persistent Volume!\nEOF\n# check whether nginx is returning the expected html.\ncurl localhost\n# exit the pod.\nexit\n# delete the test pod.\n# NB this will trigger the removal of the PV volume from the VM.\nkubectl delete pod/test\n# list the PVs and check that the pv was not deleted.\nkubectl get pv\n# create a new pod instance.\nkubectl apply -f test-pod.yaml\n# see the pod status and wait for it be Running.\nkubectl get pods -o wide\n# check whether nginx is returning the expected html.\nkubectl exec -it test -- curl localhost\n```\n\nDestroy everything:\n\n```bash\ntime terraform destroy --auto-approve\n```\n\n**NB** This will not delete the VMDKs that were used to store the Persistent Volumes. You have to manually delete them from the datastore `kubevols` folder (which is a PITA), or before `terraform destroy` manually delete the PVC with `kubectl delete pvc/test`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frgl%2Fterraform-rke-vsphere-cloud-provider-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frgl%2Fterraform-rke-vsphere-cloud-provider-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frgl%2Fterraform-rke-vsphere-cloud-provider-example/lists"}