{"id":35114020,"url":"https://github.com/ociotec/pve-k8s-talos","last_synced_at":"2026-01-13T22:54:06.112Z","repository":{"id":330858777,"uuid":"1123707396","full_name":"ociotec/pve-k8s-talos","owner":"ociotec","description":"Proxmox VE k8s Talos cluster receipt based on OpenTofu","archived":false,"fork":false,"pushed_at":"2026-01-11T22:28:47.000Z","size":331,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-13T22:53:54.959Z","etag":null,"topics":["ceph","ceph-dashboard","ceph-rbd","cephfs","grafana","grafana-loki","k8s","k8s-cluster","k8s-deployment","opentofu","prometheus","promtail","proxmox","proxmox-ve","pve","rook-ceph","shell-script","talos"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ociotec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-27T12:56:44.000Z","updated_at":"2026-01-11T22:07:53.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ociotec/pve-k8s-talos","commit_stats":null,"previous_names":["ociotec/pve-k8s-talos"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/ociotec/pve-k8s-talos","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ociotec%2Fpve-k8s-talos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ociotec%2Fpve-k8s-talos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ociotec%2Fpve-k8s-talos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ociotec%2Fpve-k8s-talos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ociotec","download_url":"https://codeload.github.com/ociotec/pve-k8s-talos/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ociotec%2Fpve-k8s-talos/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28405135,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-13T21:51:37.118Z","status":"ssl_error","status_checked_at":"2026-01-13T21:45:14.585Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ceph","ceph-dashboard","ceph-rbd","cephfs","grafana","grafana-loki","k8s","k8s-cluster","k8s-deployment","opentofu","prometheus","promtail","proxmox","proxmox-ve","pve","rook-ceph","shell-script","talos"],"created_at":"2025-12-27T20:23:16.640Z","updated_at":"2026-01-13T22:54:06.106Z","avatar_url":"https://github.com/ociotec.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Proxmox VE k8s Talos\n\nAutomatic deployment of a k8s Talos cluster on Promox VE virtual machines with OpenTofu to create VMs infra \u0026 Talos API to create k8s cluster.\n\n## Infrastructure\n\n### Install requirements\n\nFirst install OpenTofu \u0026 `talosctl`, for MacOS:\n\n```bash\nbrew install opentofu siderolabs/tap/talosctl\n```\n\nOptionally install `direnv` to auto sets the environment variables entering one directory if that directory has a file `.envrc`:\n\n```bash\nbrew install direnv\n```\n\nThen install defined dependencies in `infra\u0026/main.tf` file with:\n\n```bash\ntofu init -upgrade\n```\n\n### Customize your setup\n\nNow you need to update several files to your current needs. Samples of the files are provided for reference, just rename them removing the `.sample` from them.\n\n- `.envrc.sample` --\u003e `.envrc`\n  - This is an optional file, only proceed with this file creation if you also installed previous optional step `direnv`.\n  - Define all required PVE environment variables to allow OpenTofu to access your PVE nodes, it's prererred to use API token authentication as described at sample file.\n- `vms_constants.tf.sample` --\u003e `vms_constants.tf`\n  - Talos ISO path on PVE node.\n  - Optional datastore ID for VM disks and cloud-init (defaults to `local-lvm`).\n  - Network settings (except IP address that is configured later).\n  - DNS servers (comma-separated list, at least one required).\n  - Optional VLAN tag for all VMs (leave empty to disable).\n  - Optional NTP servers (comma-separated list, leave empty to disable).\n  - Talos version and factory image ID (used to render `patches/qemu.yaml`).\n- `vms_list.tf.sample` --\u003e `vms_list.tf`\n  - Map of VMs on PVE with VM name as key:\n    - PVE node.\n    - VM ID.\n    - Type of node: `controlplane` or `worker` node.\n    - IP address.\n- `vms_resources.tf.sample` --\u003e `vms_resources.tf`\n  - Reources for control plane \u0026 worker nodes:\n    - Count of vCPUs.\n    - RAM memory in MB.\n    - Disk sizes in GB (several could be specified, first is used for root disk).\n- `k8s-net/constants.tf.sample` --\u003e `k8s-net/constants.tf`\n  - Domain, CA organization, MetalLB pool range, and ingress fixed IP.\n- `monitoring/constants.tf.sample` --\u003e `monitoring/constants.tf`\n  - Domain, storage class, sizes, retention, and image versions for Prometheus, Loki, and Grafana.\n\nShortcut: for a one-command install, jump to [Easy deployment](#easy-deployment) to run the helper script; or continue reading for the detailed, step-by-step walkthrough below.\n\n### Generate Talos assets\n\nWhenever you change `vms_list.tf`, `vms_constants.tf`, or `patches/network.template.yaml`, regenerate the Talos inputs:\n\n```bash\n./scripts/gen-talos-assets.sh\n```\n\nThis script:\n\n- Renders per-VM network patches under `patches/network-*.yaml`\n- Removes stale patch files for deleted VMs\n- Generates `talos.tf` from several templates:\n  - [`templates/talos.template.tf`](templates/talos.template.tf) main Talos template.\n  - [`templates/controlplane-data.template.tf`](templates/controlplane-data.template.tf) template for Talos control plane nodes configuration data.\n  - [`templates/worker-data.template.tf`](templates/worker-data.template.tf) template for Talos worker nodes configuration data.\n  - [`templates/machine-config-locals.template.tf`](templates/machine-config-locals.template.tf) just create convinient local variables for easier Talos Tofu configuration steps.\n\n### Create the infrastructre\n\nApply the plan:\n\n```bash\ntofu apply -auto-approve\n```\n\nTo get Talos \u0026 k8s config (automatically generated on plan apply) on default paths just run:\n\n```bash\ntofu output -raw talosconfig |\u003e ~/.talos/config\ntofu output -raw kubeconfig |\u003e ~/.kube/config\n```\n\nNow you can run Talos \u0026 k8s commands:\n\n```bash\ntalosctl stats --nodes 192.168.1.51\nkubectl get nodes\n```\n\n### Rook Ceph\n\nRook is split into several plan applies to avoid the CRD plan-time limitation.\n\n#### CRDs + common + operator\n\nFirst, init \u0026 apply CRDs + common + operator from the rook/01 module:\n\n```bash\ntofu -chdir=rook/01-crds-common-operator init\ntofu -chdir=rook/01-crds-common-operator apply -auto-approve\n```\n\nWait till operator is running in ready state:\n\n```bash\nkubectl get pods -n rook-ceph -w\n# Something similar to this should be displayed\nNAME                                 READY   STATUS              RESTARTS   AGE\nrook-ceph-operator-f7867cb4b-j9qc4                        1/1     Running     0               30m\n```\n\n#### Operator creates the cluster\n\nThen init \u0026 apply the cluster CR in the separate module:\n\n```bash\ntofu -chdir=rook/02-cluster init\ntofu -chdir=rook/02-cluster apply -auto-approve\n```\n\nWait till operator is running in ready state:\n\n```bash\nkubectl get pods -n rook-ceph -w\n# Something similar to this should be displayed\nNAME                                                      READY   STATUS      RESTARTS        AGE\ncsi-cephfsplugin-bgc24                                    3/3     Running     1 (10m ago)     10m\ncsi-cephfsplugin-dk746                                    3/3     Running     1 (10m ago)     10m\ncsi-cephfsplugin-fvbpb                                    3/3     Running     1 (10m ago)     10m\ncsi-cephfsplugin-provisioner-76f4969f64-dksnv             6/6     Running     4 (9m26s ago)   10m\ncsi-cephfsplugin-provisioner-76f4969f64-td64m             6/6     Running     1 (10m ago)     10m\ncsi-rbdplugin-4zv88                                       3/3     Running     1 (10m ago)     10m\ncsi-rbdplugin-provisioner-7fcf98fc66-cbl6w                6/6     Running     1 (10m ago)     10m\ncsi-rbdplugin-provisioner-7fcf98fc66-p4bbn                6/6     Running     4 (9m20s ago)   10m\ncsi-rbdplugin-slw2x                                       3/3     Running     1 (10m ago)     10m\ncsi-rbdplugin-z96gx                                       3/3     Running     1 (10m ago)     10m\nrook-ceph-crashcollector-talos-g0a-1fy-6c4d7765b9-wgqw7   1/1     Running     0               9m23s\nrook-ceph-crashcollector-talos-lhn-rw4-6765469886-px7n8   1/1     Running     0               8m29s\nrook-ceph-crashcollector-talos-t5y-zub-6ff5989786-rw7jm   1/1     Running     0               8m30s\nrook-ceph-exporter-talos-g0a-1fy-85f85dbd97-7cgjj         1/1     Running     0               9m23s\nrook-ceph-exporter-talos-lhn-rw4-7c7fff48bb-dqwv6         1/1     Running     0               8m26s\nrook-ceph-exporter-talos-t5y-zub-84fbcc7594-f7m2t         1/1     Running     0               8m27s\nrook-ceph-mgr-a-57f77966b9-7xjk7                          3/3     Running     0               9m20s\nrook-ceph-mgr-b-67bd5d7648-j67zm                          3/3     Running     0               9m19s\nrook-ceph-mon-a-5f4b4f54db-b9nrz                          2/2     Running     0               10m\nrook-ceph-mon-b-9455f46b6-9lbbd                           2/2     Running     0               10m\nrook-ceph-mon-c-64d8f5665d-8wfxk                          2/2     Running     0               9m47s\nrook-ceph-operator-f7867cb4b-j9qc4                        1/1     Running     0               31m\nrook-ceph-osd-0-f96ff4b47-dgsdm                           2/2     Running     0               7m21s\nrook-ceph-osd-1-5bf66dfdc8-8zg8n                          2/2     Running     0               6m57s\nrook-ceph-osd-2-7dd7c8cf97-gmfm5                          2/2     Running     0               6m30s\nrook-ceph-osd-prepare-talos-g0a-1fy-vjq5l                 0/1     Completed   0               7m31s\nrook-ceph-osd-prepare-talos-lhn-rw4-mvqtz                 0/1     Completed   0               7m28s\nrook-ceph-osd-prepare-talos-t5y-zub-wsg42                 0/1     Completed   0               7m25s\n```\n\n#### Ceph dashboard\n\nIn order to visualize Ceph web dashboard, init \u0026 apply this separate module:\n\n```bash\ntofu -chdir=rook/03-dashboard init\ntofu -chdir=rook/03-dashboard apply -auto-approve\n```\n\nA node port service is created to access the web dashboard, to know which TCP port is used, just list the service:\n\n```bash\nkubectl -n rook-ceph get svc rook-ceph-mgr-dashboard-external-https\n# Something similar to this should be displayed\nNAME                                     TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE\nrook-ceph-mgr-dashboard-external-https   NodePort   10.96.145.137   \u003cnone\u003e        8443:32390/TCP   88s\n# The port in this k8s cluster was 32390\n```\n\nDefault `admin` user password is generated and created as a secret, to display it just run:\n\n```bash\nkubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath=\"{['data']['password']}\" | base64 --decode \u0026\u0026 echo\n```\n\n#### k8s CSI creation\n\nIn order to install k8s CSI providers based on:\n\n- CephFS: Ceph File System - file based PVCs for multiple pod access.\n- RBD: RADOS Block Device - block PVCs for only one pod access.\n\nInit \u0026 apply this separate module:\n\n```bash\ntofu -chdir=rook/04-csi init\ntofu -chdir=rook/04-csi apply -auto-approve\n```\n\nCSIs are created for all types and for erasure coded (EC) and replica modes:\n\n```bash\nkubectl -n rook-ceph get storageclasses.storage.k8s.io\n# Something similar to this should be displayed\nNAME                      PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE\nrook-ceph-block-ec        rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   9s\nrook-ceph-block-replica   rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   9s\nrook-cephfs-ec            rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   10m\nrook-cephfs-replica       rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   10m\n```\n\n### MetalLB, NGINX ingress controller \u0026 certificate manager\n\nDefine your constants in `k8s-net/constants.tf`: domain, CA organization, MetalLB pool range, and the fixed ingress IP.\nThe MetalLB pool and ingress service are rendered from templates using those values.\nDo not apply `k8s-net/metallb-pool.yaml` or `k8s-net/ingress-nginx-controller.yaml` directly; OpenTofu renders them with your constants.\n\nTo deploy these resources run the following command:\n\n```bash\ntofu -chdir=k8s-net init\ntofu -chdir=k8s-net apply -auto-approve\n```\n\n#### Install the Root CA locally\n\nInstall the generated Root CA so your browser and curl trust the `portainer.home.arpa` certificate (replace the domain if you changed it in `k8s-net/constants.tf`):\n\nmacOS:\n\n```bash\nsudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain k8s-net/home.arpa.pem\n```\n\nLinux (Debian/Ubuntu):\n\n```bash\nsudo cp k8s-net/home.arpa.pem /usr/local/share/ca-certificates/home.arpa.crt\nsudo update-ca-certificates\n```\n\nLinux (RHEL/CentOS/Fedora):\n\n```bash\nsudo cp k8s-net/home.arpa.pem /etc/pki/ca-trust/source/anchors/home.arpa.crt\nsudo update-ca-trust\n```\n\nWindows (PowerShell, admin):\n\n```powershell\nImport-Certificate -FilePath \"C:\\\\path\\\\to\\\\home.arpa.pem\" -CertStoreLocation Cert:\\\\LocalMachine\\\\Root\n```\n\n#### Local install of root CA and /etc/hosts\n\nUse `scripts/update-local.sh` to install the root CA and manage `/etc/hosts` entries based on `k8s-net/constants.tf` and `monitoring/constants.tf`. The script does not call `sudo`, so run it with `sudo` when it needs to edit system files.\n\n```bash\nsudo ./scripts/update-local.sh --root-ca\nsudo ./scripts/update-local.sh --etc-hosts\n# Or call with -a/--all to do all actions\nsudo ./scripts/update-local.sh --all\n```\n\nTo undo those changes, just run:\n\n```bash\nsudo ./scripts/update-local.sh --del-etc-hosts\n```\n\n#### Portainer\n\nPortainer is installed by OpenTofu as part of `k8s-net`. Access it at (replace the domain if you changed it in `k8s-net/constants.tf`):\n\n```text\nhttps://portainer.home.arpa\n```\n\nIf you don't have internal DNS, add an `/etc/hosts` entry using `ingress_lb_ip` from `k8s-net/constants.tf`:\n\n```bash\n192.168.1.70 portainer.home.arpa\n```\n\n#### Rook Ceph dashboard\n\nThe Rook Ceph dashboard is exposed at (replace the domain if you changed it in `k8s-net/constants.tf`):\n\n```text\nhttps://ceph.home.arpa\n```\n\nIf you don't have internal DNS, add an `/etc/hosts` entry using `ingress_lb_ip` from `k8s-net/constants.tf`:\n\n```bash\n192.168.1.70 ceph.home.arpa\n```\n\n### Monitoring (Prometheus, Loki, Grafana)\n\nDefine your constants in `monitoring/constants.tf`: domain, storage class, PVC sizes, retention settings, and image versions.\nThis stack also includes kube-state-metrics (requests/limits) and kubelet cAdvisor scrape for CPU/RAM usage.\nThe manifests are rendered from templates using those values.\nUse the same domain as `k8s-net/constants.tf` so TLS and DNS align.\n\nTo deploy the monitoring stack:\n\n```bash\ntofu -chdir=monitoring init\ntofu -chdir=monitoring apply -auto-approve\n```\n\nGrafana and Prometheus are exposed via TLS:\n\n```text\nhttps://grafana.home.arpa\nhttps://prometheus.home.arpa\n```\n\nGrafana admin credentials are generated by OpenTofu. Retrieve them with:\n\n```bash\ntofu -chdir=monitoring output -raw grafana_admin_user\ntofu -chdir=monitoring output -raw grafana_admin_password\n```\n\nGrafana dashboards are provisioned from `monitoring/grafana/dashboards/*.json`. After adding or editing a dashboard, re-run the monitoring apply and restart Grafana so it reloads the files.\n\n```bash\ntofu -chdir=monitoring apply -auto-approve\nkubectl -n monitoring rollout restart deploy/grafana\n```\n\n### Destroy the infrastructre\n\nIf you want to programmatically destroy the plan:\n\n```bash\ntofu destroy -auto-approve -refresh=false\n```\n\n## Easy deployment\n\nIn order to make easier the development of this repo an utility script [`scripts/deploy.sh`](scripts/deploy.sh) has been created to deploy full infrastructure from scratch following all described steps.\n\n:warning: **Use with caution** due to the VMs cluster will be removed if option `-d` or `--destroy` is passed.\n\n```bash\n./scripts/deploy.sh\n```\n\nRun with `-h` or `--help` to see help documentation. Common options:\n\n```text\n--help            Show usage help.\n--destroy         Destroy the cluster first and purge local state files.\n--skip-ceph       Skip Rook Ceph operator/cluster/dashboard/CSI.\n--skip-k8s-net    Skip MetalLB, ingress-nginx, cert-manager, Portainer.\n--skip-monitoring Skip Prometheus/Loki/Grafana stack.\n```\n\n## References\n\n### OpenTofu\n\n- [Installing OpenTofu via Homebrew](https://opentofu.org/docs/intro/install/homebrew/)\n- [Working with OpenTofu](https://opentofu.org/docs/intro/core-workflow/)\n- [`bpg/proxmox` OpenTofu provider for Proxmox VE](https://search.opentofu.org/provider/bpg/proxmox/latest)\n- [`bpg/proxmox` reference for `proxmox_virtual_environment_vm` resource](https://registry.terraform.io/providers/bpg/proxmox/latest/docs/resources/virtual_environment_vm)\n\n### Tutorials\n\n- [Talos cluster on Proxmox with Terraform](https://olav.ninja/talos-cluster-on-proxmox-with-terraform)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fociotec%2Fpve-k8s-talos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fociotec%2Fpve-k8s-talos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fociotec%2Fpve-k8s-talos/lists"}