{"id":13702763,"url":"https://github.com/identiops/terraform-hcloud-k3s","last_synced_at":"2026-02-10T12:01:25.825Z","repository":{"id":127442986,"uuid":"387512324","full_name":"identiops/terraform-hcloud-k3s","owner":"identiops","description":"Private k3s kubernetes terraform installer for Hetzner Cloud","archived":false,"fork":false,"pushed_at":"2026-02-05T15:00:32.000Z","size":3922,"stargazers_count":145,"open_issues_count":5,"forks_count":17,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-02-06T01:34:03.886Z","etag":null,"topics":["cloud","hcloud","hetzner","k3s","k8s","kubernetes","kubernetes-cluster","terraform","terraform-module"],"latest_commit_sha":null,"homepage":"https://identiops.com/terraform-hcloud-k3s","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"cicdteam/terraform-hcloud-k3s","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/identiops.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-07-19T15:32:27.000Z","updated_at":"2026-02-05T13:08:59.000Z","dependencies_parsed_at":"2026-02-05T17:01:59.367Z","dependency_job_id":null,"html_url":"https://github.com/identiops/terraform-hcloud-k3s","commit_stats":null,"previous_names":["identiops/terraform-hcloud-k3s"],"tags_count":58,"template":false,"template_full_name":null,"purl":"pkg:github/identiops/terraform-hcloud-k3s","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/identiops%2Fterraform-hcloud-k3s","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/identiops%2Fterraform-hcloud-k3s/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/identiops%2Fterraform-hcloud-k3s/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/identiops%2Fterraform-hcloud-k3s/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/identiops","download_url":"https://codeload.github.com/identiops/terraform-hcloud-k3s/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/identiops%2Fterraform-hcloud-k3s/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29298523,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-10T10:40:02.018Z","status":"ssl_error","status_checked_at":"2026-02-10T10:38:28.459Z","response_time":65,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud","hcloud","hetzner","k3s","k8s","kubernetes","kubernetes-cluster","terraform","terraform-module"],"created_at":"2024-08-02T21:00:42.228Z","updated_at":"2026-02-10T12:01:25.469Z","avatar_url":"https://github.com/identiops.png","language":"HCL","funding_links":[],"categories":["Tools","HCL"],"sub_categories":["Rust"],"readme":"- ![license](https://img.shields.io/github/license/identiops/terraform-hcloud-k3s)\n- [![stars](https://img.shields.io/github/stars/identiops/terraform-hcloud-k3s)](https://github.com/identiops/terraform-hcloud-k3s)\n- terraform-hcloud-k3s is an [identiops.com project](https://identiops.com)\n\n# Private k3s Kubernetes Terraform installer for Hetzner Cloud\n\nThis Terraform module creates a Kubernetes Cluster on\n[Hetzner Cloud](https://console.hetzner.cloud/) infrastructure running Ubuntu\n24.04. The module aims to be simple to use while providing an out-of-the-box\nsecure and maintainable setup. Thanks to Ubuntu's LTS version we get up to 5\nyears of peace and quiet before having to upgrade the cluster's operating\nsystem!\n\nModule published at:\n\n- Terraform registry:\n  \u003chttps://registry.terraform.io/modules/identiops/k3s/hcloud\u003e\n- OpenTofu registry: \u003chttps://search.opentofu.org/module/identiops/k3s/hcloud\u003e\n\nWhat changed in the latest version? See\n[CHANGELOG.md](https://github.com/identiops/terraform-hcloud-k3s/tree/main/CHANGELOG.md).\n\n![infrastructure](https://raw.githubusercontent.com/identiops/terraform-hcloud-k3s/main/figures/infrastructure.png)\n\n## Features\n\n- [k3s](https://k3s.io/) based Kubernetes cluster.\n- Node pools for managing cluster resources efficiently. Pools can be added,\n  resized, and removed at any time.\n- Automated Kubernetes update management via\n  [System Upgrade Controller](https://github.com/rancher/system-upgrade-controller).\n- Automated operating system updates with automatic system reboots via\n  [kured](https://kured.dev).\n- [Embedded container registry mirror](https://docs.k3s.io/installation/registry-mirror)\n  for fast pod startup times across nodes.\n- [Private registry configuration](https://docs.k3s.io/installation/private-registry?_highlight=registries#configs)\n  to avoid manual configuration of\n  [pull secrets](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).\n- Creation of placement groups for to improve availability.\n- Multi-region deployments.\n- Secured default configuration:\n  - Deletion protection for all cloud resources.\n  - SSH key required for remote access.\n  - fail2ban limits SSH brute force attacks.\n  - Cluster nodes have no public network interface.\n  - Internal firewall active on all nodes for minimal exposure.\n  - Support for network policies via [Cilium](https://www.cilium.io/).\n  - [CSI hardening guide](https://docs.k3s.io/security/hardening-guide) applied:\n    - Kernel parameters defined.\n    - Audit log directory created.\n    - Network policies, pod security policies, admission policies and the\n      enabling of audit logs are up to the administrator of the cluster to\n      configure.\n- Integration of Hetzner Cloud Controller Manager for managing cloud resources\n  from the within the cluster.\n- Integration of Hetzner Cloud Storage Interface for managing volumes from the\n  within the cluster.\n- Ansible integration with automatically generated inventory.\n- Convenience scripts for retrieving the Kubernetes configuration and accessing\n  nodes via SSH and SCP.\n- Calculation of monthly costs for every part of the deployment (see\n  `terraform output`).\n- Documentation of common administrative tasks and troubleshooting approaches.\n\n### To be added\n\n- OIDC support for user authentication. Some configuration is in place, but it\n  hasn't been tested, yet.\n- Support for [cluster auto scaler](https://github.com/kubernetes/autoscaler).\n\n\u003c!-- generated with\n!deno run --allow-read --allow-write https://deno.land/x/remark_format_cli@0.4.0/remark-format.js %\n--\u003e\n\n## Contents\n\n1. [Getting Started](#getting-started)\n   1. [Prerequisites](#prerequisites)\n   2. [Recommended Tools](#recommended-tools)\n   3. [Installation](#installation)\n   4. [Usage](#usage)\n2. [Configuration](#configuration)\n   1. [Store terraform state in S3 bucket](#store-terraform-state-in-s3-bucket)\n   2. [Enable etcd backup to S3](#enable-etcd-backup-to-s3)\n   3. [OpenID Connect (OIDC) Authentication](#openid-connect-oidc-authentication)\n   4. [Persistent Volume Encryption](#persistent-volume-encryption)\n   5. [Adjust Sysctl Parameters](#adjust-sysctl-parameters)\n   6. [Private Registry Access](#private-registry-access)\n3. [Maintenance](#maintenance)\n   1. [Access Kubernetes API via Port-Forwarding from Gateway](#access-kubernetes-api-via-port-forwarding-from-gateway)\n   2. [Ansible: Execute Commands on Nodes](#ansible-execute-commands-on-nodes)\n   3. [Add Ingress Controller and Load Balancer](#add-ingress-controller-and-load-balancer)\n   4. [Add Nodes or Node Pools](#add-nodes-or-node-pools)\n   5. [Remove Nodes or Node Pools](#remove-nodes-or-node-pools)\n   6. [Stop Automated Node Reboots](#stop-automated-node-reboots)\n   7. [Upgrade Operating System](#upgrade-operating-system)\n      1. [Gateway Node](#gateway-node)\n      2. [Node Pools](#node-pools)\n   8. [Update Kubernetes](#update-kubernetes)\n   9. [Update Cilium](#update-cilium)\n   10. [Update Hetzner Cloud Controller Manager (CCM)](#update-hetzner-cloud-controller-manager-ccm)\n   11. [Update Hetzner Cloud Storage Interface (CSI)](#update-hetzner-cloud-storage-interface-csi)\n   12. [Update Kured](#update-kured)\n   13. [Update Metrics Server](#update-metrics-server)\n   14. [Update System Upgrade Controller](#update-system-upgrade-controller)\n4. [Deletion](#deletion)\n5. [Troubleshooting](#troubleshooting)\n   1. [Gateway](#gateway)\n      1. [Verify packet masquerading is set up properly](#verify-packet-masquerading-is-set-up-properly)\n      2. [Verify firewall is set up properly](#verify-firewall-is-set-up-properly)\n   2. [Nodes](#nodes)\n      1. [Inspect local firewall settings](#inspect-local-firewall-settings)\n      2. [Verify correctness of date/timezone and locale](#verify-correctness-of-datetimezone-and-locale)\n      3. [Inspect cloud-init logs](#inspect-cloud-init-logs)\n   3. [Cluster](#cluster)\n      1. [Verify default route](#verify-default-route)\n      2. [Verify connectivity to the Internet](#verify-connectivity-to-the-internet)\n      3. [Verify name resolution](#verify-name-resolution)\n      4. [Verify cluster status](#verify-cluster-status)\n      5. [Verify Cilium Networking Status](#verify-cilium-networking-status)\n      6. [Verify k3s Cluster Configuration](#verify-k3s-cluster-configuration)\n      7. [Inspect cluster status and logs](#inspect-cluster-status-and-logs)\n6. [Related Documentation](#related-documentation)\n7. [Similar Projects](#similar-projects)\n8. [Special Thanks](#special-thanks)\n\n## Getting Started\n\n### Prerequisites\n\nWARNING: Apple Silicon is apparently not supported by important terraform\nmodules, see \u003chttps://github.com/identiops/terraform-hcloud-k3s/issues/19\u003e.\n\n- [Terraform](https://terraform.io) or [OpenTofu](https://opentofu.org/). Note\n  that you'll need Terraform v1.0 or newer to run this project.\n- `bash` for executing the generated scripts.\n- `jq` for executing the generated scripts.\n- `kubectl` for interacting with the Kubernetes cluster.\n- `ssh` for connecting to cluster nodes.\n\n### Recommended Tools\n\n- [Ansible cli](https://www.ansible.com/) for executing commands simultaneously\n  on multiple cluster nodes.\n- [Cilium cli](https://github.com/cilium/cilium-cli) for verifying and\n  interacting with the Kubernetes networking layer.\n- [Helm cli](https://helm.sh/) for installing and updating components inside the\n  cluster.\n\n### Installation\n\n1. Create a Hetzner Cloud API token.\n   - Register with [Hetzner Cloud](https://console.hetzner.cloud).\n   - Create a new project.\n   - Navigate to the security settings.\n   - Select the \"API tokens\" tab and add a new token with **read \u0026 write**\n     access and a second token with just **read** access.\n\n![API Token Generation](https://raw.githubusercontent.com/identiops/terraform-hcloud-k3s/main/figures/hetzner_api_token.png)\n\n2. Store Hetzner Cloud API token locally. Either, pass the tokens to terraform\n   via an environment variable or create a file called `terraform.tfvars`:\n\n```bash\n# Either, enter your Hetzner Cloud API Token (it will be hidden)\nread -sp \"Hetzner Cloud API Token: \" TF_VAR_hcloud_token\nexport TF_VAR_hcloud_token\nread -sp \"Hetzner Cloud API read only Token: \" TF_VAR_hcloud_token_read_only\nexport TF_VAR_hcloud_token_read_only\n\n# Or store the token in terraform.tfvars\ntouch terraform.tfvars\nchmod 600 terraform.tfvars\ncat \u003eterraform.tfvars \u003c\u003cEOF\nhcloud_token = \"XYZ\"\nhcloud_token_read_only = \"ABC\"\nEOF\n```\n\n3. Download\n   [`examples/1Region_3ControlPlane_3Worker_Nodes/main.tf`](https://github.com/identiops/terraform-hcloud-k3s/blob/main/examples/1Region_3ControlPlane_3Worker_Nodes/main.tf):\n   `curl -LO https://github.com/identiops/terraform-hcloud-k3s/raw/main/examples/1Region_3ControlPlane_3Worker_Nodes/main.tf`\n4. Adjust the cluster configuration in `main.tf`, e.g.\n   - `cluster_name`\n   - `default_location`\n   - `k3s_version`\n   - `ssh_keys` (to create a new ssh key run: `ssh-keygen -t ed25519`)\n   - `node_pools`\n   - More options described in section [Configurtion](#configuration)\n5. For multi-region deployments, there are a few things to consider:\n   - It is recommended to distribute the control plane nodes across multiple\n     regions. If 3 control plane nodes shall be used, create 3 node pools and\n     configure a different `location` for each pool.\n   - etcd's default configuration expects a low-latency local network. When\n     distributing nodes across multiple regions, latency will increase. The\n     timing parameters therefore need to be adjusted, see\n     [etcd Tuning](https://etcd.io/docs/v3.4/tuning/#time-parameters). Set\n     `control_plane_k3s_additional_options`, e.g. to\n     `--etcd-arg=heartbeat-interval=120 --etcd-arg=election-timeout=1200`\n     Measurements between Falkenstein, Nuremberg and Helsinki: I measured a\n     latency of 0.7ms (within Nuremberg region), 3ms (Nuremberg -\u003e Falkenstein),\n     and 24ms (Nuremberg -\u003e Helsinki).\n   - Hetzner doesn't support mounting volumes on servers in another region! The\n     most simple setup is to just distribute the control plane nodes across\n     multiple regions, disable the scheduling of workloads on control plane\n     nodes and keep all worker nodes pools within one region. For better\n     availability of workloads, worker node pools should be distributed across\n     regions. This requires a configuration of\n     [taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)\n     and\n     [node affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)\n     to ensure that pods with volumes are schedules in the correct region.\n   - Depending on the selected server regions the `network_zone` setting should\n     be adjusted.\n   - Hetzner's load balancers are bound to one region. Therefore, a\n     multi-regional cluster with one load balancer is not sufficent for dealing\n     with zone outages. The load balancer would go down with the region it's\n     hosted in. If a protection against zone outages is required, a global load\n     balancer should be deployed elsewhere, as\n     [described by Google](https://cloud.google.com/load-balancing/docs/load-balancing-overview).\n6. Initialize the configuration: `terraform init`\n7. Apply the configuration: `terraform apply`\n8. Grab a coffee and enjoy the servers popping up in Hetzner's cloud console.\n   Wait for about 5 minutes.\n\n![Servers overview](https://raw.githubusercontent.com/identiops/terraform-hcloud-k3s/main/figures/hetzner_servers.png)\n\n9. Test SSH access to the cluster: `./ssh-node cluster`\n   - ATTENTION: don't hammer the cluster with failing SSH requests, or you'll be\n     banned by your cluster automatically! If the request fails, because the\n     cluster node isn't ready yet, wait another minute.\n10. Once SSH connection is established, double check that everything is working\n    as expected:\n    - Did the node initialization finish successfully? `cloud-init status`\n    - Is the cluster up and running? `kubectl cluster-info`\n11. If the tests were successful, retrieve the Kubernetes configuration and\n    store it locally: `./setkubeconfig`\n12. Forward the cluster port locally since it's not exposed to the Internet by\n    default. Do this every time you want to interact with the cluster:\n    `./ssh-node gateway`\n13. **On your local machine** test cluster access: `kubectl get nodes`\n\nEnjoy your new cluster! 🚀\n\n![k9s CLI](https://raw.githubusercontent.com/identiops/terraform-hcloud-k3s/main/figures/k9s.png)\n\n### Usage\n\nStart using your favorite Kubernetes tools to interact with the cluster. One of\nthe first steps usually involves\n[deploying an ingress controller](#add-ingress-controller-and-load-balancer)\nsince this configuration doesn't ship one.\n\nIn addition, a few convenience scripts were created to help with maintenance:\n\n- `setkubeconfig`: retrieves and stores the Kubernetes configuration locally.\n- `unsetkubeconfig`: removes the cluster from the local Kubernetes\n  configuration.\n- `ls-nodes`: lists the nodes that are part of the cluster for access via\n  `ssh-node` and `scp-node`.\n- `ssh-node`: SSH wrapper for connecting to cluster nodes.\n- `scp-node`: SCP wrapper for connecting to cluster nodes.\n- `.ssh/config`: SSH configuration for connecting to cluster nodes.\n- `.ansible/hosts`: Ansible hosts configuration for executing commands on\n  multiple nodes in parallel.\n\n## Configuration\n\n### Store terraform state in S3 bucket\n\n- Add an \"s3\" backend section to `main.tf`, with the appropriate parameters. See\n  example below for Wasabi.\n- Set access key id and access key in environment variables `AWS_ACCESS_KEY_ID`\n  and `AWS_SECRET_ACCESS_KEY`.\n- Reconfigure terraform: `terraform init -reconfigure`\n\n```terraform\nterraform {\n  # ...\n  backend \"s3\" {\n    # INFO: Set via `terraform init -reconfigure`\n    bucket = \"terraform\"\n    key    = \"prod/terraform.tfstate\"\n    # access_key                  = {}\n    # secret_key                  = {}\n    # skip_get_ec2_platforms      = true\n    region                      = \"eu-central-2\"\n    skip_credentials_validation = true\n    skip_metadata_api_check     = true\n    skip_region_validation      = true\n    skip_requesting_account_id  = true\n    skip_s3_checksum            = true\n    use_path_style              = true\n    endpoints = {\n      iam = \"https://iam.eu-central-2.wasabisys.com\" # special endpoint URL required, see https://wasabi-support.zendesk.com/hc/en-us/articles/360003362071-How-do-I-use-Terraform-with-Wasabi-\n      sts = \"https://sts.eu-central-2.wasabisys.com\" # special endpoint URL required, see https://wasabi-support.zendesk.com/hc/en-us/articles/360003362071-How-do-I-use-Terraform-with-Wasabi-\n      s3  = \"https://s3.eu-central-2.wasabisys.com\"  # special endpoint URL required, see https://wasabi-support.zendesk.com/hc/en-us/articles/360003362071-How-do-I-use-Terraform-with-Wasabi-\n    }\n  }\n  # ...\n}\n```\n\n### Enable etcd backup to S3\n\n- Extend `main.tf` with the following variables:\n\n```terraform\nvariable \"etcd_s3_region\" {\n  type      = string\n  sensitive = true\n}\n\nvariable \"etcd_s3_endpoint\" {\n  type      = string\n  sensitive = true\n}\n\nvariable \"etcd_s3_access_key\" {\n  type      = string\n  sensitive = true\n}\n\nvariable \"etcd_s3_secret_key\" {\n  type      = string\n  sensitive = true\n}\n\nvariable \"etcd_s3_bucket\" {\n  type      = string\n  sensitive = true\n}\n```\n\n- Set the following module parameters before initializing the cluster's control\n  plane nodes:\n\n```terraform\ncontrol_plane_k3s_init_additional_options = \"--etcd-s3 --etcd-s3-region=${var.etcd_s3_region} --etcd-s3-endpoint=s3.${var.etcd_s3_endpoint} --etcd-s3-access-key=${var.etcd_s3_access_key} --etcd-s3-secret-key=${var.etcd_s3_secret_key} --etcd-s3-bucket=${var.etcd_s3_bucket} --etcd-s3-folder=$(hostname)\"\n```\n\n- Etcd will automatically create a snapshot every day and keep it for three\n  days.\n\n### OpenID Connect (OIDC) Authentication\n\nTODO: add example\n\n### Persistent Volume Encryption\n\n- Ensure `cryptsetup` is installed on all nodes. By default it is installed.\n- Encryption is configured per `StorageClass`.\n- Follow the instructions at\n  [hcloud-csi Volumes Encrypted with LUKS](https://github.com/hetznercloud/csi-driver/blob/main/docs/kubernetes/README.md#volumes-encrypted-with-luks)\n  to create a secret and link it to a storage class.\n- All future volumes created with this `StorageClass` will be encrypted.\n\n### Adjust Sysctl Parameters\n\n- Sysctl paramerts can be adjusted if needed via the `sysctl_settings` variable.\n- Caution is advised!\n- A common issue are truncated `kubectl log` outputs when the variables\n  `fs.inotify.max_user_instances` and `fs.inotify.max_user_watches` are set too\n  low. The values are increased by the default configuration. Furthere\n  adjustments might be needed.\n\n### Private Registry Access\n\nUsually,\n[pull secrets](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/)\nneed to be configured to access container images stored in private registries.\nThis terraform module provides access to k3s'\n[Private Registry Configuration](https://docs.k3s.io/installation/private-registry).\n\nAdd a private registry configuration with the following setting:\n\n```terraform\nmodule \"cluster\" {\n  # ...\n  registries = {\n    mirrors = {\n      \"*\" = null\n    }\n    configs = {\n      \"registry-1.docker.io\" = {\n        auth = {\n          token = var.dockerio_access_token\n        }\n      }\n    }\n  }\n  # ...\n}\nvariable \"dockerio_access_token\" {\n type      = string\n sensitive = true\n}\n```\n\n## Maintenance\n\n### Access Kubernetes API via Port-Forwarding from Gateway\n\nBy default, the Kubernetes API port is not exposed on the internet, see option\n`gateway_firewall_k8s_open`. Therefore, it is required to forward the Kubernetes\nAPI port `6443` from the gateway.\n\nA limited account `kubeapi` was created on the gateway that restricts SSH keys\nconfigured via option `ssh_keys_kubeapi` to forward the single port. A\ncorresponding SSH config entry is created. The following command connects to the\ngateway with the restricted user account:\n\n```bash\n./ssh-node kubeapi\n```\n\nWith the established connection and a proper `KUBECONFIG`, the cluster can be\naccessed:\n\n```bash\nkubectl get nodes\n```\n\n### Ansible: Execute Commands on Nodes\n\nThis module automatically generates an\n[Ansible inventory](https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.html#connecting-to-hosts-behavioral-inventory-parameters)\nin file `.ansible/hosts`. It can be leveraged to interact with the nodes and\nnode pools of the cluster.\n\nExample: Execute a command on all control plane nodes\n\n```bash\nANSIBLE_INVENTORY=\"$PWD/.ansible/hosts\" ansible all_control_plane_nodes -a \"kubectl cluster-info\"\n```\n\n### Add Ingress Controller and Load Balancer\n\nSince this module doesn't ship an ingress controller, one of the first\nconfigurations applied to the cluster is usually an ingress controller. Good\nstarting points for an ingress controller are:\n\n- [traefik](https://artifacthub.io/packages/helm/traefik/traefik)\n- [ingress-nginx](https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx)\n\nThe ingress controller, like the rest of the cluster, is not directly exposed to\nthe Internet. Therefore, it is necessary to add a load balancer that is directly\nexposed to the Internet and has access to the local network of the cluster. The\nload balancer is added to the cluster simply by adding annotations to the\ningress controller's service. Hetzner's Cloud Controller Manager will use the\nannotations to deploy and configure the load balancer.\n\nThe following annotations should be used:\n\n- Resource name: `load-balancer.hetzner.cloud/name: \"ingress-lb\"`\n- Protocol, just `tcp` - the ingress controller will take care of the HTTP\n  connection: `load-balancer.hetzner.cloud/protocol: \"tcp\"`\n- Location, same as the one used for the cluster:\n  `load-balancer.hetzner.cloud/location: \"nbg1\"`\n- Connection to the servers, must be set to `true`:\n  `load-balancer.hetzner.cloud/use-private-ip: \"true\"`\n- Size, see [options](https://docs.hetzner.com/cloud/load-balancers/overview):\n  `load-balancer.hetzner.cloud/type: \"lb11\"`\n- See\n  [list of all supported Load Balancer Annotations](https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/main/internal/annotation/load_balancer.go).\n\nFurthermore, for domain names to work, it is required to point DNS records to\nthe IP address of load balancer.\n[external-dns](https://artifacthub.io/packages/helm/external-dns/external-dns)\nis a helpful tool that can automate this task from within the cluster. For this\nto work well with Ingress resources, the ingress controller needs to\n[expose the published service information](https://kubernetes-sigs.github.io/external-dns/v0.14.0/faq/#which-service-and-ingress-controllers-are-supported)\non the Ingress resources.\n\n### Add Nodes or Node Pools\n\nThe number of nodes in a node pool can be increased at any point. Just increase\nthe count and apply the new configuration via `terraform apply`. After a few\nminutes the additional nodes will appear in the cluster.\n\nIn the same way, node pools can be added to the configuration without any\nprecaution.\n\n### Remove Nodes or Node Pools\n\nRemoving nodes requires the following steps:\n\n1. Identify the nodes and node pools that shall be removed. If the number of\n   nodes in a node pool needs to be decreased, the nodes will be removed from\n   the highest to the lowest number. Example: when the number of nodes in pool\n   `system` is decreased from `3` to `2`, node `cluster-system-02` will be\n   removed and nodes `cluster-system-01` and `cluster-system-00` will remain.\n2. Drain all nodes that will be removed of pods:\n   `kubectl drain cluster-system-02`\n3. Wait until all pods have been migrated to other nodes before continuing.\n   - If you drained the wrong node, you can reactivate the node with:\n     `kubectl uncordon cluster-system-02`\n4. Update the terrafrom configuration and apply it: `terraform apply`\n   - Review the plan to verify that the drained nodes will be deleted.\n5. Delete nodes from cluster: `kubectl delete node cluster-system-02`\n\n### Stop Automated Node Reboots\n\nNodes are rebooting automatically when they receive updates that require a\nreboot. The kured service triggers reboots of nodes one by one. Reboots can be\ndisabled system-wide by annotating the Daemonset, see\n\u003chttps://kured.dev/docs/operation/\u003e.\n\n### Upgrade Operating System\n\nAn operating system update is not recommended, e.g. from Ubuntu 22.04 to 24.04.\nInstead, the corresponding nodes should be replaced!\n\n#### Gateway Node\n\n1. Set new image as `default_image`. Attention: before changing the default\n   image, make sure that each node pool has its own appropriate `image` setting.\n2. Delete the node in the [Hetzner Console](https://console.hetzner.cloud/)\n3. Reapply the configuration: `terraform apply`\n\nThe gateway will reappear again within a few minutes. This will disrupt the\nInternet access of the cluster's nodes for tasks like fetching package updates.\nHowever, it will not affect the services that are provided via load balancers!\n\nAfter redeploying the gateway, ssh connections will fail because a new\ncryptographic has been generated for the node. Delete the deprecated key from\nthe `.ssh/known_hosts` file, open a new ssh connection and accept the new public\nkey.\n\n#### Node Pools\n\nNodes should not be updated manually via `agt-get`, but be replaced. For control\nplane nodes, it is recommended to create a back-up of the etcd data store on an\nexternal s3 storage, see [k3s Cluster Datastore](https://docs.k3s.io/datastore).\n\n1. For control plane pools only: Create a new etcd snapshot, see\n   [k3s etcd-snapshot](https://docs.k3s.io/cli/etcd-snapshot).\n2. Then, perform the following steps on consecutively on all existing node pools\n   until they have all been replaced.\n\nStart the replacement with the node pool with the `cluster_can_init` setting:\n\n3. Ensure that there's another control plane node pool. If there's no other\n   control plane node pool, create a temporary one that is deleted after the\n   successful replacement of the node pool with the `cluster_can_init` setting.\n4. When the second control plane node pool is up and running, the node pool with\n   the `cluster_can_init` setting must be deleted and replaced in one\n   application of the configuration.\n   - Ensure that the `cluster_init_action.init` and `cluster_init_action.reset`\n     settings are disabled.\n   - Drain the old nodes: `kubectl drain node-xyz`\n   - Once all pods have been migrated, delete the old nodes:\n     `kubectl delete node node-xyz`\n   - Then rename the node pool to achieve deletion and replacement in one\n     configuration change.\n   - Apply the configuration: `terraform apply`\n   - Once the new control plane node pool with the `cluster_can_init` setting is\n     again up and running, the temporary control plane node pool can be deleted.\n\nPerform these steps for all remaining node pools:\n\n5. Add a new node pool and set the `image` setting to the new version.\n6. Once the new node pool is up and running, drain the old nodes:\n   `kubectl drain node-xyz`\n7. Once all pods have been migrated, delete the old nodes:\n   `kubectl delete node node-xyz`\n8. Remove the node pool from the terraform configuration.\n9. Reapply the configuration: `terraform apply`\n\n### Update Kubernetes\n\n1. Determine the next Kubernetes version, see\n   [k3s channels](https://update.k3s.io/v1-release/channels),\n   [k3s images tags](https://hub.docker.com/r/rancher/k3s/tags), and\n   [k3s upgrade image tags](https://hub.docker.com/r/rancher/k3s-upgrade/tags).\n2. Write the upgrade plan, see\n   [instructions](https://docs.k3s.io/upgrades/automated) and\n   [examples](https://github.com/rancher/system-upgrade-controller#example-plans).\n3. Apply an upgrade plan.\n4. Update the `image` variable in the configuration for future nodes to be\n   deployed with the correct image.\n\nExample upgrade plan:\n\n```yaml\n# Documentation: https://github.com/rancher/system-upgrade-controller\n# Upgrade plan for control plane nodes\napiVersion: upgrade.cattle.io/v1\nkind: Plan\nmetadata:\n  name: k3s-server\n  namespace: cattle-system\nspec:\n  # When a channel is used, rather than a specific version,\n  # the plan keeps updating the cluster continuously\n  channel: https://update.k3s.io/v1-release/channels/v1.32\n  # version v1.30.0+k3s1\n  concurrency: 1\n  drain:\n    # disableEviction: true\n    force: true\n    skipWaitForDeleteTimeout: 60\n  nodeSelector:\n    matchExpressions:\n      - { key: k3s-upgrade, operator: Exists }\n      # Updates can be disabled by setting this label on nodes\n      - { key: k3s-upgrade, operator: NotIn, values: [\"disabled\", \"false\"] }\n      - { key: k3os.io/mode, operator: DoesNotExist }\n      - { key: node-role.kubernetes.io/control-plane, operator: Exists }\n  serviceAccountName: system-upgrade-controller\n  upgrade:\n    image: rancher/k3s-upgrade\n---\n# Upgrade plan for worker nodes\napiVersion: upgrade.cattle.io/v1\nkind: Plan\nmetadata:\n  name: k3s-agent\n  namespace: cattle-system\nspec:\n  # When a channel is used, rather than a specific version,\n  # the plan keeps updating the cluster continuously\n  channel: https://update.k3s.io/v1-release/channels/v1.32\n  # version v1.30.0+k3s1\n  concurrency: 1\n  drain:\n    # disableEviction: true\n    force: true\n    skipWaitForDeleteTimeout: 60\n  nodeSelector:\n    matchExpressions:\n      - { key: k3s-upgrade, operator: Exists }\n      # Updates can be disabled by setting this label on nodes\n      - { key: k3s-upgrade, operator: NotIn, values: [\"disabled\", \"false\"] }\n      - { key: k3os.io/mode, operator: DoesNotExist }\n      - { key: node-role.kubernetes.io/control-plane, operator: DoesNotExist }\n  serviceAccountName: system-upgrade-controller\n  prepare:\n    # Defaults to the same \"resolved\" tag that is used for the `upgrade` container, NOT `latest`\n    image: rancher/k3s-upgrade\n    args: [\"prepare\", \"k3s-server\"]\n  upgrade:\n    image: rancher/k3s-upgrade\n```\n\n### Update Cilium\n\n- Available versions: \u003chttps://github.com/cilium/cilium\u003e\n- Update instructions: \u003chttps://docs.cilium.io/en/stable/operations/upgrade/\u003e\n\n```bash\nhelm repo add cilium https://helm.cilium.io/\nhelm repo update\nhelm upgrade --reuse-values cilium cilium/cilium -n kube-system --version '\u003cNEW_VERSION\u003e'\n```\n\n`values.yaml`:\n\n```yaml\n# Documentation: https://artifacthub.io/packages/helm/cilium/cilium\n# Chart Values: https://github.com/cilium/cilium/blob/main/install/kubernetes/cilium/values.yaml\n\n# Hetzner networking configuration: https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/main/docs/deploy_with_networks.md\n# See also kube-hetzner configuration of cilium: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner\n\n# MTU: 1450\nmasquerade: true\nbpf:\n  masquerade: true\nendpointRoutes:\n  enabled: true\nk8s:\n  requireIPv4PodCIDR: true\n# Native routing saves one layer of packet encapsulation\nroutingMode: native\n# WARNING: IP address ranges need to be in line with the cluster configuration\nipv4NativeRoutingCIDR: 10.0.0.0/8\nipam:\n  mode: kubernetes\n  operator:\n    clusterPoolIPv4PodCIDRList: 10.244.0.0/16\noperator:\n  replicas: 2\nkubeProxyReplacement: true\nnodePort:\n  enabled: true\nloadBalancer:\n  acceleration: native\nhubble:\n  enabled: true\n# Enable, if needed. By default it isn't needed\n# egressGateway:\n#   enabled: true\n\n# HAproxy runs on every node to provide HA access to control plane nodes\nk8sServiceHost: 127.0.0.1\nk8sServicePort: 16443\n```\n\n### Update Hetzner Cloud Controller Manager (CCM)\n\n- Available versions:\n  \u003chttps://github.com/hetznercloud/hcloud-cloud-controller-manager#versioning-policy\u003e\n- Update instructions:\n  \u003chttps://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/main/CHANGELOG.md\u003e\n\n```bash\nhelm repo add hcloud https://charts.hetzner.cloud\nhelm repo update\nhelm upgrade --reuse-values hcloud-ccm hcloud/hcloud-cloud-controller-manager -n kube-system --version '\u003cNEW_VERSION\u003e'\n```\n\n`values.yaml`:\n\n```yaml\n# Documentation: https://github.com/hetznercloud/hcloud-cloud-controller-manager/tree/main/chart\n\n# WARNING: needs to be in line with the cluster configuration\nnetworking:\n  enabled: true\n  clusterCIDR: 10.244.0.0/16\n\nadditionalTolerations:\n  # INFO: this taint occurred but isn't coveryd by default .. and caused the\n  # whole cluster to not start properly\n  - key: node.kubernetes.io/not-ready\n    value: NoSchedule\n```\n\n### Update Hetzner Cloud Storage Interface (CSI)\n\n- Available versions:\n  \u003chttps://github.com/hetznercloud/csi-driver/blob/main/docs/kubernetes/README.md#versioning-policy\u003e\n- Update instructions:\n  \u003chttps://github.com/hetznercloud/csi-driver/blob/main/CHANGELOG.md\u003e\n\n```bash\nhelm repo add hcloud https://charts.hetzner.cloud\nhelm repo update\nhelm upgrade --reuse-values hcloud-csi hcloud/hcloud-csi -n kube-system --version '\u003cNEW_VERSION\u003e'\n```\n\n`values.yaml`:\n\n```yaml\n# Documentation: https://github.com/hetznercloud/csi-driver/tree/main/chart\n\nstorageClasses:\n  - name: hcloud-volumes-retain\n    defaultStorageClass: true\n    reclaimPolicy: Retain\n  - name: hcloud-volumes-delete\n    defaultStorageClass: false\n    reclaimPolicy: Delete\n```\n\n### Update Kured\n\n- Available versions: \u003chttps://artifacthub.io/packages/helm/kured/kured\u003e\n- Update instructions: \u003chttps://github.com/kubereboot/kured\u003e\n\n```bash\nhelm repo add kubereboot https://kubereboot.github.io/charts\nhelm repo update\nhelm upgrade --reuse-values kured kubereboot/kured -n kube-system --version '\u003cNEW_VERSION\u003e'\n```\n\n`values.yaml`:\n\n```yaml\n# Documentation: https://artifacthub.io/packages/helm/kured/kured\n\nconfiguration:\n  timeZone: Europe/Berlin\n  startTime: 1am\n  endTime: 5am\n  rebootDays:\n    - mo\n    - tu\n    - we\n    - th\n    - fr\n    - sa\n    - su\ntolerations:\n  - key: CriticalAddonsOnly\n    operator: Exists\n```\n\n### Update Metrics Server\n\n- Available versions:\n  \u003chttps://artifacthub.io/packages/helm/metrics-server/metrics-server\u003e\n- Update instructions: \u003chttps://github.com/kubernetes-sigs/metrics-server\u003e\n\n```bash\nhelm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/\nhelm repo update\nhelm upgrade --reuse-values metrics-server metrics-server/metrics-server -n kube-system --version '\u003cNEW_VERSION\u003e'\n```\n\n`values.yaml`:\n\n```yaml\n# Documentation: https://artifacthub.io/packages/helm/metrics-server/metrics-server\n```\n\n### Update System Upgrade Controller\n\n- Available versions:\n  \u003chttps://github.com/rancher/charts/tree/release-v2.12/charts/system-upgrade-controller\u003e\n- Update instructions: \u003chttps://github.com/rancher/system-upgrade-controller\u003e\n\n```bash\nhelm repo add rancher https://charts.rancher.io\nhelm repo update\nhelm upgrade --reuse-values system-upgrade-controller rancher/system-upgrade-controller -n cattle-system --version '\u003cNEW_VERSION\u003e'\n```\n\n`values.yaml`:\n\n```yaml\n# Documentation: https://github.com/rancher/system-upgrade-controller\n# Documentation: https://github.com/rancher/charts/tree/release-v2.8/charts/system-upgrade-controller\n\nsystemUpgradeJobTTLSecondsAfterFinish: 86400 # retain jobs for 1 day for inspection purposes\n```\n\n## Deletion\n\nAfter applying the Terraform plan you'll see several output variables like the\nload balancer's, control plane's, and node pools' IP addresses.\n\n```bash\nterraform destroy -force\n```\n\nBe sure to clean up any CSI created Block Storage Volumes, and CCM created\nLoadBalancers that you no longer require.\n\n## Troubleshooting\n\n### Gateway\n\nEnsure gateway is set up correctly: `./ssh-node gateway`\n\n#### Verify packet masquerading is set up properly\n\n```bash\niptables -L -t nat\n\n# Expected output:\n# Chain PREROUTING (policy ACCEPT)\n# target     prot opt source               destination\n#\n# Chain INPUT (policy ACCEPT)\n# target     prot opt source               destination\n#\n# Chain OUTPUT (policy ACCEPT)\n# target     prot opt source               destination\n#\n# Chain POSTROUTING (policy ACCEPT)\n# target     prot opt source               destination\n# MASQUERADE  all  --  10.0.1.0/24          anywhere\n```\n\n#### Verify firewall is set up properly\n\n```bash\nufw status\n\n# Expected output:\n# Status: active\n#\n# To                         Action      From\n# --                         ------      ----\n# 22,6443/tcp                ALLOW       Anywhere\n# 22,6443/tcp (v6)           ALLOW       Anywhere (v6)\n#\n# Anywhere on eth0           ALLOW FWD   Anywhere on enp7s0\n# Anywhere (v6) on eth0      ALLOW FWD   Anywhere (v6) on enp7s0\n```\n\n### Nodes\n\n#### Inspect local firewall settings\n\n```bash\nufw status\n```\n\n#### Verify correctness of date/timezone and locale\n\n```bash\ndate\n\necho $LANG\n```\n\n#### Inspect cloud-init logs\n\n```bash\n# Retrieve status\ncloud-init status\n\n# Verify configuration\ncloud-init schema --system\n\n# Collect logs for inspection\ncloud-init collect-logs\ntar xvzf cloud-init.tar.gz\n# Inspect cloud-init.log for error messages\n\n# Quickly find runcmd\nfind /var/lib/cloud/instances -name runcmd\nsh -ex PATH_TO_RUNCMD\n```\n\n### Cluster\n\nEnsure cluster is set up correctly: `./ssh-node cluster`\n\n#### Verify default route\n\n```bash\nip r s\n\n# Expected output:\n# default via 10.0.0.1 dev enp7s0 proto static onlink \u003c-- this is the important line\n# 10.0.0.0/8 via 10.0.0.1 dev enp7s0 proto dhcp src 10.0.1.2 metric 1024\n# 10.0.0.1 dev enp7s0 proto dhcp scope link src 10.0.1.2 metric 1024\n# 169.254.169.254 via 10.0.0.1 dev enp7s0 proto dhcp src 10.0.1.2 metric 1024\n```\n\n#### Verify connectivity to the Internet\n\n```bash\nping 1.1.1.1\n\n# Expected output:\n# PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.\n# 64 bytes from 1.1.1.1: icmp_seq=1 ttl=53 time=4.60 ms\n# 64 bytes from 1.1.1.1: icmp_seq=2 ttl=53 time=6.82 ms\n# ...\n```\n\n#### Verify name resolution\n\n```bash\nhost k3s.io\n\n# Expected output:\n# k3s.io has address 185.199.108.153\n# k3s.io has address 185.199.110.153\n# k3s.io has address 185.199.111.153\n# k3s.io has address 185.199.109.153\n# ...\n```\n\n#### Verify cluster status\n\n```bash\nk3s kubectl get nodes\n\n# Expected output:\n# k3s.io has address 185.199.108.153\n# k3s.io has address 185.199.110.153\n# k3s.io has address 185.199.111.153\n# k3s.io has address 185.199.109.153\n# ...\n```\n\n#### Verify Cilium Networking Status\n\nThis command only works after installing the\n[cilium cli](https://github.com/cilium/cilium-cli).\n\n- [cilium cli](https://www.klarmobil.de/)\n\n```bash\ncilium status\n\n# Expected output:\n#     /¯¯\\\n#  /¯¯\\__/¯¯\\    Cilium:             OK\n#  \\__/¯¯\\__/    Operator:           OK\n#  /¯¯\\__/¯¯\\    Envoy DaemonSet:    disabled (using embedded mode)\n#  \\__/¯¯\\__/    Hubble Relay:       disabled\n#     \\__/       ClusterMesh:        disabled\n#\n# Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1\n# DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3\n# Containers:            cilium             Running: 3\n#                        cilium-operator    Running: 1\n# Cluster Pods:          9/9 managed by Cilium\n# Helm chart version:    1.14.5\n# Image versions         cilium             quay.io/cilium/cilium:v1.14.5@sha256:d3b287029755b6a47dee01420e2ea469469f1b174a2089c10af7e5e9289ef05b: 3\n#                        cilium-operator    quay.io/cilium/operator-generic:v1.14.5@sha256:303f9076bdc73b3fc32aaedee64a14f6f44c8bb08ee9e3956d443021103ebe7a: 1\n```\n\n#### Verify k3s Cluster Configuration\n\nThis command only works out of the box on the first node of the control plane\nnode pool with the `cluster_can_init` setting.\n\n```bash\nk3s check-config\n\n# Expected output:\n# ...\n# STATUS: pass\n```\n\n#### Inspect cluster status and logs\n\n```bash\nsystemctl status k3s.service\n\njournalctl -u k3s.service\n```\n\n## Related Documentation\n\n- [Cilium](https://docs.cilium.io/)\n- [Cloud-init](https://cloudinit.readthedocs.io/)\n- [Hetzner API](https://docs.hetzner.cloud/)\n- [Hetzner Cloud Platform](https://docs.hetzner.com/cloud)\n- [Terraform Module Registry](https://registry.terraform.io/)\n- [Terraform](https://www.terraform.io/docs/)\n\n## Similar Projects\n\n- [hcloud-kube-hetzner](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner)\n  very popular k3s stack based on openSUSE MicroOS.\n- [hcloud-k3s](https://github.com/cicdteam/terraform-hcloud-k3s) Original\n  project that this project has been forked from.\n- [hetzner-cloud-k3s](https://github.com/vitobotta/hetzner-cloud-k3s) A fully\n  functional, super cheap Kubernetes cluster in Hetzner Cloud in 1m30s or less\n  - Not terraform-based.\n  - Scripts that make it easy to manage a cluster.\n- [hetzner-k3s](https://github.com/vitobotta/hetzner-k3s/) A CLI tool to create\n  and manage Kubernetes clusters in Hetzner Cloud using the lightweight\n  distribution k3s by Rancher. Successor of\n  [hetzner-cloud-k3s](https://github.com/vitobotta/hetzner-cloud-k3s).\n  - Not terraform-based.\n- [k-andy](https://github.com/StarpTech/k-andy) Zero friction Kubernetes stack\n  on Hetzner Cloud.\n  - Terraform-based stack.\n  - Distributed across multiple Hetzner sites and data centers.\n  - Support for multiple control-plane servers.\n- [terraform-hcloud-kube-hetzner](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner).\n  Optimized and Maintenance-free Kubernetes on Hetzner Cloud in one command!\n\n## Special Thanks\n\n- The initiators of and contributors to this project for getting the k3s cluster\n  running via terraform.\n- And to God for providing and enabling me to do my share of this work. Solo Deo\n  Gloria.\n\n\u003cscript async src=\"https://scripts.simpleanalyticscdn.com/latest.js\"\u003e\u003c/script\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidentiops%2Fterraform-hcloud-k3s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidentiops%2Fterraform-hcloud-k3s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidentiops%2Fterraform-hcloud-k3s/lists"}