{"id":14957866,"url":"https://github.com/garutilorenzo/k3s-oci-cluster","last_synced_at":"2025-04-06T02:08:21.209Z","repository":{"id":44873018,"uuid":"461957674","full_name":"garutilorenzo/k3s-oci-cluster","owner":"garutilorenzo","description":"Deploy a Kubernetes cluster for free, using k3s and Oracle always free resources","archived":false,"fork":false,"pushed_at":"2024-09-06T11:04:17.000Z","size":134,"stargazers_count":234,"open_issues_count":3,"forks_count":81,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-30T00:08:24.180Z","etag":null,"topics":["automation","iac","k3s","k3s-cluster","kubernetes","kubernetes-cluster","oci","oracle","oracle-cloud","oracle-cloud-infrastructure","terraform","terraform-module"],"latest_commit_sha":null,"homepage":"https://garutilorenzo.github.io/deploy-kubernetes-for-free-oracle-cloud","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/garutilorenzo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"security.tf","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"garutilorenzo"}},"created_at":"2022-02-21T17:07:57.000Z","updated_at":"2025-03-21T13:45:02.000Z","dependencies_parsed_at":"2024-09-26T19:00:29.627Z","dependency_job_id":"834ba0df-44b9-4f91-9987-2ef755a5fa40","html_url":"https://github.com/garutilorenzo/k3s-oci-cluster","commit_stats":{"total_commits":109,"total_committers":8,"mean_commits":13.625,"dds":"0.12844036697247707","last_synced_commit":"be490005e3350485a2e538eacd350a034f5338cb"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garutilorenzo%2Fk3s-oci-cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garutilorenzo%2Fk3s-oci-cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garutilorenzo%2Fk3s-oci-cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garutilorenzo%2Fk3s-oci-cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/garutilorenzo","download_url":"https://codeload.github.com/garutilorenzo/k3s-oci-cluster/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247423514,"owners_count":20936626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","iac","k3s","k3s-cluster","kubernetes","kubernetes-cluster","oci","oracle","oracle-cloud","oracle-cloud-infrastructure","terraform","terraform-module"],"created_at":"2024-09-24T13:15:44.131Z","updated_at":"2025-04-06T02:08:21.189Z","avatar_url":"https://github.com/garutilorenzo.png","language":"HCL","funding_links":["https://github.com/sponsors/garutilorenzo"],"categories":[],"sub_categories":[],"readme":"[![Wordpress CI](https://github.com/garutilorenzo/k3s-oci-cluster/actions/workflows/ci.yml/badge.svg)](https://github.com/garutilorenzo/k3s-oci-cluster/actions/workflows/ci.yml)\n[![GitHub issues](https://img.shields.io/github/issues/garutilorenzo/k3s-oci-cluster)](https://github.com/garutilorenzo/k3s-oci-cluster/issues)\n![GitHub](https://img.shields.io/github/license/garutilorenzo/k3s-oci-cluster)\n[![GitHub forks](https://img.shields.io/github/forks/garutilorenzo/k3s-oci-cluster)](https://github.com/garutilorenzo/k3s-oci-cluster/network)\n[![GitHub stars](https://img.shields.io/github/stars/garutilorenzo/k3s-oci-cluster)](https://github.com/garutilorenzo/k3s-oci-cluster/stargazers)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://garutilorenzo.github.io/images/k3s-logo-large.png?\" alt=\"k3s Logo\"/\u003e\n\u003c/p\u003e\n\n# OCI K3s cluster\n\nDeploy a Kubernetes cluster for free, using K3s and Oracle [always free](https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier_topic-Always_Free_Resources.htm) resources.\n\n# Table of Contents\n\n- [OCI K3s cluster](#oci-k3s-cluster)\n- [Table of Contents](#table-of-contents)\n    - [Important notes](#important-notes)\n    - [Requirements](#requirements)\n    - [Supported OS](#supported-os)\n    - [Terraform OCI user creation (Optional)](#terraform-oci-user-creation-optional)\n      - [Example RSA key generation](#example-rsa-key-generation)\n    - [Project setup](#project-setup)\n    - [Oracle provider setup](#oracle-provider-setup)\n    - [Pre flight checklist](#pre-flight-checklist)\n      - [How to find the availability domain name](#how-to-find-the-availability-domain-name)\n      - [How to list all the OS images](#how-to-list-all-the-os-images)\n  - [Notes about OCI always free resources](#notes-about-oci-always-free-resources)\n  - [Notes about K3s](#notes-about-k3s)\n  - [Infrastructure overview](#infrastructure-overview)\n  - [Cluster resource deployed](#cluster-resource-deployed)\n    - [Nginx ingress controller](#nginx-ingress-controller)\n    - [Cert-manager](#cert-manager)\n  - [Deploy](#deploy)\n      - [Public LB check](#public-lb-check)\n      - [Longhorn check](#longhorn-check)\n      - [Argocd check](#argocd-check)\n  - [Deploy a sample stack](#deploy-a-sample-stack)\n  - [Clean up](#clean-up)\n  - [Known Bugs](#known-bugs)\n    - [409-Conflict](#409-conflict)\n    - [kubectl exec failure](#kubectl-exec-failure)\n\n**Note** choose a region with enough ARM capacity\n\n### Important notes\n\n* This is repo shows only how to use terraform with the Oracle Cloud infrastructure and use only the **always free** resources. This examples are **not** for a production environment.\n* At the end of your trial period (30 days). All the paid resources deployed will be stopped/terminated\n* At the end of your trial period (30 days), if you have a running compute instance it will be stopped/hibernated\n\n### Requirements\n\nTo use this repo you will need:\n\n* an Oracle Cloud account. You can register [here](https://cloud.oracle.com)\n\nOnce you get the account, follow the *Before you begin* and *1. Prepare* step in [this](https://docs.oracle.com/en-us/iaas/developer-tutorials/tutorials/tf-provider/01-summary.htm) document.\n\n### Supported OS\n\nThis module was tested with:\n\n* Ubuntu 20.04, 22.04 (ubuntu remote user)\n* Ubuntu 20.04, 22.04 Minimal (ubuntu remote user)\n* Oracle Linux 8, 9 (opc remote user)\n\n### Terraform OCI user creation (Optional)\n\nIs always recommended to create a separate user and group in your preferred [domain](https://cloud.oracle.com/identity/domains) to use with Terraform.\nThis user must have less privileges possible (Zero trust policy). Below is an example policy that you can [create](https://cloud.oracle.com/identity/policies) (using the \"Advanced\" mode in Policy Builder) to allow `terraform-group` to manage all the resources needed by this module:\n\n```\nAllow group terraform-group to manage virtual-network-family  in compartment id \u003ccompartment_ocid\u003e\nAllow group terraform-group to manage instance-family  in compartment id \u003ccompartment_ocid\u003e\nAllow group terraform-group to manage compute-management-family  in compartment id \u003ccompartment_ocid\u003e\nAllow group terraform-group to manage volume-family  in compartment id \u003ccompartment_ocid\u003e\nAllow group terraform-group to manage load-balancers  in compartment id \u003ccompartment_ocid\u003e\nAllow group terraform-group to manage network-load-balancers  in compartment id \u003ccompartment_ocid\u003e\nAllow group terraform-group to manage dynamic-groups in compartment id \u003ccompartment_ocid\u003e\nAllow group terraform-group to manage policies in compartment id \u003ccompartment_ocid\u003e\nAllow group terraform-group to read network-load-balancers  in compartment id \u003ccompartment_ocid\u003e\nAllow group terraform-group to manage dynamic-groups in tenancy\n```\n\nSee [how](#oracle-provider-setup) to find the compartment ocid. The user and the group have to be manually created before using this module.\nTo create the user go to **Identity \u0026 Security -\u003e Users**, then create the group in **Identity \u0026 Security -\u003e Groups** and associate the newly created user to the group. The last step is to create the policy in **Identity \u0026 Security -\u003e Policies**.\n\n#### Example RSA key generation\n\nTo use terraform with the Oracle Cloud infrastructure you need to generate an RSA key. Generate the rsa key with:\n\n```\nopenssl genrsa -out ~/.oci/\u003cyour_name\u003e-oracle-cloud.pem 4096\nchmod 600 ~/.oci/\u003cyour_name\u003e-oracle-cloud.pem\nopenssl rsa -pubout -in ~/.oci/\u003cyour_name\u003e-oracle-cloud.pem -out ~/.oci/\u003cyour_name\u003e-oracle-cloud_public.pem\n```\nreplace `\u003cyour_name\u003e` with your name or a string you prefer.\n\nOnce generated, the RSA key can be uploaded in **Identity \u0026 Security -\u003e Domains -\u003e {domain} -\u003e Users -\u003e {user} -\u003e API keys** and the given `fingerprint` will be your `\u003cfingerprint\u003e` in `terraform.tfvars`\n\n**NOTE**: `~/.oci/\u003cyour_name\u003e-oracle-cloud_public.pem` will be used in `terraform.tfvars` by the Oracle provider plugin, so please take note of this string.\n\n### Project setup\n\nClone this repo and go in the `example/` directory:\n\n```\ngit clone https://github.com/garutilorenzo/k3s-oci-cluster.git\ncd k3s-oci-cluster/example/\n```\n\nNow you have to edit the `main.tf` file and you have to create the `terraform.tfvars` file. For more detail see [Oracle provider setup](#oracle-provider-setup) and [Pre flight checklist](#pre-flight-checklist).\n\n#### Use this repository as template\n\nIf you prefer you can create an new empty directory in your workspace and create this three files:\n\n* `terraform.tfvars` - More details in [Oracle provider setup](#oracle-provider-setup)\n* `main.tf`\n* `provider.tf`\n\nThe `main.tf` file will look like:\n\n```\nvariable \"compartment_ocid\" {}\nvariable \"tenancy_ocid\" {}\nvariable \"user_ocid\" {}\nvariable \"fingerprint\" {}\nvariable \"private_key_path\" {}\nvariable \"public_key_path\" {}\nvariable \"availability_domain\" {}\nvariable \"my_public_ip_cidr\" {}\nvariable \"cluster_name\" {}\nvariable \"os_image_id\" {}\nvariable \"certmanager_email_address\" {}\nvariable \"k3s_server_pool_size\" {\n  default = 1\n}\nvariable \"k3s_worker_pool_size\" {\n  default = 2\n}\nvariable \"region\" {}\n\nmodule \"k3s_cluster\" {\n  # k3s_version               = \"v1.23.8+k3s2\" # Fix kubectl exec failure\n  # k3s_version               = \"v1.24.4+k3s1\" # Kubernetes version compatible with longhorn\n  region                    = var.region\n  availability_domain       = var.availability_domain\n  tenancy_ocid              = var.tenancy_ocid\n  compartment_ocid          = var.compartment_ocid\n  my_public_ip_cidr         = var.my_public_ip_cidr\n  cluster_name              = var.cluster_name\n  public_key_path           = var.public_key_path\n  environment               = \"staging\"\n  os_image_id               = var.os_image_id\n  certmanager_email_address = var.certmanager_email_address\n  k3s_server_pool_size      = var.k3s_server_pool_size\n  k3s_worker_pool_size      = var.k3s_worker_pool_size\n  ingress_controller        = \"nginx\"\n  source                    = \"github.com/garutilorenzo/k3s-oci-cluster\"\n}\n\noutput \"k3s_servers_ips\" {\n  value = module.k3s_cluster.k3s_servers_ips\n}\n\noutput \"k3s_workers_ips\" {\n  value = module.k3s_cluster.k3s_workers_ips\n}\n\noutput \"public_lb_ip\" {\n  value = module.k3s_cluster.public_lb_ip\n}\n```\n\nFor all the possible variables see [Pre flight checklist](#pre-flight-checklist)\n\nThe `provider.tf` will look like:\n\n```\nprovider \"oci\" {\n  tenancy_ocid     = var.tenancy_ocid\n  user_ocid        = var.user_ocid\n  private_key_path = var.private_key_path\n  fingerprint      = var.fingerprint\n  region           = var.region\n}\n```\n\n### Terraform initialization\n\nNow we can init terraform with:\n\n```\nterraform init\n\nterraform init\nInitializing modules...\nDownloading git::https://github.com/garutilorenzo/k3s-oci-cluster.git for k3s_cluster...\n- k3s_cluster in .terraform/modules/k3s_cluster\n\nInitializing the backend...\n\nInitializing provider plugins...\n- Reusing previous version of hashicorp/oci from the dependency lock file\n- Reusing previous version of hashicorp/template from the dependency lock file\n- Using previously-installed hashicorp/template v2.2.0\n- Using previously-installed hashicorp/oci v4.64.0\n\nTerraform has been successfully initialized!\n\nYou may now begin working with Terraform. Try running \"terraform plan\" to see\nany changes that are required for your infrastructure. All Terraform commands\nshould now work.\n\nIf you ever set or change modules or backend configuration for Terraform,\nrerun this command to reinitialize your working directory. If you forget, other\ncommands will detect it and remind you to do so if necessary.\n```\n\n### Oracle provider setup\n\nIn the `example/` directory of this repo you need to create a `terraform.tfvars` file, the file will look like:\n\n```\nfingerprint      = \"\u003crsa_key_fingerprint\u003e\"\nprivate_key_path = \"~/.oci/\u003cyour_name\u003e-oracle-cloud.pem\"\nuser_ocid        = \"\u003cuser_ocid\u003e\"\ntenancy_ocid     = \"\u003ctenancy_ocid\u003e\"\ncompartment_ocid = \"\u003ccompartment_ocid\u003e\"\n```\n\nTo find your `tenancy_ocid` in the Ocacle Cloud console go to: **Governance and Administration \u003e Tenancy details**, then copy the OCID.\n\nTo find you `user_ocid` in the Ocacle Cloud console go to **User setting** (click on the icon in the top right corner, then click on User settings), click your username and then copy the OCID.\n\nThe `compartment_ocid` is the same as `tenancy_ocid`.\n\nThe fingerprint is the fingerprint of your RSA key, you can find this vale under **User setting \u003e API Keys**.\n\n### Pre flight checklist\n\nOnce you have created the terraform.tfvars file edit the `main.tf` file (always in the `example/` directory) and set the following variables:\n\n| Var   | Required | Desc |\n| ------- | ------- | ----------- |\n| `region`       | `yes`       | set the correct region based on your needs (note that this requires the name, not the OCID. Example: `eu-frankfurt-1`) |\n| `availability_domain` | `yes`        | Set the correct availability domain. See [how](#how-to-find-the-availability-domain-name) to find the availability domain (note that this requires the name, not the OCID. Example: `TYPo:EU-FRANKFURT-1-AD-2`)|\n| `compartment_ocid` | `yes`        | Set the correct compartment ocid. See [how](#oracle-provider-setup) to find the compartment ocid |\n| `cluster_name` | `yes`        | the name of your K3s cluster. Default: k3s-cluster |\n| `my_public_ip_cidr` | `yes`        |  your local public IP in CIDR format (Example: `195.102.xxx.xxx/32`) |\n| `private_key_path`     | `yes`       | Path to your private **OCI RSA key** |\n| `environment`  | `yes`  | Current work environment (Example: staging/dev/prod). This value is used for tag all the deployed resources |\n| `os_image_id`  | `yes`  | Image id to use. See [how](#how-to-list-all-the-os-images) to list all available OS images |\n| `public_key_path`     | `no`       | Path to your public **workstation SSH key** |\n| `k3s_version`  | `no`  | K3s version. Default: latest |\n| `k3s_subnet`  | `no`  | Subnet where K3s will be exposed. Rquired if the subnet is different from the default gw subnet (Eg. 192.168.1.0/24). Default: default_route_table |\n| `compute_shape`  | `no`  | Compute shape to use. Default VM.Standard.A1.Flex. **NOTE** Is mandatory to use this compute shape for provision 4 always free VMs |\n| `oci_core_vcn_dns_label`  | `no`  | VCN DNS label. Default: defaultvcn |\n| `oci_core_subnet_dns_label10`  | `no`  | First subnet DNS label. Default: defaultsubnet10 |\n| `oci_core_subnet_dns_label11`  | `no`  | Second subnet DNS label. Default: defaultsubnet11 |\n| `oci_core_vcn_cidr`  | `no`  | VCN CIDR. Default: oci_core_vcn_cidr |\n| `oci_core_subnet_cidr10`  | `no`  | First subnet CIDR. Default: 10.0.0.0/24 |\n| `oci_core_subnet_cidr11`  | `no`  | Second subnet CIDR. Default: 10.0.1.0/24 |\n| `oci_identity_dynamic_group_name`  | `no`  | Dynamic group name. This dynamic group will contains all the instances of this specific compartment. Default: Compute_Dynamic_Group |\n| `oci_identity_policy_name`  | `no`  | Policy name. This policy will allow dynamic group 'oci_identity_dynamic_group_name' to read OCI api without auth. Default: Compute_To_Oci_Api_Policy |\n| `k3s_load_balancer_name`  | `no`  | Internal LB name. Default: k3s internal load balancer  |\n| `public_load_balancer_name`  | `no`  | Public LB name. Default: K3s public LB  |\n| `kube_api_port`  | `no`  | Kube api default port Default: 6443  |\n| `public_lb_shape`  | `no`  | LB shape for the public LB. Default: flexible. **NOTE** is mandatory to use this kind of shape to provision two always free LB (public and private)  |\n| `http_lb_port`  | `no`  | http port used by the public LB. Default: 80  |\n| `https_lb_port`  | `no`  | http port used by the public LB. Default: 443  |\n| `k3s_server_pool_size`  | `no`  | Number of k3s servers deployed. Default 1  |\n| `k3s_worker_pool_size`  | `no`  | Number of k3s workers deployed. Default 2  |\n| `k3s_extra_worker_node`  | `no`  | Boolean value, default true. Deploy the third worker nodes. The node will be deployed outside the worker instance pools. Using OCI always free account you can't create instance pools with more than two servers. This workaround solve this problem. |\n| `ingress_controller`  | `no`  | Define the ingress controller to use. Valid values are: [default](https://docs.k3s.io/networking#traefik-ingress-controller), [nginx](#nginx-ingress-controller), [traefik2](https://traefik.io/) or [istio](https://istio.io/latest/docs/tasks/traffic-management/ingress/kubernetes-ingress/) |\n| `disable_ingress`  | `no`  | Boolean value, disable all ingress controllers. Default: false |\n| `ingress_controller_http_nodeport`  | `no`  | NodePort where nginx ingress will listen for http traffic. Default 30080  |\n| `ingress_controller_https_nodeport`  | `no`  | NodePort where nginx ingress will listen for https traffic.  Default 30443 |\n| `install_longhorn`  | `no`  | Boolean value, install longhorn \"Cloud native distributed block storage for Kubernetes\". Default: true. To use longhorn set the *k3s_version* \u003c v1.25.x [Ref.](https://github.com/longhorn/longhorn/issues/4003)  |\n| `longhorn_release`  | `no`  | Longhorn release. Default: v1.4.0  |\n| `install_certmanager`  | `no`  | Boolean value, install [cert manager](https://cert-manager.io/) \"Cloud native certificate management\". Default: true  |\n| `nginx_ingress_release`  | `no`  | Longhorn release. Default: v1.5.1  |\n| `certmanager_release`  | `no`  | Cert manager release. Default: v1.11.0  |\n| `certmanager_email_address`  | `no`  | Email address used for signing https certificates. Defaul: changeme@example.com  |\n| `install_argocd`  | `no`  | Boolean value, install [Argo CD](https://argo-cd.readthedocs.io/en/stable/) \"a declarative, GitOps continuous delivery tool for Kubernetes.\". Default: true  |\n| `argocd_release`  | `no`  | Argo CD release. Default: v2.4.11  |\n| `install_argocd_image_updater`  | `no`  | Boolean value, install [Argo CD Image Updater](https://argocd-image-updater.readthedocs.io/en/stable/) \"A tool to automatically update the container images of Kubernetes workloads that are managed by Argo CD.\". Default: true  |\n| `argocd_image_updater_release`  | `no`  | Argo CD release Image Updater. Default: v0.12.0  |\n| `unique_tag_key`  | `no`  | Unique tag name used for tagging all the deployed resources. Default: k3s-provisioner |\n| `unique_tag_value`  | `no`  | Unique value used with  unique_tag_key. Default: https://github.com/garutilorenzo/k3s-oci-cluster |\n| `expose_kubeapi`  | `no`  | Boolean value, default false. Expose or not the kubeapi server to the internet. Access is granted only from *my_public_ip_cidr* for security reasons. |\n\n\n#### How to find the availability domain name\n\nTo find the list of the availability domains run this command on che Cloud Shell:\n\n```\noci iam availability-domain list\n{\n  \"data\": [\n    {\n      \"compartment-id\": \"\u003ccompartment_ocid\u003e\",\n      \"id\": \"ocid1.availabilitydomain.oc1..xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\",\n      \"name\": \"iAdc:EU-ZURICH-1-AD-1\"\n    }\n  ]\n}\n```\n\n#### How to list all the OS images\n\nTo filter the OS images by shape and OS run this command on che Cloud Shell. You can filter by OS: Canonical Ubuntu or Oracle Linux:\n\n```\noci compute image list --compartment-id \u003ccompartment_ocid\u003e --operating-system \"Canonical Ubuntu\" --shape \"VM.Standard.A1.Flex\"\n{\n  \"data\": [\n    {\n      \"agent-features\": null,\n      \"base-image-id\": null,\n      \"billable-size-in-gbs\": 2,\n      \"compartment-id\": null,\n      \"create-image-allowed\": true,\n      \"defined-tags\": {},\n      \"display-name\": \"Canonical-Ubuntu-20.04-aarch64-2022.01.18-0\",\n      \"freeform-tags\": {},\n      \"id\": \"ocid1.image.oc1.eu-zurich-1.aaaaaaaag2uyozo7266bmg26j5ixvi42jhaujso2pddpsigtib6vfnqy5f6q\",\n      \"launch-mode\": \"NATIVE\",\n      \"launch-options\": {\n        \"boot-volume-type\": \"PARAVIRTUALIZED\",\n        \"firmware\": \"UEFI_64\",\n        \"is-consistent-volume-naming-enabled\": true,\n        \"is-pv-encryption-in-transit-enabled\": true,\n        \"network-type\": \"PARAVIRTUALIZED\",\n        \"remote-data-volume-type\": \"PARAVIRTUALIZED\"\n      },\n      \"lifecycle-state\": \"AVAILABLE\",\n      \"listing-type\": null,\n      \"operating-system\": \"Canonical Ubuntu\",\n      \"operating-system-version\": \"20.04\",\n      \"size-in-mbs\": 47694,\n      \"time-created\": \"2022-01-27T22:53:34.270000+00:00\"\n    },\n```\n\n## Notes about OCI always free resources\n\nIn order to get the maximum resources available within the oracle always free tier, the max amount of the k3s servers and k3s workers must be 2. So the **max value** for `k3s_server_pool_size` and `k3s_worker_pool_size` is `2`.\n\nIn this setup we use two LB, one internal LB and one public LB (Layer 7). In order to use two LB using the always free resources, one lb must be a [network load balancer](https://docs.oracle.com/en-us/iaas/Content/NetworkLoadBalancer/introducton.htm#Overview) an the other must be a [load balancer](https://docs.oracle.com/en-us/iaas/Content/Balance/Concepts/balanceoverview.htm). The public LB **must** use the `flexible` shape (`public_lb_shape` variable).\n\n## Notes about K3s\n\nIn this environment the High Availability of the K3s cluster is provided using the Embedded DB. More details [here](https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/)\n\nThe default installation of K3s install [Traefik](https://docs.k3s.io/networking#traefik-ingress-controller) as ingress the controller. In this environment Traefik is replaced by [Nginx ingress controller](https://kubernetes.github.io/ingress-nginx/). To install Traefik as the ingress controller set the variable `ingress_controller` to `default`.\nFor more details on Nginx ingress controller see the [Nginx ingress controller](#nginxingress-controller) section.\n\n## Infrastructure overview\n\nThe final infrastructure will be made by:\n\n* two instance pool:\n  * one instance pool for the server nodes named `k3s-servers`\n  * one instance pool for the worker nodes named `k3s-workers`\n* one internal load balancer that will route traffic to K3s servers\n* one external load balancer that will route traffic to K3s workers\n\nThe other resources created by terraform are:\n\n* two instance configurations (one for the servers and one for the workers) used by the instance pools\n* one vcn\n* two public subnets\n* two security list\n* one dynamic group\n* one identity policy\n\n![k3s infra](https://garutilorenzo.github.io/images/k3s-oci-always-free.drawio.png?)\n\n## Cluster resource deployed\n\nThis setup will automatically install [longhorn](https://longhorn.io/). Longhorn is a *Cloud native distributed block storage for Kubernetes*. To disable the longhorn deployment set `install_longhorn` variable to `false`.\n\n**NOTE** to use longhorn set the `k3s_version` \u003c `v1.25.x` [Ref.](https://github.com/longhorn/longhorn/issues/4003)\n\n### Nginx ingress controller\n\nIn this environment [Nginx ingress controller](https://kubernetes.github.io/ingress-nginx/) is used instead of the standard [Traefik](https://docs.k3s.io/networking#traefik-ingress-controller) ingress controller.\n\nThe installation is the [bare metal](https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal-clusters) installation, the ingress controller then is exposed via a NodePort Service.\n\n```yaml\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: ingress-nginx-controller-loadbalancer\n  namespace: ingress-nginx\nspec:\n  selector:\n    app.kubernetes.io/component: controller\n    app.kubernetes.io/instance: ingress-nginx\n    app.kubernetes.io/name: ingress-nginx\n  ports:\n    - name: http\n      port: 80\n      protocol: TCP\n      targetPort: 80\n      nodePort: ${ingress_controller_http_nodeport} # default to 30080\n    - name: https\n      port: 443\n      protocol: TCP\n      targetPort: 443\n      nodePort: ${ingress_controller_https_nodeport} # default to 30443\n  type: NodePort\n```\n\nTo get the real ip address of the clients using a public L4 load balancer we need to use the proxy protocol feature of nginx ingress controller:\n\n```yaml\n---\napiVersion: v1\ndata:\n  allow-snippet-annotations: \"true\"\n  enable-real-ip: \"true\"\n  proxy-real-ip-cidr: \"0.0.0.0/0\"\n  proxy-body-size: \"20m\"\n  use-proxy-protocol: \"true\"\nkind: ConfigMap\nmetadata:\n  labels:\n    app.kubernetes.io/component: controller\n    app.kubernetes.io/instance: ingress-nginx\n    app.kubernetes.io/managed-by: Helm\n    app.kubernetes.io/name: ingress-nginx\n    app.kubernetes.io/part-of: ingress-nginx\n    app.kubernetes.io/version: 1.1.1\n    helm.sh/chart: ingress-nginx-4.0.16\n  name: ingress-nginx-controller\n  namespace: ingress-nginx\n```\n\n**NOTE** to use nginx ingress controller with the proxy protocol enabled, an external nginx instance is used as proxy (since OCI LB doesn't support proxy protocol at the moment). Nginx will be installed on each worker node and the configuation of nginx will:\n\n* listen in proxy protocol mode\n* forward the traffic from port `80` to `ingress_controller_http_nodeport` (default to `30080`) on any server of the cluster\n* forward the traffic from port `443` to `ingress_controller_https_nodeport` (default to `30443`) on any server of the cluster\n\nThis is the final result:\n\nClient -\u003e Public L4 LB -\u003e nginx proxy (with proxy protocol enabled) -\u003e nginx ingress (with proxy protocol enabled) -\u003e k3s service -\u003e pod(s)\n\n### Cert-manager\n\n[cert-manager](https://cert-manager.io/docs/) is used to issue certificates from a variety of supported source. To use cert-manager take a look at [nginx-ingress-cert-manager.yml](deployments/nginx/nginx-ingress-cert-manager.yml) and [nginx-configmap-cert-manager.yml](deployments/nginx/nginx-configmap-cert-manager.yml) example. To use cert-manager and get the certificate you **need** set on your DNS configuration the public ip address of the load balancer.\n\n## Deploy\n\nWe are now ready to deploy our infrastructure. First we ask terraform to plan the execution with:\n\n```\nterraform plan\n\n...\n...\n      + id                             = (known after apply)\n      + ip_addresses                   = (known after apply)\n      + is_preserve_source_destination = false\n      + is_private                     = true\n      + lifecycle_details              = (known after apply)\n      + nlb_ip_version                 = (known after apply)\n      + state                          = (known after apply)\n      + subnet_id                      = (known after apply)\n      + system_tags                    = (known after apply)\n      + time_created                   = (known after apply)\n      + time_updated                   = (known after apply)\n\n      + reserved_ips {\n          + id = (known after apply)\n        }\n    }\n\nPlan: 27 to add, 0 to change, 0 to destroy.\n\nChanges to Outputs:\n  + k3s_servers_ips = [\n      + (known after apply),\n      + (known after apply),\n    ]\n  + k3s_workers_ips = [\n      + (known after apply),\n      + (known after apply),\n    ]\n  + public_lb_ip    = (known after apply)\n\n──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n\nNote: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run \"terraform apply\" now.\n```\n\nnow we can deploy our resources with:\n\n```\nterraform apply\n\n...\n...\n      + is_preserve_source_destination = false\n      + is_private                     = true\n      + lifecycle_details              = (known after apply)\n      + nlb_ip_version                 = (known after apply)\n      + state                          = (known after apply)\n      + subnet_id                      = (known after apply)\n      + system_tags                    = (known after apply)\n      + time_created                   = (known after apply)\n      + time_updated                   = (known after apply)\n\n      + reserved_ips {\n          + id = (known after apply)\n        }\n    }\n\nPlan: 27 to add, 0 to change, 0 to destroy.\n\nChanges to Outputs:\n  + k3s_servers_ips = [\n      + (known after apply),\n      + (known after apply),\n    ]\n  + k3s_workers_ips = [\n      + (known after apply),\n      + (known after apply),\n    ]\n  + public_lb_ip    = (known after apply)\n\n  Do you want to perform these actions?\n  Terraform will perform the actions described above.\n  Only 'yes' will be accepted to approve.\n  Enter a value: yes\n\n...\n...\n\nmodule.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [50s elapsed]\nmodule.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Still creating... [1m0s elapsed]\nmodule.k3s_cluster.oci_network_load_balancer_backend.k3s_kube_api_backend[0]: Creation complete after 1m1s [...]\n\nApply complete! Resources: 27 added, 0 changed, 0 destroyed.\n\nOutputs:\n\nk3s_servers_ips = [\n  \"X.X.X.X\",\n  \"X.X.X.X\",\n]\nk3s_workers_ips = [\n  \"X.X.X.X\",\n  \"X.X.X.X\",\n]\npublic_lb_ip = tolist([\n  \"X.X.X.X\",\n])\n```\n\nNow on one master node you can check the status of the cluster with:\n\n```\nssh X.X.X.X -lubuntu\n\nubuntu@inst-iwlqz-k3s-servers:~$ sudo su -\nroot@inst-iwlqz-k3s-servers:~# kubectl get nodes\n\nNAME                     STATUS   ROLES                       AGE     VERSION\ninst-axdzf-k3s-workers   Ready    \u003cnone\u003e                      4m34s   v1.22.6+k3s1\ninst-hmgnl-k3s-servers   Ready    control-plane,etcd,master   4m14s   v1.22.6+k3s1\ninst-iwlqz-k3s-servers   Ready    control-plane,etcd,master   6m4s    v1.22.6+k3s1\ninst-lkvem-k3s-workers   Ready    \u003cnone\u003e                      5m35s   v1.22.6+k3s1\n```\n\n#### Public LB check\n\nWe can now test the public load balancer, nginx ingress controller and the security list ingress rules. On your local PC run:\n\n```\ncurl -v http://\u003cPUBLIC_LB_IP\u003e\n\n*   Trying PUBLIC_LB_IP:80...\n* TCP_NODELAY set\n* Connected to PUBLIC_LB_IP (PUBLIC_LB_IP) port 80 (#0)\n\u003e GET / HTTP/1.1\n\u003e Host: PUBLIC_LB_IP\n\u003e User-Agent: curl/7.68.0\n\u003e Accept: */*\n\u003e \n* Mark bundle as not supporting multiuse\n\u003c HTTP/1.1 404 Not Found\n\u003c Date: Fri, 25 Feb 2022 14:03:09 GMT\n\u003c Content-Type: text/html\n\u003c Content-Length: 146\n\u003c Connection: keep-alive\n\u003c \n\u003chtml\u003e\n\u003chead\u003e\u003ctitle\u003e404 Not Found\u003c/title\u003e\u003c/head\u003e\n\u003cbody\u003e\n\u003ccenter\u003e\u003ch1\u003e404 Not Found\u003c/h1\u003e\u003c/center\u003e\n\u003chr\u003e\u003ccenter\u003enginx\u003c/center\u003e\n\u003c/body\u003e\n\u003c/html\u003e\n* Connection #0 to host PUBLIC_LB_IP left intact\n```\n\n`404` is a correct response since the cluster is empty. We can test also the https listener/backends:\n\n```\ncurl -k -v https://\u003cPUBLIC_LB_IP\u003e\n\n* Trying PUBLIC_LB_IP:443...\n* TCP_NODELAY set\n* Connected to PUBLIC_LB_IP (PUBLIC_LB_IP) port 443 (#0)\n* ALPN, offering h2\n* ALPN, offering http/1.1\n* successfully set certificate verify locations:\n*   CAfile: /etc/ssl/certs/ca-certificates.crt\n  CApath: /etc/ssl/certs\n* TLSv1.3 (OUT), TLS handshake, Client hello (1):\n* TLSv1.3 (IN), TLS handshake, Server hello (2):\n* TLSv1.2 (IN), TLS handshake, Certificate (11):\n* TLSv1.2 (IN), TLS handshake, Server key exchange (12):\n* TLSv1.2 (IN), TLS handshake, Server finished (14):\n* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):\n* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):\n* TLSv1.2 (OUT), TLS handshake, Finished (20):\n* TLSv1.2 (IN), TLS handshake, Finished (20):\n* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384\n* ALPN, server accepted to use http/1.1\n* Server certificate:\n*  subject: C=IT; ST=Italy; L=Brescia; O=GL Ltd; OU=IT; CN=testlb.domainexample.com; emailAddress=email@you.com\n*  start date: Feb 25 10:28:29 2022 GMT\n*  expire date: Feb 25 10:28:29 2023 GMT\n*  issuer: C=IT; ST=Italy; L=Brescia; O=GL Ltd; OU=IT; CN=testlb.domainexample.com; emailAddress=email@you.com\n*  SSL certificate verify result: self signed certificate (18), continuing anyway.\n\u003e GET / HTTP/1.1\n\u003e Host: PUBLIC_LB_IP\n\u003e User-Agent: curl/7.68.0\n\u003e Accept: */*\n\u003e \n* Mark bundle as not supporting multiuse\n\u003c HTTP/1.1 404 Not Found\n\u003c Date: Fri, 25 Feb 2022 13:48:19 GMT\n\u003c Content-Type: text/html\n\u003c Content-Length: 146\n\u003c Connection: keep-alive\n\u003c \n\u003chtml\u003e\n\u003chead\u003e\u003ctitle\u003e404 Not Found\u003c/title\u003e\u003c/head\u003e\n\u003cbody\u003e\n\u003ccenter\u003e\u003ch1\u003e404 Not Found\u003c/h1\u003e\u003c/center\u003e\n\u003chr\u003e\u003ccenter\u003enginx\u003c/center\u003e\n\u003c/body\u003e\n\u003c/html\u003e\n* Connection #0 to host PUBLIC_LB_IP left intact\n```\n\n#### Longhorn check\n\nTo check if longhorn was successfully installed run on one master nodes:\n\n```\nkubectl get ns\nNAME              STATUS   AGE\ndefault           Active   9m40s\nkube-node-lease   Active   9m39s\nkube-public       Active   9m39s\nkube-system       Active   9m40s\nlonghorn-system   Active   8m52s   \u003c- longhorn namespace \n\n\nroot@inst-hmgnl-k3s-servers:~# kubectl get pods -n longhorn-system\nNAME                                        READY   STATUS    RESTARTS        AGE\ncsi-attacher-5f46994f7-8w9sg                1/1     Running   0               7m52s\ncsi-attacher-5f46994f7-qz7d4                1/1     Running   0               7m52s\ncsi-attacher-5f46994f7-rjqlx                1/1     Running   0               7m52s\ncsi-provisioner-6ccbfbf86f-fw7q4            1/1     Running   0               7m52s\ncsi-provisioner-6ccbfbf86f-gwmrg            1/1     Running   0               7m52s\ncsi-provisioner-6ccbfbf86f-nsf84            1/1     Running   0               7m52s\ncsi-resizer-6dd8bd4c97-7l67f                1/1     Running   0               7m51s\ncsi-resizer-6dd8bd4c97-g66wj                1/1     Running   0               7m51s\ncsi-resizer-6dd8bd4c97-nksmd                1/1     Running   0               7m51s\ncsi-snapshotter-86f65d8bc-2gcwt             1/1     Running   0               7m50s\ncsi-snapshotter-86f65d8bc-kczrw             1/1     Running   0               7m50s\ncsi-snapshotter-86f65d8bc-sjmnv             1/1     Running   0               7m50s\nengine-image-ei-fa2dfbf0-6rpz2              1/1     Running   0               8m30s\nengine-image-ei-fa2dfbf0-7l5k8              1/1     Running   0               8m30s\nengine-image-ei-fa2dfbf0-7nph9              1/1     Running   0               8m30s\nengine-image-ei-fa2dfbf0-ndkck              1/1     Running   0               8m30s\ninstance-manager-e-31a0b3f5                 1/1     Running   0               8m26s\ninstance-manager-e-37aa4663                 1/1     Running   0               8m27s\ninstance-manager-e-9cc7cc9d                 1/1     Running   0               8m20s\ninstance-manager-e-f39d9f2c                 1/1     Running   0               8m29s\ninstance-manager-r-1364d994                 1/1     Running   0               8m26s\ninstance-manager-r-c1670269                 1/1     Running   0               8m20s\ninstance-manager-r-c20ebeb3                 1/1     Running   0               8m28s\ninstance-manager-r-c54bf9a5                 1/1     Running   0               8m27s\nlonghorn-csi-plugin-2qj94                   2/2     Running   0               7m50s\nlonghorn-csi-plugin-4t8jm                   2/2     Running   0               7m50s\nlonghorn-csi-plugin-ws82l                   2/2     Running   0               7m50s\nlonghorn-csi-plugin-zmc9q                   2/2     Running   0               7m50s\nlonghorn-driver-deployer-784546d78d-s6cd2   1/1     Running   0               8m58s\nlonghorn-manager-l8sd8                      1/1     Running   0               9m1s\nlonghorn-manager-r2q5c                      1/1     Running   1 (8m30s ago)   9m1s\nlonghorn-manager-s6wql                      1/1     Running   0               9m1s\nlonghorn-manager-zrrf2                      1/1     Running   0               9m\nlonghorn-ui-9fdb94f9-6shsr                  1/1     Running   0               8m59s\n```\n\n#### Argocd check\n\nYou can verify that all pods are running:\n```\nroot@inst-hmgnl-k3s-servers:~# kubectl get pods -n argocd\nNAME                                                READY   STATUS    RESTARTS   AGE\nargocd-application-controller-0                     1/1     Running   0          8m51s\nargocd-applicationset-controller-7b74965f8c-mjl97   1/1     Running   0          8m53s\nargocd-dex-server-7f75d56bc6-j62hb                  1/1     Running   0          8m53s\nargocd-notifications-controller-54dd686846-lggrz    1/1     Running   0          8m53s\nargocd-redis-5dff748d9c-s5q2l                       1/1     Running   0          8m52s\nargocd-repo-server-5576f8d84b-sgbbt                 1/1     Running   0          8m52s\nargocd-server-76cf7d4c7b-jq9qx                      1/1     Running   0          8m52s\n```\n\nTo fetch the initial admin password, to be able to do this you need to expose your kubeapi-server (set *expose_kubeapi* variable to ture) and fetch the\nkubeconfig from one of the server nodes, it will be in (/var/lib/rancher/k3s/server/cred/admin.kubeconfig): \n\n```\nkubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\"{.data.password}\" | base64 -d\n```\n\nTo connect to the UI (make sure to copy the kubeconfig to your local machine first):\n\n```\nkubectl -n argocd port-forward service/argocd-server -n argocd 8080:443\n```\n\nAfter that you should be able to visit the ArgoCD UI: https://localhost:8080\n\n\n## Deploy a sample stack\n\nFinally to test all the components of the cluster we can deploy a sample stack. The stack is composed by the following components:\n\n* MariaDB\n* Nginx\n* Wordpress\n\nEach component is made by: one deployment and one service.\nWordpress and nginx share the same persistent volume (ReadWriteMany with longhorn storage class). The nginx configuration is stored in four ConfigMaps and  the nginx service is exposed by the nginx ingress controller.\n\nDeploy the resources with:\n\n```\nkubectl apply -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/mariadb/all-resources.yml\nkubectl apply -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/wordpress/all-resources.yml\n```\n\n**NOTE** The Wordpress installation is **secured**. To allow external traffic to `/wp-admin`, `/xmlrpc.php` and `wp-login.php` you have to edit the  [deployments/nginx/all-resources.yml](https://github.com/garutilorenzo/k3s-oci-cluster/blob/master/deployments/nginx/all-resources.yml) and change this line:\n\n```yaml\n- name: SECURE_SUBNET\n  value: 8.8.8.8/32 # change-me\n```\n\nwith your public ip address CIDR.\n\n```\ncurl -o nginx-all-resources.yml https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/nginx/all-resources.yml\n\nvi nginx-all-resources.yml\n\nchange SECURE_SUBNET and save the file\n\nkubectl apply -f nginx-all-resources.yml\n```\n\nnow check the status:\n\n```\nkubectl get deployments\nNAME        READY   UP-TO-DATE   AVAILABLE   AGE\nmariadb       1/1     1            1           92m\nnginx         1/1     1            1           79m\nwordpress     1/1     1            1           91m\n\nkubectl get svc\nNAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE\nkubernetes        ClusterIP   10.43.0.1       \u003cnone\u003e        443/TCP    5h8m\nmariadb-svc       ClusterIP   10.43.184.188   \u003cnone\u003e        3306/TCP   92m\nnginx-svc         ClusterIP   10.43.9.202     \u003cnone\u003e        80/TCP     80m\nwordpress-svc     ClusterIP   10.43.242.26    \u003cnone\u003e        9000/TCP   91m\n```\n\nNow you are ready to setup WP, open the LB public ip and follow the wizard. **NOTE** nginx and the Kubernetes Ingress rule are configured without virthual host/server name.\n\n![k3s wp install](https://garutilorenzo.github.io/images/k3s-wp.png?)\n\nTo clean the deployed resources:\n\n```\nkubectl delete -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/mariadb/all-resources.yml\nkubectl delete -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/nginx/all-resources.yml\nkubectl delete -f https://raw.githubusercontent.com/garutilorenzo/k3s-oci-cluster/master/deployments/wordpress/all-resources.yml\n```\n\n## Clean up\n\n```\nterraform destroy\n```\n\n## Known Bugs\n\n### 409-Conflict\n\nIf you see this error during the infrastructure destruction:\n\n```\nError: 409-Conflict, Invalid State Transition of NLB lifeCycle state from Updating to Updating\n│ Suggestion: The resource is in a conflicted state. Please retry again or contact support for help with service: Network Load Balancer Listener\n│ Documentation: https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/network_load_balancer_listener \n│ API Reference: https://docs.oracle.com/iaas/api/#/en/networkloadbalancer/20200501/Listener/DeleteListener \n```\n\nre-run `terraform destroy`\n\n### kubectl exec failure\n\nThe runc version in k3s containerd version 1.6.6 contains a regression that prevents anyone from executing a command and attaching to the container's TTY (exec -it) whenever someone runs systemctl daemon-reload. Alternatively, the user may run into this issue on SELinux-enforced systems. [Ref](https://github.com/k3s-io/k3s/issues/6064).\n\n```\nkubectl exec -it -n kube-system cilium-6lqp9 -- cilium status\nDefaulted container \"cilium-agent\" out of: cilium-agent, mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init)\nerror: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"b67e6e00172071996430dac5c97352e4d0c9fa3b3888e8daece5197c4649b4d1\": OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown\n```\n\nTo solve this issue downgrade to k3s v1.23\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgarutilorenzo%2Fk3s-oci-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgarutilorenzo%2Fk3s-oci-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgarutilorenzo%2Fk3s-oci-cluster/lists"}