{"id":30252062,"url":"https://github.com/nce/oci-free-cloud-k8s","last_synced_at":"2026-03-12T03:38:42.305Z","repository":{"id":41189224,"uuid":"507862536","full_name":"nce/oci-free-cloud-k8s","owner":"nce","description":"always free kubernetes on oracle cloud","archived":false,"fork":false,"pushed_at":"2026-03-09T22:18:52.000Z","size":766,"stargazers_count":229,"open_issues_count":9,"forks_count":41,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-03-10T04:39:56.705Z","etag":null,"topics":["free","helm","k8s","kubernetes","oci","oke","oracle-cloud","oracle-cloud-infrastructure","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-06-27T10:26:52.000Z","updated_at":"2026-03-08T13:28:01.000Z","dependencies_parsed_at":"2023-10-16T10:40:00.900Z","dependency_job_id":"b9bcc8d6-4c36-40c7-9a17-3873a1515985","html_url":"https://github.com/nce/oci-free-cloud-k8s","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nce/oci-free-cloud-k8s","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nce%2Foci-free-cloud-k8s","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nce%2Foci-free-cloud-k8s/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nce%2Foci-free-cloud-k8s/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nce%2Foci-free-cloud-k8s/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nce","download_url":"https://codeload.github.com/nce/oci-free-cloud-k8s/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nce%2Foci-free-cloud-k8s/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30414497,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-12T00:40:14.898Z","status":"online","status_checked_at":"2026-03-12T02:00:07.260Z","response_time":114,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["free","helm","k8s","kubernetes","oci","oke","oracle-cloud","oracle-cloud-infrastructure","terraform"],"created_at":"2025-08-15T11:02:37.004Z","updated_at":"2026-03-12T03:38:42.292Z","avatar_url":"https://github.com/nce.png","language":"HCL","funding_links":[],"categories":["terraform"],"sub_categories":[],"readme":"# ⎈ Oracle Cloud Kubernetes free tier setup\n\nThis repository leverages Oracle Cloud's [always free tier][oci-free-tier] to provision a kubernetes cluster.\nIn its current setup there are **no monthly costs** anymore, as I've now moved\nthe last part (DNS) from oci to cloudflare.\n\n[oci-free-tier]: https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier_topic-Always_Free_Resources.htm\n\nOracle Kubernetes Engine (OKE) is free to use, and you only pay for worker\nnodes _if_ you exceed the Always Free tier — which we don’t.\nThe free tier provides **4 oCPUs and 24GB of memory**, which are split between two\nworker nodes (`VM.Standard.A1.Flex`), allowing for efficient resource\nutilization. Each node has a 100GB boot volume, with around 60GB available for\nin-cluster storage via Longhorn. For ingress, we use the `GatewayAPI` implementation\nprovided by envoy-gateway together with Oracle’s\nFlexible Load Balancer (10Mbps; layer 7). For `teleport` we use the network LB (layer 4).\nIn this configuration both load balancers are free to use.\n\nGetting an Always Free account can sometimes be tricky, but there are several\nguides on Reddit that explain how to speed up the creation process.\n\nThe initial infra setup is inspired by this great tutorial: https://arnoldgalovics.com/free-kubernetes-oracle-cloud/\n\n\u003e [!WARNING]\n\u003e This project uses arm instances, no x86 architecture; due to limitations\n\u003e of the always free tier.\n\u003e\n\u003e *And please mind:*\n\u003e This setup is loosely documented and opinionated. It's working and in use\n\u003e by myself. It's public, to showcase how this setup can be recreated, but\n\u003e you need to know what you're doing and where to make modification for yourself.\n\nThis repo hosts my personal stuff and is a playground for my kubernetes tooling.\n\n\u003e [!TIP]\n\u003e In case you want to reproduce another `oke` setup, you might [find this guide](https://github.com/piontec/free-oci-kubernetes)\n\u003e also helpful.\n\n## :wrench: Tooling\n\n- [x] K8s control plane\n- [x] Worker Nodes\n- [x] Ingress with GatewayAPI\n  * envoy-gateway on a layer 7 lb\n  * teleport svc on a layer 4 lb\n  * `httproutes` with oidc protection\n- [x] Certmanager\n  * with letsencrypt setup for cloudflare dns challenge\n- [x] External DNS\n  * with sync to the cloudflare dns management\n  * CR to provide `A` records for my home-network\n- [x] Dex as OIDC Provider\n  * with github as idP\n- [x] FluxCD for Gitops\n  * deployed with the new fluxcd operator\n  * github → flux webhook receiver for instant reconciliation\n  * flux → github commit annotation about conciliation status\n- [x] Teleport for k8s cluster access\n- [x] Storage\n   * with longhorn\n- [x] Grafana with Dex Login\n   * Dashboards for Flux\n   * [ ] Switch to Grafana Operator\n- [ ] Loki for log aggregation\n- [x] Metrics Server for cpu/mem usage overview\n- [ ] Kyverno and Image Signing\n- [x] [S3 Proxy](https://github.com/oxyno-zeta/s3-proxy) for http access of buckets\n\n# :keyboard: Setup\n\u003e [!Note]\n\u003e I've recently updated the tf-backend config,  to utilizes the oci native\n\u003e backend now. This requires terraform \u003e= v1.12\n\nThis setup uses terraform to manage the oci **and** a bit of the kubernetes part.\n\n## Tooling on the client side\n\n* terraform\n* oci-binary\n* `oci setup config` successfully run\n\nThe terraform state is pushed to oracle object storage (free as well). For that\nwe have to create a bucket initially:\n```\n❯ oci os bucket create --name terraform-states --versioning Enabled --compartment-id xxx\n```\n\nWith the bucket created we can configure the `~/.oci/config`:\n```\n[DEFAULT]\nuser=ocid1.user.xxx\nfingerprint=ee:f4:xxx\ntenancy=ocid1.tenancy.oc1.xxx\nregion=eu-frankfurt-1\nkey_file=/Users/xxxx.pem\n\n[default]\naws_access_key_id = xxx \u003c- this needs to be created via UI: User -\u003e customer secret key\naws_secret_access_key = xxx \u003c- this needs to be created via UI: User -\u003e customer secret key\n\n```\nRefer to my [backend config](./terraform/infra/_terraform.tf) for the terraform s3 configuration.\n\n## 🏗️ Terraform Layout\n* The infrastructure (everything to an usable k8s-api endpoint) is managed by\nterraform in [infra](terraform/infra/)\n* The k8s-modules (OCI specific config for secrets etc.) are managed by terraform in [config](terraform/config/)\n* The k8s apps/config is done with flux; see below\n\nThese components are independent from each other, but obv. the infra should\nbe created first.\n\nFor the config part, you need to add a private `*.tfvars` file:\n```\ncompartment_id   = \"ocid1.tenancy.zzz\"\n... # this list is currently not complete; there's more to add\n```\n\nRunning the `config` section you need more variables, which either get output\nby the `infra`-run or have to be extracted from the webui.\n\n\u003e [!TIP]\n\u003e During the initial provisioning the terraform run of `config` might fail,\n\u003e it's trying to create a `ClusterSecretStore` which only exist after the\n\u003e initial deployment of `external secrets` with flux. This is expected.\n\u003e Just rerun terraform after external secrets is successfully deployed.\n\n## Kubernets Access - kubeconfig\nAfter running terraform in the [infra](./terraform/infra) folder, a kubeconfig file\nshould be created in the terraform folder called `.kube.config`.\nThis can be used to access the cluster.\nFor a more regulated access, see the Teleport section below.\n\nThe terraform resources in the [config](./terraform/config) folder will rely on the kubeconfig.\n\n## FluxCD\nMost resources and core components of the k8s cluster are provisioned with fluxcd.\nTherefore we need a Github Personal acccess Token (`pat` - fine grained) in your repo.\n```\n# github permission scope for the token:\ncontents - read, write\ncommit statuses - read, write\nwebhooks - read, write\n```\n\n* Place this token in a private tfvars. This is used to\ngenerate the fluxcd webhook url, which triggers fluxcd reconciliation after each\ncommit\n* Place this token in the oci vault (`github-fluxcd-token`). This allows fluxcd\nto annotate the github commit status, depending on the state of the `Kustomization`.\n\n### Fluxcd Operator\nMigrating from the `flux bootstrap` method to the flux-operator might be tricky.\nI lost most installed apps during my upgrade, because i misconfigured the\n`FluxInstace.path` (this could've mitigated by setting `prune: false` on the KS).\nDestroying the old Bootstrap resource during the TF apply, lead\nto the removal of the fluxcd crds like `GitRepo, HelmRelease` etc\n(had the remove the finalizers of the crds\nto allow removal). This didn't impact my already deployed CRs though.\nThe Flux Operator takes care of reinstalling everything.\n\nI've setup a Githup App and mostly followed the official guide,\nthis was pretty straightforward.\n\n#### Development\nSwitching to a feature/dev branch is rather simple, just modify the\ninCluster `FluxInstance` - search for the `sync` block and update the `ref`\nsection to the according branch.\n\n## Teleport\n[Teleport](https://goteleport.com/) is my preferred way to access the kuberentes api\n### Prerequisites\nIn it's current state, teleport wants to setup a wildcard domain like `*.teleport.example.com` (could be disabled).\nWith OracleCloud managing the dns, this is not possible, as `cert-manager` is not\nable to do a `dns01` challenge against orcale dns.\nI've now switched to Cloudflare (also to mitigate costs of a few cents).\n\nThe Teleport \u003c-\u003e K8s Role (`k8s/system:masters`) is created by the teleport\noperator (see the `fluxcd-addons/Kustomization`). The SSO setup is created with\n[fluxcd](./gitops/core/teleport/rbac).\n\n### ~Login via local User~ - removed\nI've removed local users in teleport and am using SSO with github as idP.\n\nThis might still be useful for local setups not using SSO:\n\nThe login process must be `reset` for each user, so that\npassword and 2FA can be configured by each user in the WebUI.\nThe User can be created via the teleport-operator by creating a `TelepertUser` in\nkubernetes.\n```\n# reset the user once\n❯ k --kubeconfig ~/.kube/oci.kubeconfig exec -n teleport -ti deployment/teleport-cluster-auth -- tctl users reset nce\n\n# login to teleport\n❯ tsh login --proxy teleport.nce.wtf:443 --auth=local --user nce teleport.nce.wt\n```\n\n### Login via Github\nThere's no user management in teleport, so no reset, or 2FA setup is needed.\n```\n❯ tsh login --proxy teleport.nce.wtf:443 --auth=github-acme --user nce teleport.nce.wtf\n\n# login to the k8s cluster\n❯ tsh kube login oci\n\n# test\n❯ k get po -n teleport\n```\n\n### Certificates\nThe x509 certs are managed by `cert-manager`. With the dns management done by\ncloudflare, i've removed all `http01` challenges. The renewal process with\n`http01` and cloudflare is [out of the box not possible][cert-manager-cloudflare]\n\nSwitching to `dns` challenge solves this issue.\n\n### Known Issues\n```\n❯ tsh kube login oci\nERROR: connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"\n```\nWhen reinstalling teleport, during a logged in session happens, the tsh Client\nthrows this error. A logout \u0026 login has to happen.\n\n## Gateway API - Loadbalancer setup\nWith the recent deprecation of the `nginx-ingress` project, i [switched][pr] to\n[GatewayAPI][gatewayapi]. All `ingress` resources are replaced by `httproutes`.\nI chose the [envoy-gateway][envoy-gateway] implementation, as they feature\n`SecurityPolicies` which allow the protection of `httproutes` with OIDC auth.\n\nWith orcale providing a layer 7 loadbalancer, the oci-controller needs to\nknow which securitygroup should be associated with the lb. This is done by\nannotating the Service. Using envoy, we have to add the annotations on an\n`EnvoyProxy` CR. Unfortunately the value for the sg is created via terraform,\nbut is later needed on the object. In the past i used to write it in a configmap,\nand helm was able to use the CM as value.yaml and annotated the resource.\nThis is now no longer possible, therefore the sg is currently hardcoded.\n\n[pr]: https://github.com/nce/oci-free-cloud-k8s/pull/152\n[gatewayapi]: https://gateway-api.sigs.k8s.io/\n[envoy-gateway]: https://gateway.envoyproxy.io/\n\n## Monitoring\n\n# :money_with_wings: Cost\nOverview of my monthly costs:\n![](docs/cost.25.png)\n\n# :books: Docs\nA collection of relevant upstream documentation for reference\n\n## Ingress\n* LB Annotation [for oracle cloud][lb-annotations]\n* Providing OCI-IDs to Helm Releases on [nginx][nginx-helm-lb-annotations]\n\n## Cert Manager\n* [DNS01 Challenge][cert-manager-dns-challenge]\n\n## External Secrets\n* [Advanced Templating for secrets][secrets-templating]\n\n## External Dns\n* [CRDs for DNS records][dns-crds]\n\n## Teleport\n* [teleport-operator][teleport-operator]\n* Teleport [User/Roles RBAC][teleport-rbac]\n* Mapping to teleport role\n* [SSO with GithubConnector][teleport-github-sso] and [External Client Secret][teleport-client-secret]\n* [Helm Chart Deploy Infos][teleport-helm-doc] \u0026 [Helm Chart ref][teleport-helm-chart]\n\n## FluxCD\n* [Monitoring setup][flux-monitoring]\n* [Webhook Config][flux-webhook]\n* [Webhook Url Hashing][flux-webhook-hashing]\n* [FluxCD Operator][flux-operator-migration]\n\n[lb-annotations]: https://github.com/oracle/oci-cloud-controller-manager/blob/master/docs/load-balancer-annotations.md\n[nginx-helm-lb-annotations]: https://github.com/kubernetes/ingress-nginx/blob/74ce7b057e8d4ac96d2e11e027930397e5f70010/charts/ingress-nginx/templates/controller-service.yaml#L7\n[cert-manager-dns-challenge]: https://cert-manager.io/docs/configuration/acme/dns01/\n[cert-manager-cloudflare]: https://ryanschiang.com/cloudflare-letsencrypt-http-01\n\n[secrets-templating]: https://external-secrets.io/v0.15.0/guides/templating/#helm\n\n[dns-crds]: https://kubernetes-sigs.github.io/external-dns/latest/docs/sources/crd/#using-crd-source-to-manage-dns-records-in-different-dns-providers\n\n[teleport-client-secret]: https://goteleport.com/docs/admin-guides/infrastructure-as-code/teleport-operator/secret-lookup/#step-23-create-a-custom-resource-referencing-the-secret\n[teleport-github-sso]: https://goteleport.com/docs/admin-guides/access-controls/sso/github-sso/\n[teleport-rbac]: https://goteleport.com/docs/admin-guides/access-controls/getting-started/#step-13-add-local-users-with-preset-roles\n[teleport-helm-chart]: https://goteleport.com/docs/reference/helm-reference/teleport-cluster/\n[teleport-helm-doc]: https://goteleport.com/docs/admin-guides/deploy-a-cluster/helm-deployments/kubernetes-cluster/\n[teleport-operator]: https://goteleport.com/docs/admin-guides/infrastructure-as-code/teleport-operator/\n\n[flux-monitoring]: https://fluxcd.io/flux/monitoring/metrics/#monitoring-setup\n[flux-webhook]: https://fluxcd.io/flux/guides/webhook-receivers/\n[flux-webhook-hashing]: https://github.com/fluxcd/notification-controller/issues/1067\n[flux-operator-migration]: https://fluxcd.control-plane.io/operator/flux-bootstrap-migration/\n\n# Upgrading the Kubernetes Version\nI recommend only upgrading to the version the first command (`available-kubernetes-upgrades`) shows.\nOther upgrades, or jumps to the latest version not being shown, might break the process.\nThe [K8s Skew policy][k8s-skew] allows the worker nodes (`kubelets`)\nto be three minor versions behind, so you might be alright, if you incrementally update the controlplane,\nbefore updating the nodepool.\n\n[k8s-skew]: https://kubernetes.io/releases/version-skew-policy/#kubelet\n\nThe commands should be executed inside [terraform/infra/](terraform/infra)\n```\n# get new cluster versions\n❯ oci ce cluster get --cluster-id $(terraform output --raw k8s_cluster_id) | jq -r '.data.\"available-kubernetes-upgrades\"'\n\n# update the cluster version with the information from above\n❯ sed -i '' 's/default = \"'$(terraform output --raw kubernetes_version)'\"/default = \"v1.31.1\"/' _variables.tf\n\n# upgrade the controlplane and the nodepool \u0026 images\n# this shouldn't roll the nodes and might take around 10mins\n❯ terraform apply\n```\n To roll the nodes, i cordon \u0026 drain the k8s node:\n```\n❯ k drain \u003ck8s-node-name\u003e --force --ignore-daemonsets --delete-emptydir-data\n❯ k cordon \u003ck8s-node-name\u003e\n```\nA node deletion in k8s doesn't trigger a change in the `nodepool`. For that, we\nneed to terminate the correct instance. But i haven't figured out how to delete the -\ncurrently cordoned - node, only using `oci`.\n\nSo, login to the webui -\u003e Oke Cluster -\u003e Node pool and check for the right\ninstance by looking at the private_ip and copy the id.\n\nNow terminate that instance:\n```\n❯ oci compute instance terminate --force --instance-id \u003coidc.id\u003e\n```\nThis triggers a node recreation. Now wait till the node is Ready; And then wait for\nlonghorn to sync the volumes.\n```\n# wait until all volumes are healthy again\n❯ k get -w volumes.longhorn.io -A\n```\nRepeat the cordon/drain/terminate for the second node.\n### OKE Upgrade 1.31.1\nFor the current update, i've written above upgrade instructions.\nWorked flawlessly, though still with a bit of manual interaction in the webui...\n\n### OKE Upgrade 1.29.1\nI mostly skipped `1.27.2` \u0026 `1.28.2` (on the workers) and went for the `1.29` release. As the UI didn't\nprompt for a direct upgrade path of the control-plane, i upgraded the k8s-tf\nversion to the prompted next release, ran the upgrade, and continued with the next version.\n\nThe worker nodes remained at `1.26.7` during the oke upgrade, which worked because with 1.28\nthe new skew policy allows for worker nodes to be three versions behind.\n\n### OKE Upgrade 1.25.4\n\n:warning: remember to remove any `PSP`s first\n\n1. Upgrade the nodepool \u0026 cluster version by setting the k8s variable; Run terrafrom (takes ~10min)\n2. Drain/Cordon worker01\n3. Go to the UI; delete the worker01 from the nodepool\n4. Scale the Nodepool back to 2 (takes ~10min)\n5. Wait for longhorn to sync (no volume in state `degraded`)\n6. repeat for second node (2-5)\n\n### OKE Upgrade 1.24\n\nThe 1.23.4 -\u003e 1.24.1 Kubernetes Upgrade went pretty smooth, but by hand.\n\nI followed the official guide:\n\n- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengupgradingk8smasternode.htm\n- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengupgradingk8sworkernode.htm\n\nLonghorn synced all volumes after the new node got ready. No downtime experienced.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnce%2Foci-free-cloud-k8s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnce%2Foci-free-cloud-k8s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnce%2Foci-free-cloud-k8s/lists"}