{"id":15869981,"url":"https://github.com/michaelbeaumont/k8rn","last_synced_at":"2025-03-16T03:30:28.706Z","repository":{"id":252598713,"uuid":"833393113","full_name":"michaelbeaumont/k8rn","owner":"michaelbeaumont","description":"My self-hosted Talos Kubernetes cluster running on Tailscale, configured using terraform and flux","archived":false,"fork":false,"pushed_at":"2024-10-18T14:43:53.000Z","size":146,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-20T00:54:39.853Z","etag":null,"topics":["bare-metal","cloudflare","flux","self-hosted","tailscale","talos","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michaelbeaumont.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-25T00:51:43.000Z","updated_at":"2024-10-18T14:43:56.000Z","dependencies_parsed_at":"2024-10-20T15:49:25.511Z","dependency_job_id":null,"html_url":"https://github.com/michaelbeaumont/k8rn","commit_stats":{"total_commits":110,"total_committers":2,"mean_commits":55.0,"dds":"0.10909090909090913","last_synced_commit":"21ea1ddf9b794969679dccefeb8ac70a48cd9d62"},"previous_names":["michaelbeaumont/k8rn"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelbeaumont%2Fk8rn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelbeaumont%2Fk8rn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelbeaumont%2Fk8rn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelbeaumont%2Fk8rn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michaelbeaumont","download_url":"https://codeload.github.com/michaelbeaumont/k8rn/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221640907,"owners_count":16856243,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bare-metal","cloudflare","flux","self-hosted","tailscale","talos","terraform"],"created_at":"2024-10-06T00:02:57.334Z","updated_at":"2024-10-27T07:02:23.640Z","avatar_url":"https://github.com/michaelbeaumont.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# k8rn\n\nThis holds everything for my bare metal Talos k8s cluster.\n\nThe cluster is initialized with Terraform, including flux, which then runs off of this repo.\n\n## Setup\n\nThis configuration depends on:\n\n- Tailscale\n  - In particular an oauth client that is tagged to own the tags assigned to each device in\n    [the tailscale Terraform](./infra/tailscale.tf)\n- Cloudflare\n  - A DNS zone to assign a subdomain in\n- A KMS service for talos, I'm using [talos-unlockr](https://github.com/michaelbeaumont/talos-unlockr) to easily unlock things on demand.\n- SSH key for flux\n  - Must be a deploy key for this repo\n- Age key for SOPS-encrypted data with flux\n\nThe cluster nodes have to be assigned IPs reachable from your machine.\n\n### Install\n\n#### Talos images\n\nEach image:\n\n- gets a UUID in the META partition because the mini PCs I have don't have proper UUIDs generated\n- has `stable_secret` set as a kernel param\n\nCreating the image factory config and fetching each image is handled in\nTerraform and exposed as outputs. The `justfile` makes it easy to grab the\nupdate image and the ISOs.\n\nEach node is added to the Terraform variable `nodes` before it's added to either\n`control_plane_nodes` or `worker_nodes`.\n\n### DNS\n\nThis Terraform uses Cloudflare to assign a public DNS name to load balance\nbetween the Tailscale IPs.\n\nAnother option would be to run a something like\n[coredns-tailscale](https://github.com/damomurf/coredns-tailscale) to handle\nthis.\n\nThis has two problems:\n\n- [Talos can't take advantage of split DNS yet](https://github.com/siderolabs/talos/issues/7287)\n  so it can't reach itself.\n- It's another piece of infrastructure that has to be running \"somewhere\"\n\n### Tailscale\n\nThis terraform can use IPv6 addresses, which Tailscale provisions dynamically.\n\nDepending on which DNS you're using, note that there might be a\nlarge delay between when the nodes start querying the API endpoint\n/we try to contact the API endpoint and when the DNS records are propagated.\n\nSomewhat out of scope but for ease of use I also assign IPv4 Tailscale addresses\nstatically.\nAssuming `cluster_name = \"k8rn\"`, this can be achieved via ACLs:\n\n```json\n    \"nodeAttrs\": [\n        {\n            \"target\": [\"tag:k8rn-cp-0\"],\n            \"ipPool\": [\"100.64.1.10/32\"],\n        },\n        {\n            \"target\": [\"tag:k8rn-cp-1\"],\n            \"ipPool\": [\"100.64.1.11/32\"],\n        },\n        {\n            \"target\": [\"tag:k8rn-cp-2\"],\n            \"ipPool\": [\"100.64.1.12/32\"],\n        },\n    ],\n```\n\nEach node is assigned `tag:\u003cnode hostname\u003e` which Tailscale then uses to assign an IP pool,\nin this case a pool of size one.\n\n## Kubernetes\n\n### Cilium\n\nCilium is set up with native routing. This is a bit tricky with Tailscale\nbecause each node needs to advertise routes for its `podCIDRs` but we can't\ndirectly put this in the `ExtensionServiceConfig` since it isn't known when\nwe initialize the extension.\n\nInstead, two extra init containers run with the cilium agent.\nThe first retrieves the node's `podCIDRs` and the second calls\n`tailscale set --advertise-routes` with those `podCIDRs`.\nThe Tailscale extension is configured with `--accept-routes`\nand `--snat-subnet-routes=false`,\nsince we don't want or need to SNAT between Pods on different nodes.\n\nIdeally this functionality would be upstreamed but for now, this just works.\n\nThis requires some ACLs set up in Tailscale:\n\n```json\n{\n    \"ipsets\": {\n        \"ipset:k8rn-pods\": [\n            \"add 10.244.0.0/16\",\n        ],\n    },\n    \"autoApprovers\": {\n        \"routes\": {\n            \"10.244.0.0/16\": [ // can't use ipset here\n                \"tag:k8rn-node\",\n            ],\n        },\n    },\n    \"acls\": {\n        ...\n        // k8rn: nodes and pods can reach each other\n        {\n            \"action\": \"accept\",\n            \"src\":    [\"tag:k8rn-node\", \"ipset:k8rn-pods\"],\n            \"dst\": [\n                \"tag:k8rn-node:*\",\n                \"ipset:k8rn-pods:*\",\n            ],\n        },\n    },\n}\n```\n\n### Intel GPU plugin\n\nMy nodes are Beelink EQ12s, their N100 CPU is a 12th generation Intel,\nso it has Intel Quick Sync Video support for hardware acceleration with Jellyfin.\n\nThis repo uses\n[Intel's GPU plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/cmd/gpu_plugin/README.md)\nto [make everything available to the Jellyfin `Pods`](./k8s/base/gpu).\n\n### Envoy gateway\n\nk8rn is running Envoy gateway with a relatively complicated architecture.\n\nIdeally we'd use the Tailscale operator to expose the gateway as a `LoadBalancer` `Service`\nand the `Gateway` would get its own Tailscale machine.\nHowever due to tailscale/tailscale#12393 this is currently infeasible.\n\nInstead we rely on the nodes being Tailscale machines:\n\n- Set `net.ipv4.ip_unprivileged_port_start=0` on the nodes so we can listen on `443` without root\n- The Envoy gateway `Pods` run with `hostNetwork: true`\n- Set `useListenerPortAsContainerPort: true` so the container really listens on `443`\n- An `EnvoyPatchPolicy` modifies the listener to bind only to `tailscale0` with `SO_BINDTODEVICE`\n- The Service is configured as `NodePort` so that `external-dns` creates A/AAAA records pointing to each of the node IPs.\n  - Note this depends on clients failing over between DNS records because only one of the nodes will\n    have Envoy gateway running. But this seems to work from my testing\n  - Otherwise Envoy can be deployed as a `DaemonSet`\n\n### External services\n\nSome services are running on a different server in the tailnet and a `Service`\nwith a manually managed `EndpointSlice` handles forwarding the traffic.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelbeaumont%2Fk8rn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichaelbeaumont%2Fk8rn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelbeaumont%2Fk8rn/lists"}