{"id":13582576,"url":"https://github.com/szinn/k8s-homelab","last_synced_at":"2025-04-05T02:08:33.984Z","repository":{"id":36954441,"uuid":"478533432","full_name":"szinn/k8s-homelab","owner":"szinn","description":"My home operations repository using k8s/gitops","archived":false,"fork":false,"pushed_at":"2024-10-29T11:11:14.000Z","size":14702,"stargazers_count":182,"open_issues_count":2,"forks_count":5,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-10-29T13:12:01.863Z","etag":null,"topics":["flux","gitops","k8s","k8s-at-home","kubernetes","renovate","selfhosted","talos","terraform"],"latest_commit_sha":null,"homepage":"https://szinn.github.io/k8s-homelab","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/szinn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-06T11:40:21.000Z","updated_at":"2024-10-29T11:10:45.000Z","dependencies_parsed_at":"2023-09-28T23:29:30.788Z","dependency_job_id":"24a80db4-d70b-4309-bd9e-d0fe96829ff0","html_url":"https://github.com/szinn/k8s-homelab","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/szinn%2Fk8s-homelab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/szinn%2Fk8s-homelab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/szinn%2Fk8s-homelab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/szinn%2Fk8s-homelab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/szinn","download_url":"https://codeload.github.com/szinn/k8s-homelab/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247276164,"owners_count":20912288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flux","gitops","k8s","k8s-at-home","kubernetes","renovate","selfhosted","talos","terraform"],"created_at":"2024-08-01T15:02:51.497Z","updated_at":"2025-04-05T02:08:33.977Z","avatar_url":"https://github.com/szinn.png","language":"HCL","funding_links":[],"categories":["YAML","HCL","Shell"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/szinn/k8s-homelab/main/docs/assets/logo.png\" align=\"center\" width=\"144px\" height=\"144px\"/\u003e\n\n\u003c!-- markdownlint-disable no-trailing-punctuation --\u003e\n\n### My home operations repository :octocat:\n\n_... managed with Flux, Renovate and GitHub_ 🤖\n\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n[![Talos](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.zinn.ca%2Ftalos_version\u0026style=for-the-badge\u0026logo=talos\u0026logoColor=white\u0026color=blue\u0026label=%20)](https://talos.dev)\u0026nbsp;\u0026nbsp;\n[![Kubernetes](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.zinn.ca%2Fkubernetes_version\u0026style=for-the-badge\u0026logo=kubernetes\u0026logoColor=white\u0026color=blue\u0026label=%20)](https://kubernetes.io)\u0026nbsp;\u0026nbsp;\n[![Flux](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.devbu.io%2Fflux_version\u0026style=for-the-badge\u0026logo=flux\u0026logoColor=white\u0026color=blue\u0026label=%20)](https://fluxcd.io)\u0026nbsp;\u0026nbsp;\n[![Renovate](https://img.shields.io/github/actions/workflow/status/szinn/k8s-homelab/renovate.yaml?branch=main\u0026label=\u0026logo=renovatebot\u0026style=for-the-badge\u0026color=blue)](https://github.com/szinn/k8s-homelab/actions/workflows/renovate.yaml)\n\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n[![Age-Days](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.zinn.ca%2Fcluster_age_days\u0026style=flat-square\u0026label=Age)](https://github.com/kashalls/kromgo/)\u0026nbsp;\u0026nbsp;\n[![Uptime-Days](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.zinn.ca%2Fcluster_uptime_days\u0026style=flat-square\u0026label=Uptime)](https://github.com/kashalls/kromgo/)\u0026nbsp;\u0026nbsp;\n[![Node-Count](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.zinn.ca%2Fcluster_node_count\u0026style=flat-square\u0026label=Nodes)](https://github.com/kashalls/kromgo/)\u0026nbsp;\u0026nbsp;\n[![Pod-Count](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.zinn.ca%2Fcluster_pod_count\u0026style=flat-square\u0026label=Pods)](https://github.com/kashalls/kromgo/)\u0026nbsp;\u0026nbsp;\n[![CPU-Usage](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.zinn.ca%2Fcluster_cpu_usage\u0026style=flat-square\u0026label=CPU)](https://github.com/kashalls/kromgo/)\u0026nbsp;\u0026nbsp;\n[![Memory-Usage](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.zinn.ca%2Fcluster_memory_usage\u0026style=flat-square\u0026label=Memory)](https://github.com/kashalls/kromgo/)\u0026nbsp;\u0026nbsp;\n[![Power-Usage](https://img.shields.io/endpoint?url=https%3A%2F%2Fkromgo.zinn.ca%2Fcluster_power_usage\u0026style=flat-square\u0026label=Power)](https://github.com/kashalls/kromgo/)\u0026nbsp;\u0026nbsp;\n\n\u003c/div\u003e\n\n---\n\n## Overview\n\nThis is my mono repo for my home infrastructure. It's based loosely on the template at [onedr0p/flux-cluster-template](https://github.com/onedr0p/flux-cluster-template) as well as many of the exemplar repos, searchable via [https://nanne.dev/k8s-at-home-search](https://nanne.dev/k8s-at-home-search/).\n\nIt follows the concept of Infrastructure as Code and by using tools such [Flux](https://github.com/fluxcd/flux2),\n[Renovate](https://github.com/renovatebot/renovate),\n[go-task](https://github.com/go-task/task) and shell scripts, creates a reproducible, mostly self-managing implementation.\n\nMy original implementation was running on the Ryzen using custom shell scripts and 35+ docker containers managed by hand. Any upgrades, system resets, etc, all had to be manually resolved.\nIt mostly ran just fine. Applying the principle \"If it ain't broke, it isn't complicated enough\" led me to add machines, memory, functionality to achieve a much more automated, self-managing cluster. Plus I have learned a lot!\n\nAt the bottom of this page, is the bringup process that I follow for this cluster. I recommend reading top-to-bottom to understand the cluster structure which will help understand what's needed for the bringup.\n\n---\n\n## Hardware\n\nMy HomeLab consists of a bunch of machines and Ubiquity networking.\n\n| Device                                             | Count | OS Disk Size | Data Disk Size        | RAM  | Operating System          |\n| -------------------------------------------------- | ----- | ------------ | --------------------- | ---- | ------------------------- |\n| Ryzen 3900 12c24t NAS server                       | 1     | 1TB          | 1TB NVME, 6x16Tb SATA | 64GB | TrueNAS Scale - Ragnar    |\n| Raspberry Pi                                       | 1     |              |                       |      | OctoPrint                 |\n| Raspberry Pi 4B                                    | 1     |              |                       |      | Artemis - AdGuardHome DNS |\n| Raspberry Pi 5                                     | 1     |              |                       |      | Raspberry PiOS            |\n| TESmart 16-port HDMI Switch                        | 1     |              |                       |      |                           |\n| PiKVM                                              | 1     |              |                       |      |                           |\n| Intel NUC11PAHi7 (worker nodes)                    | 3     | 500GB SSD    | 1TB NVMe              | 64GB | Talos                     |\n| Beelink MiniPC, Celeron J4125 (controlplane nodes) | 3     | 256GB SSD    |                       | 8GB  | Talos                     |\n| Synology 1019+ (NFS server)                        | 1     |              | 5x12TB SATA           |      |                           |\n| UniFi UDM SE                                       | 1     |              |                       |      |                           |\n| USW-Pro-24-PoE                                     | 1     |              |                       |      |                           |\n| USW-Aggregation                                    | 1     |              |                       |      |                           |\n| USW-Enterprise-8-PoE                               | 2     |              |                       |      |                           |\n| USW-Flex XG                                        | 1     |              |                       |      | Desktop Hub               |\n| USW-Flex                                           | 1     |              |                       |      | Outside Camera Hub        |\n| UNVR                                               | 1     |              | 3x4TB SATA            |      |                           |\n| USP-PDU Pro                                        | 2     |              |                       |      |                           |\n| 6-port NUC                                         | 1     | 512GB SSD    |                       | 32GB | Fedora, AdGuardHome DNS   |\n| Intel NUC11TNHi7                                   | 1     | 1Tb          |                       | 64GB | Proxmox                   |\n| Intel NUC13 Pro                                    | 1     | 1Tb          |                       | 32GB | Fedora - Hera             |\n| UVC G4 Doorbell                                    | 1     |              |                       |      | Front Door Camera         |\n| UVC G4 Pro                                         | 3     |              |                       |      | Additional Cameras        |\n\nThe Proxmox Intel NUC runs a 3-node Talos staging cluster where I can try out various patterns before deploying in the main cluster.\n\nThe Intel NUC13 (Hera) is a spare NUC that I'm currently using as a Fedora platform with a graphical UI.\n\nTitan used to be the VyOS router which has since gone out of favour. It now runs AdGuardHome DNS as a secondary DNS on Fedora.\n\nArtemis runs services that need to be outside the cluster:\n\n- DNS (AdGuard Home)\n- Cloudflare DDNS\n- gatus to track machine and non-cluster services\n\n## Kubernetes\n\nThe cluster is based on [Talos](https://www.talos.dev) with 3 control-plane nodes running on the Beelink MiniPCs and 3 worker nodes running on the Intel NUCs.\n\n### Core Components\n\n- [kubernetes/ingress-nginx](https://github.com/kubernetes/ingress-nginx): Manages reverse-proxy access to Kubernetes services.\n- [rook/rook](https://github.com/rook/rook): Distributed block storage for persistent storage.\n- [jetstack/cert-manager](https://cert-manager.io/docs/): Creates SSL certificates for services in my Kubernetes cluster.\n- [kubernetes-sigs/external-dns](https://github.com/kubernetes-sigs/external-dns): Automatically manages DNS records from my cluster in a cloud DNS provider.\n\n### GitOps\n\n[Flux](https://github.com/fluxcd/flux2) watches my [main cluster](./kubernetes/main) folder (see Directories below) and makes the changes to my cluster based on the YAML manifests.\n\n[Renovate](https://github.com/renovatebot/renovate) watches my **entire** repository looking for dependency updates, when they are found a PR is automatically created.\nWhen PRs are merged [Flux](https://github.com/fluxcd/flux2) applies the changes to my cluster.\n\nCharts and images are tagged in the various YAML files to enable Renovate to watch and update them as needed.\n\n## Network Configuration\n\nSee [diagram](./docs/Network-Backbone.png) of the backbone.\n\nThe external network is connected to the UDM SE with a [Wireguard](https://www.wireguard.com) port being the only exposed access.\nThis allows me to connect into the network when I'm traveling.\nInbound services are managed with cloudflared.\n\nThe main cluster and IPs are on the 10.11.x.x subnet on VLAN HOMELAB.\nThe stagint cluster and IPs are on the 10.12.x.x subnet on VLAN STAGING.\nExternal machines (Synology, etc) are SERVERS VLAN subnet. IoT devices are on an isolated IoT VLAN.\nThey cannot reach the other VLANs directly but will answer when spoken to.\n\nDNS is managed by CoreDNS in the cluster which then forwards unresolved requests to DNS running on the Titan server.\nTitan will forward accepted addresses onto the UDM-SE for resolution.\n\nThe external DNS is managed via [Cloudflare](https://www.cloudflare.com/en-ca/).\nExternal names are managed by [external-dns](https://github.com/kubernetes-sigs/external-dns) on the cluster and, since my home IP can be changed at any time, DDNS is maintained by the\n[oznu/cloudflare-ddns](https://hub.docker.com/r/oznu/cloudflare-ddns/) docker image. Certificates are managed through CloudFlare as well using cert-manager and the DNS01 challenge protocol.\n\n## Repository Structure\n\nThe repository supports multiple clusters -- in particular, I have a \"main\" cluster which runs on the above hardware. I previously had a staging cluster that could use the same source, but have since\nmoved that to a separate repo.\n\nAdding something new to the cluster usually requires a lot of trial and error initially. When I am trying something out, I will work in a staging environment as much as possible and then move to the main cluster.\nIf additional iterations are required, I will usually try and do amended commits rather than a chain of commits with comments such as \"Trying again\" or \"Maybe this will work\", etc.\n\nThe repository directories are:\n\n- **.github**: GitHub support files and renovate configuration.\n- **.taskfiles**: Auxiliary files used for the task command-line tool.\n- **kubernetes**: The clusters themselves.\n  - **main**: The main cluster\n    - **apps**: The applications to load.\n    - **bootstrap**: The initial code loaded on the cluster to bootstrap it.\n    - **flux**: The definition of the cluster.\n      - **cluster**: The configuration of the cluster to use flux.\n  - **repositories**: Sources of code for the cluster.\n  - **staging**: The staging cluster that follows the same structure as the main cluster.\n- **hack**: Miscellaneous stuff that really has nothing to do with managing the cluster.\n\n### Environment Setup\n\nInstall pre-commit with\n\n```shell\npre-commit install --install-hooks\n```\n\nAnd update the hooks occasionally with (they should auto-update themselves though)\n\n```shell\npre-commit auto-update\n```\n\n## Cluster Configuration\n\n### External Environment Configuration\n\nAll values are defined as shell environment variables.\n\n### Application Secrets\n\nApplication secrets are maintained by using [external-secrets](https://external-secrets.io).\n\n## Persistent Volume Management\n\nApplications usually require data to work well. Persistent volumes on the cluster are stored in two places - Rook/Ceph and an external NFS.\n\n### Rook/Ceph\n\nEach worker has a 1Tb NVMe drive that is managed with Rook/Ceph. The data stored here is replicated across the multiple workers so it will be fast and available locally.\n\n### NFS\n\nI'm still experimenting with NAS storage with regards to the cluster. Most of my home data is stored on the NAS drive, but file-level permission management has been a bit of a pain.\n\nApplications that don't require fast access to data or only use it for temporary storage (e.g., a download directory) will store the data in NFS.\nThe NFS drives are available across the cluster but are at a slower speed than the Rook/Ceph storage.\n\n### Data Backup and Recovery\n\nCurrently, I use a combination of built-in application backups (e.g., \\*arr applications will backup weekly),\na backup job in cluster that will backup databases (mysql and postgres),\nI am also using volsync to backup the PVCs to NFS that will automatically restore the most recent backup (if it exists) .\n\n## Installation\n\nThe initial bootstrap relies on some configuration, including secrets stored in 1Password such as:\n\n- 1Password credentials required for access by external-secrets\n- TLS keys that pre-seed the certificates to prevent hammering on lets-encrypt.\n\n### Cluster Bringup\n\nThe initial bootstrap of the cluster is launched by the task `bootstrap:main` or `bootstrap:staging` which apply the initial configuration\nin the `{{cluster}}/bootstrap` directories.\n\n### Adding a New Package / Updating Configuration\n\nAdding a new package is simply done by following the patterns of an existing, similar package and pushing the commit to GitHub.\nFlux will then notice the change and apply it.\n\n### Ongoing Maintenance\n\nMaintenance of the cluster is fairly minimal.\n\n- renovate creates PRs to update helm charts, flux system files, or docker images in the cluster;\n- flux applies any merged PRs or changes to the repo to the cluster automatically.\n\nThrough Wireguard and [Kubenav](https://kubenav.io), I can pretty much manage the cluster remotely from my phone.\nOn my desktop/laptop, I use [Lens](https://k8slens.dev) and `k9s` to manage the cluster which works remotely through Wireguard as well.\n\n## Gratitude and Thanks\n\nMany thanks to the folks at [k8s-at-home](https://github.com/k8s-at-home) that maintain the many great Helm charts, have opened their own repos for the rest of us to learn, and answer many questions on the discord server.\nA special thanks to Devin (@onedr0p) who's cluster I modelled mine after.\nAnother special thanks to Nat (@Truxnell) who answered a bunch of my questions about Talos while I was on vacation reading github repos and docs on my iPhone.\nA tongue-in-cheek thanks to Jeff (@billimek) who's dang YouTube video on Home Assistant and his cluster repo led me into this rabbit hole.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fszinn%2Fk8s-homelab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fszinn%2Fk8s-homelab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fszinn%2Fk8s-homelab/lists"}