https://github.com/vaughnw128/immanent-grove
Bright the hawks flight on the empty sky
https://github.com/vaughnw128/immanent-grove
devops homelab kubernetes talos terraform
Last synced: 2 months ago
JSON representation
Bright the hawks flight on the empty sky
- Host: GitHub
- URL: https://github.com/vaughnw128/immanent-grove
- Owner: vaughnw128
- Created: 2025-01-18T18:59:22.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2026-04-13T02:02:41.000Z (2 months ago)
- Last Synced: 2026-04-13T02:16:30.096Z (2 months ago)
- Topics: devops, homelab, kubernetes, talos, terraform
- Language: HCL
- Homepage: https://vaughn.sh
- Size: 694 KB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 11
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# immanent-grove
[](https://github.com/vaughnw128/immanent-grove/actions/workflows/renovate.yaml)
[](https://github.com/vaughnw128/immanent-grove/actions/workflows/test.yaml)
K8s cluster bootstrapping, because nuking and rebuilding your homelab should be _easy_.
## Overview
This repository serves to let me control my entire k8s homelab environment with IaC, and is a test bed for my knowledge on modern DevOps principles.
The cluster is bootstrapped with Terraform via Digger and then managed with FluxCD.
## Deployment
This cluster is built around Talos and Cilium, and heavily leverages the new Gateway API for exposing HTTPRoutes to both public and local audiences.
I like building publicly facing things, so I want that proccess to be easy for me. This cluster is deployed on low-powered intel mini-pcs, using Talos as the base OS image.
This makes for an easily IaC controlled environment, and the bootstrap process is quite quickly. The single Terraform script can get the entire lab fully deployed.
## Hardware
My k8s hardware is quite simple and 'cheap':
- 1x UN100L Amd64 16gb memory
- 2x UN150p Amd64 16gb memory
- 1x UN1290 Amd64 32gb memory
I love these stupid Mini PCs, and they keep my power bill quite low. This is supplemented by a TrueNAS Scale
system that runs some nice persistence services like Minio and is the backend for Democratic CSI.
## Terraform Deployment
To bootstrap the cluster simply deploy with Terraform `terraform plan` & `terraform apply`. All of the variables will need to be
customized to your own cluster.
### Backend
All of my TF code w/ Digger in Github Actions is backed by Minio for S3-like storage of the Terraform state. This makes things far easier, and even works well when running locally.
Keep the state off of my PC!!
### Variables
I pull super-duper secret values in with the OnePassword provider straight from my OnePassword vault. If this doesn't work to your liking or you don't use OnePassword, it can be commented out in favor
of `secret.auto.tfvars`:
```
# Proxmox
proxmox_api_token = "tf-provisioner@pam!tf-provisioner="
proxmox_ssh_username = "root"
proxmox_ssh_password = ""
```
## Testing
Tests can be run by applying manifests in `/examples` and verifying their output.
## Talos Upgrades
```bash
# First upgrade the worker nodes
$ talosctl upgrade --nodes 10.0.0.52 -e 10.0.0.50 --image factory.talos.dev/nocloud-installer/84f66f3fa52900a0234636ae1da07d5b356cce774673951af35866142158fce6:v1.10.5
$ talosctl upgrade --nodes 10.0.0.51 -e 10.0.0.50 --image factory.talos.dev/nocloud-installer/84f66f3fa52900a0234636ae1da07d5b356cce774673951af35866142158fce6:v1.10.5
# Then finally the control plane node
$ talosctl upgrade --nodes 10.0.0.50 -e 10.0.0.50 --image factory.talos.dev/nocloud-installer/84f66f3fa52900a0234636ae1da07d5b356cce774673951af35866142158fce6:v1.10.5
```
## Upgrading Controlplanes with Terraform
Upgrading via terraform can be a bit nerve-wracking and annoying, but is sometimes needed in order to get the images all set up properly.
> NOTE!!!! DO NOT DO THIS IF YOU ONLY HAVE ONE CONTROLPLANE NODE!! THIS WILL BREAK EVERYTHING!!
>
> THIS UPDATE PROCEDURE REQUIRES AT LEAST THREE CONTROLPLANE NODES FOR HA!!
1. Generate a new Talos factory image at [factory.talos.dev](https://factory.talos.dev)
1. The image must be: amd64, Nocloud, and have the addons iscsi-tools, qemu-guest-agent, and nfsd.
2. Replace the node image in proxmox-nodes.tf
3. Plan and apply the Terraform, ensuring that there are only the talos machine config, proxmox node, and talos image modified.
4. If the node fails to join etcd, delete the member from etcd to ensure it can reach quorum
```bash
talosctl --endpoints 10.0.0.53 -n 10.0.0.53,10.0.0.54 etcd remove-member
```
6. Check etcd to ensure quorum. This should show learners as well. Do not proceed to upgrade another controlplane unless there are no learners.
```bash
talosctl --endpoints 10.0.0.53 etcd status
```
7. Check that the node is started
```bash
k get nodes
```