https://github.com/yoramvandevelde/k3s-gitops
Personal homelab GitOps setup running on k3s. This is an experiment to learn GitOps patterns — not a production setup. The goal is to be able to tear down the cluster and have everything back up within half an hou
https://github.com/yoramvandevelde/k3s-gitops
argocd argocd-config democratic-csi gitops kubernetes metallb observability
Last synced: 5 days ago
JSON representation
Personal homelab GitOps setup running on k3s. This is an experiment to learn GitOps patterns — not a production setup. The goal is to be able to tear down the cluster and have everything back up within half an hou
- Host: GitHub
- URL: https://github.com/yoramvandevelde/k3s-gitops
- Owner: yoramvandevelde
- License: mit
- Created: 2026-05-03T18:41:10.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2026-05-05T20:39:17.000Z (about 2 months ago)
- Last Synced: 2026-05-05T22:28:13.978Z (about 2 months ago)
- Topics: argocd, argocd-config, democratic-csi, gitops, kubernetes, metallb, observability
- Homepage:
- Size: 101 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# k3s-gitops
Personal homelab GitOps setup running on Talos Linux (Kubernetes 1.36). This is an experiment to learn GitOps patterns — not a production setup. The goal is to be able to tear down the cluster and have everything back up within half an hour.
This repo manages the full cluster state via ArgoCD using an app-of-apps pattern. Everything except the initial bootstrap (automated via `tofu apply` in the [k8s-infra](https://github.com/yoramvandevelde/k8s-infra) repo) is managed through Git.
**Infrastructure**
- [cert-manager](https://cert-manager.io) — TLS certificates via Let's Encrypt + Cloudflare DNS challenge
- [ingress-nginx](https://kubernetes.github.io/ingress-nginx) — ingress controller
- [MetalLB](https://metallb.universe.tf) — bare metal load balancer (L2, pool `10.10.99.1–10.10.99.250`)
- [Cilium](https://cilium.io) — CNI with Hubble, Envoy proxy and SPIRE for service mesh and mTLS
- [democratic-csi](https://github.com/democratic-csi/democratic-csi) — iSCSI and NFS storage via TrueNAS
- [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) — encrypted secrets safe to store in Git
- [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) — monitoring (Prometheus + Grafana + Alertmanager)
- [snapshot-controller](https://github.com/kubernetes-csi/external-snapshotter) — volume snapshots
- [Kyverno](https://kyverno.io) — policy engine, all policies in Enforce mode (see below); policies managed as a separate ArgoCD Application (`kyverno-policies`) with retry to handle CRD ordering on bootstrap
- [Tetragon](https://tetragon.io) — eBPF security observability (kernel-level process, file, and network event tracing via `TracingPolicy`)
- [Argo Rollouts](https://argoproj.github.io/rollouts/) — progressive delivery (canary deployments)
- [chaoskube](https://github.com/linki/chaoskube) — chaos engineering, random pod termination every 10 minutes
- [metrics-server](https://github.com/kubernetes-sigs/metrics-server) — resource usage metrics API (`kubectl top`); configured with `--kubelet-insecure-tls` for Talos self-signed kubelet certs
- [VPA](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler) — recommender only; provides resource usage recommendations without auto-mutating pods
- [Goldilocks](https://goldilocks.docs.fairwinds.com) — dashboard for VPA recommendations at `goldilocks.local.sifft.io`; namespaces opt in via `goldilocks.fairwinds.com/enabled: "true"` label
- [Reflector](https://github.com/emberstack/kubernetes-reflector) — mirrors Secrets and ConfigMaps across namespaces; used to propagate the wildcard TLS secret to app namespaces
- [Reloader](https://github.com/stakater/Reloader) — automatically restarts pods when their ConfigMap or Secret changes
- [Trivy Operator](https://aquasecurity.github.io/trivy-operator) — continuous vulnerability scanning of running container images; results available via `kubectl get vulnerabilityreports -A`
**Apps**
- [Forgejo](https://forgejo.org) — self-hosted Git (SQLite, iSCSI storage, SSH via MetalLB)
- [Authentik](https://goauthentik.io) — identity provider (OIDC); SSO for cluster tools. Headlamp authenticates via kube-apiserver OIDC integration — JWTs issued by Authentik are validated directly by the Kubernetes API (not by the app itself)
- [Headlamp](https://headlamp.dev) — Kubernetes web UI with OIDC login via Authentik (`akadmin` → `cluster-admin` via ClusterRoleBinding)
- [Wordpress](https://wordpress.org) — dev and prod environments (MySQL + Redis + custom image with Redis plugin)
- [Recipit](https://github.com/yoramvandevelde/recipit) — personal recipe manager with Home Assistant integration; full CI/CD pipeline that automatically updates the image tag in this repo on push
- [Tuwunel](https://github.com/matrix-construct/tuwunel) — Matrix homeserver (Rust-based, RocksDB, no federation); WebRTC calls via coturn TURN server
- [coturn](https://github.com/coturn/coturn) — TURN/STUN server for Matrix WebRTC calls; MetalLB IP `10.10.99.81`, relay ports `49152-49161 UDP`
## Structure
```bash
bootstrap/ # ArgoCD root app and app-of-apps entrypoints
infrastructure/ # Cluster infrastructure (networking, storage, monitoring, policy, etc.)
apps/ # Application deployments with kustomize base/overlay structure
config/ # Public sealed secrets cert and encrypted master key
```
## Security
All workloads are hardened and comply with the following Kyverno policies:
- `disallow-latest-tag` — image tags must be pinned (Enforce)
- `require-resource-requests-limits` — all containers must define CPU/memory requests and limits (Enforce; exceptions: infra namespaces + `cilium-spire` + `sealed-secrets`)
- `block-privileged-containers` — privileged containers are blocked (Enforce; exceptions: Cilium, democratic-csi node drivers, `kube-proxy`)
- `require-non-root` — containers must run with `runAsNonRoot: true` (Enforce; exceptions: `kube-system`, `storage`, `metallb-system`, `cilium-spire`, `tetragon`, `coturn`, `vpa`, `goldilocks`, `trivy-system`)
- `require-readonly-rootfs` — containers must set `readOnlyRootFilesystem: true` (Enforce; exceptions: `ingress-nginx`, `kube-system`, `storage`, `metallb-system`, `cilium-spire`, `tetragon`, `coturn`, `vpa`, `goldilocks`, `trivy-system`)
All application namespaces have CiliumNetworkPolicy with default-deny, explicit allow rules per workload, and `authentication: mode: required` on all ingress-nginx connections (verified via `Auth: SPIRE` in Hubble). Internal pod-to-pod connections (e.g. wordpress → mysql/redis) are also mTLS-authenticated.
Infrastructure namespaces (kube-system, argocd, monitoring, storage, kyverno, etc.) are intentionally excluded from mTLS requirements. If SPIRE has any issue and mTLS blocks coredns, Cilium itself, or ArgoCD, the cluster becomes unreachable or unrecoverable — a fragility that isn't worth the security gain in this context.
### Tetragon — kernel-level observability
Tetragon runs as a privileged DaemonSet and hooks into the Linux kernel via eBPF. Three `TracingPolicy` resources are active:
- `shell-exec-detector` — cluster-wide, fires on `execve` when the target binary is any shell (`/bin/sh`, `/bin/bash`, `/bin/dash`). Detects shell spawning in any pod.
- `sensitive-file-access` — scoped to `recipit-dev` (`TracingPolicyNamespaced`), fires on `openat` for paths under `/etc/shadow`, `/etc/passwd`, `/etc/ssh`, `/proc/1`, and `/var/run/secrets`. Detects credential and token access attempts.
- `tcp-connect-tracker` — cluster-wide, hooks the `tcp_connect` kernel function and captures full socket metadata (src/dst IP + port) for every outbound TCP connection.
Events are consumed live via:
```bash
kubectl exec -n tetragon daemonset/tetragon -c tetragon -- tetra getevents -o compact
```
## Secrets
Secrets are encrypted with Sealed Secrets. The public cert is in `config/sealed-secret-pub.crt` and can be used to encrypt new secrets without cluster access:
```bash
kubeseal --cert config/sealed-secret-pub.crt --format yaml < secret.yaml > sealed-secret.yaml
```
The master key is stored as a GPG-encrypted file in `config/sealed-secrets-master-key.yaml.gpg`. Without it, sealed secrets cannot be decrypted on a new cluster.
## Bootstrap
The full bootstrap is automated via `tofu apply` in the [k8s-infra](https://github.com/yoramvandevelde/k8s-infra) repo. It runs a Docker container that installs Cilium, ArgoCD, and Sealed Secrets in order, restores the sealed secrets master key, and then applies the root app:
```bash
kubectl apply -f bootstrap/root.yaml
```
ArgoCD takes over from there and syncs all infrastructure and apps from Git.
The sealed secrets master key must be restored before ArgoCD syncs, otherwise secrets will fail to decrypt.