Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/angelnu/k8s-gitops

My home Kubernetes cluster managed with git-ops
https://github.com/angelnu/k8s-gitops

ansible git-ops gitops home-assistant k8s-at-home kubernetes sops

Last synced: about 2 months ago
JSON representation

My home Kubernetes cluster managed with git-ops

Host: GitHub
URL: https://github.com/angelnu/k8s-gitops
Owner: angelnu
License: apache-2.0
Created: 2021-02-12T19:01:26.000Z (over 3 years ago)
Default Branch: main
Last Pushed: 2024-04-13T21:05:20.000Z (5 months ago)
Last Synced: 2024-04-14T08:51:34.923Z (5 months ago)
Topics: ansible, git-ops, gitops, home-assistant, k8s-at-home, kubernetes, sops
Language: Shell
Homepage:
Size: 4.42 MB
Stars: 121
Watchers: 5
Forks: 15
Open Issues: 9
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

        

#### k8s-gitops - Home Cloud via Flux v2 | GitOps Toolkit

> GitOps state for my cluster using flux v2

[![Discord](https://img.shields.io/badge/discord-chat-7289DA.svg?maxAge=60&style=flat-square)](https://discord.gg/DNCynrJ)

[![test](https://github.com/angelnu/k8s-gitops/workflows/test/badge.svg)](https://github.com/angelnu/k8s-gitop/workflows/actions)

[![renovate](https://github.com/angelnu/k8s-gitops/workflows/renovate/badge.svg)](https://github.com/angelnu/k8s-gitop/workflows/renovate/actions)

[![update-flux](https://github.com/angelnu/k8s-gitops/workflows/update-flux/badge.svg)](https://github.com/angelnu/k8s-gitop/workflows/update-flux/actions)




K3S multi-arch highly available cluster installed via [Ansible](ansible/README.md) on Proxmox VMs.

The cluster is designed to allow tearing it completely without any data lost.

Stack is ordered in multiple layers (Flux kustomizations) depending on the lower one (example apps depend on infrasteructure).

* Applications

  * [default](apps/default)

  * [kube-system](apps/kube-system)

  * [podinfo](apps/podinfo)

  * [vpn](apps/vpn)

* Core

  * [ceph](core/ceph)

  * [ceph-rbd](core/ceph-rbd)

  * [cert-manager](core/cert-manager)

  * [flux-system](core/flux-system)

  * [kube-system](core/kube-system)

  * [monitoring](core/monitoring)

  * [nginx](core/nginx)

  * [postgres](core/postgres)

  * [redis](core/redis)

  * [vpn](core/vpn)

  * [vpn-gateway](core/vpn-gateway)

* Custom Resource Definitions

  * [cert-manager](crds/cert-manager)

  * [multus](crds/multus)

  * [postgres](crds/postgres)

* Settings and Secrets

  * [global settings](settings/settings.yaml)

  * [global secrets](settings/secrets.yaml)

  * [production global settings](settings/production/settings.yaml)

  * [production global secrets](settings/production/secrets.yaml)

  * [staging global settings](settings/staging/settings.yaml)

  * [staging global secrets](settings/staging/secrets.yaml)

* Base

  * [sources](base/sources)

  * [namespaces](base/namespaces)

  * [flux-system](base/flux-system)

* Clusters:

  * [production](clusters/production)

  * [staging](clusters/staging)

* Persistance:

  * Cluster configuration:

    * [flux2](https://github.com/fluxcd/flux2) - Keep cluster in sync with this repo

  * Secrets - see [Secret Management](##-Secret-Management)::

    - [Ansible Vault](ansible) - Ansible, Deployment

    - [SOPS](##-Secret-Management) - Flux, K8S GitOps

  * Files:

    * Fast but depending on Sinology NAS: nfs

    * Slower but replicated: Ceph in Promox

  * Databases:

    * postgres: 3 instances deployed via [Zalando´s Postgres Operator](https://github.com/zalando/postgres-operator)

## HW setup

- 3x Intel NUC 11 vPro (NUC11TNHv5) with:

  - 11th Gen Intel® Core™ i5-1145G7 @ 2.60GHz

  - 32 GB DDR4 

  - 250 GB Sata SSD for local disks - 2x Samsung SSD 850 EVO, 1x CT240BX500SSD1

  - 500 GB NVME for Ceph - WDC WDS500G1B0C-00S6U0

  - 2 Thunderbolt 4/3 connected as network mesh for ceph:

    - Node 1, Port 1 <-> Node 2, Port 1

    - Node 1, Port 2 <-> Node 3, Port 1

    - Node 2, Port 2 <-> Node 2, Port 2

## Installation

### Install / Update / Uninstall

Installed via [Ansible](ansible/README.md). It creates the VMs for the 3 nodes

The cluster is designed to allow tearing the cluster completly without any data lost. 

## Secret Management

Master secret is stored in [Ansible Vault](ansible/README.md).

Kubernetes passwords and secrets encrypted with [mozilla SOPS](https://github.com/mozilla/sops) which it is [supported out of the box in Flux2](https://toolkit.fluxcd.io/guides/mozilla-sops/).

GPG key is deployed via [Ansible](ansible/README.md). Its hash must be kept in sync with [.sops.yaml](.sops.yaml).

Based on [Vaskozl](https://github.com/Vaskozl/home-infra) I use a [pre-commit hook](scripts/find-unencrypted-secrets.sh) to ensure that secrets are never pushed unencrypted. The hook is deployed by running `cd scripts; ./install_git_hooks.sh`

To encrypt files with secrets use:

```

sops -e -i my-secret.yaml # Initial encrypt

sops my-secret.yaml # To edit it directly in you $EDITOR

```

## Useful commands

- Delete stuck objects (PVs, PVCs)

  ```

  kubectl patch   -p '{"metadata":{"finalizers": []}}' --type=merge

  ```

- Delete stuck NSs

  ```

  NAMESPACE=your-rogue-namespace

  kubectl proxy &

  kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >/tmp/patch.json

  curl -k -H "Content-Type: application/json" -X PUT --data-binary @/tmp/patch.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize

  ```

## :handshake:  Community

This cluster in inspired by the work of others shared at [awesome-home-kubernetes](https://github.com/k8s-at-home/awesome-home-kubernetes). These projects

can be [searched in GitHub](https://nanne.dev/k8s-at-home-search/). They include the [k8s-at-home topic](https://github.com/topics/k8s-at-home).

There is also an active [k8s@home Discord](https://discord.gg/7PbmHRK) for this community.