Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ricsanfre/pi-cluster

Pi Kubernetes Cluster. Homelab kubernetes cluster automated with Ansible and FluxCD
https://github.com/ricsanfre/pi-cluster

ansible arm64 cluster fluxcd gitops homelab k3s kubernetes raspberry-pi x86-64

Last synced: 9 days ago
JSON representation

Pi Kubernetes Cluster. Homelab kubernetes cluster automated with Ansible and FluxCD

Awesome Lists containing this project

README 

        
# Pi Kubernetes Cluster






  

    pi-cluster-1.0

    

  

    



**K3S Kubernetes Cluster at home automated with Ansible and FluxCD**



This is an educational project to build a hybrid x86/ARM Kubernetes cluster at home, using Raspberry Pi and refurbished x86 mini PCs, learn to deploy basic kubernetes services and automate its deployment and configuration applying IaC (infrastructure as a code) and GitOps methodologies.



The entire process for creating this cluster at home, from cluster design and architecture to step-by-step manual configuration guides, has been documented and it is published in the project website: https://picluster.ricsanfre.com.



This repository contains all source code used to automate all manual tasks described in the documentation: Cloud-init's configuration files, Ansible's source code (playbooks/roles), and packaged Kubernetes applications (helm and kustomize) to be deployed using FluxCD. 



Since its deployment is completely automated, the cluster can be re-deployed in minutes as many times as needed for testing new cluster configurations, new software versions or just take you out of any mesh you could cause playing with the cluster.



## Scope



The scope of this project is to build a hybrid x86/ARM kubernetes cluster at home, using low cost Raspeberry PIs and old refurbished mini PCs, and automate its deployment and configuration applying **IaC (infrastructure as a code)** and **GitOps** methodologies with tools like [Ansible](https://docs.ansible.com/), [cloud-init](https://cloudinit.readthedocs.io/en/latest/) and [Flux CD](https://fluxcd.io/).



As part of the project, the goal is to use a lightweight Kubernetes flavor based on [K3S](https://k3s.io/) and deploy cluster basic services such as:

- Distributed block storage for POD's persistent volumes, [LongHorn](https://longhorn.io/).

- S3 Object storage, [Minio](https://min.io/).

- Backup/restore solution for the cluster, [Velero](https://velero.io/) and [Restic](https://restic.net/).

- Certificate management, [Cert-Manager](https://cert-manager.io).

- Secrets Management solution with [Vault](https://www.vaultproject.io/) and [External Secrets](https://external-secrets.io/)

- Identity Access Management(IAM) providing Single-sign On, [Keycloak](https://www.keycloak.org/)

- Observability platform based on:

   - Metrics monitoring solution, [Prometheus](https://prometheus.io/)

   - Logging and analytics solution, combined EFK+LG stacks ([Elasticsearch](https://www.elastic.co/elasticsearch/)-[Fluentd](https://www.fluentd.org/)/[Fluentbit](https://fluentbit.io/)-[Kibana](https://www.elastic.co/kibana/) + [Loki](https://grafana.com/oss/loki/)-[Grafana](https://grafana.com/oss/grafana/))

   - Distributed tracing solution, [Tempo](https://grafana.com/oss/tempo/).



Also deployment of services for building a cloud-native microservices architecture are include as part of the scope:



- Service mesh architecture, [Istio](https://istio.io/)

- API security with Oauth2.0 and OpenId Connect, using IAM solution, [Keycloak](https://www.keycloak.org/)

- Streaming platform, [Kafka](https://kafka.apache.org/)



## Technology Stack



The following picture shows the set of opensource solutions used so far in the cluster, which installation process has been documented and its deployment has been automated with Ansible/FluxCD:





  







    

        

        Name

        Description

    

    

        

        Ansible

        Automate OS configuration, external services installation and k3s installation and bootstrapping

    

    

        fluxcd logo

        FluxCD

        GitOps tool for deploying applications to Kubernetes

    

    

        

        Cloud-init

        Automate OS initial installation

    

    

        

        Ubuntu

        Cluster nodes OS

    

    

        

        K3S

        Lightweight distribution of Kubernetes

    

    

        

        containerd

        Container runtime integrated with K3S

    

    

        cilium logo

        Cilium

        Kubernetes Networking (CNI) and Load Balancer

    

    

        

        CoreDNS

        Kubernetes DNS

    

    

        

        HA Proxy

        Kubernetes API Load-balancer

    

    

        

        Metal LB

        Load-balancer implementation for bare metal Kubernetes clusters (Cilium LB alternative)

    

    

        

        Ingress NGINX

        Kubernetes Ingress Controller

     

    

        istio logo

        Istio

        Kubernetes Service Mesh

    

    

        

        Longhorn

        Kubernetes distributed block storage

    

    

        

        Minio

        S3 Object Storage solution

    

    

        

        Cert-manager

        TLS Certificates management

    

    

        

        Hashicorp Vault

        Secrets Management solution

    

    

        

        External Secrets Operator

        Sync Kubernetes Secrets from Hashicorp Vault

    

    

        keycloak logo

        Keycloak

        Identity Access Management

    

    

        oauth2-proxy logo

        OAuth2.0 Proxy

        OAuth2.0 Proxy

    

    

        

        Velero

        Kubernetes Backup and Restore solution

    

    

        

        Restic

        OS Backup and Restore solution

    

    

        

        Prometheus

        Metrics monitoring and alerting

    

    

        

        Fluentd

        Logs forwarding and distribution

    

    

        

        Fluentbit

        Logs collection

    

    

        

        Loki

        Logs aggregation

    

    

        

        Elasticsearch

        Logs analytics

    

    

        

        Kibana

        Logs analytics Dashboards

    

    

        

        Tempo

        Distributed tracing monitoring

    

    

        

        Grafana

        Monitoring Dashboards

    






## External Resources and Services



Even whe the premise is to deploy all services in the kubernetes cluster, there is still a need for a few external services/resources. Below is a list of external resources/services and why we need them.



### Cloud external services



|  |Provider | Resource | Purpose |

| --- | --- | --- | --- |

| | [Letsencrypt](https://letsencrypt.org/) | TLS CA Authority | Signed valid TLS certificates |

|  |[IONOS](https://www.ionos.es/) | DNS | DNS and [DNS-01 challenge](https://letsencrypt.org/docs/challenge-types/#dns-01-challenge) for certificates |



> **NOTE:** These resources are optional, the homelab still works without them but it won't have trusted certificates



**Alternatives:**



1. Use a private PKI (custom CA to sign certificates).



   Currently supported. Only minor changes are required. See details in [Doc: Quick Start instructions](https://picluster.ricsanfre.com/docs/ansible).



2. Use other DNS provider.



   Cert-manager / Certbot, which are the tools that automatically obtain certificates from Let's Encrypt, can be configured to use other DNS providers. It will need further modifications in the way cert-manager application is deployed (new providers and/or webhooks/plugins might be required).



   Currently only acme issuer (letsencytp) using IONOS as dns-01 challenge provider is configured. Check list of [supported dns01 providers](https://cert-manager.io/docs/configuration/acme/dns01/#supported-dns01-providers).



### Self-hosted external services 



There is another list of services that I have decided to run outside the kuberentes cluster selfhosting them..



|  |External Service | Resource | Purpose |

| --- | --- | --- | --- |

|  |[Minio](https://min.io) | S3 Object Store | Cluster Backup  |

|  |[Hashicorp Vault](https://www.vaultproject.io/) | Secrets Management | Cluster secrets management |



Minio backup servive is hosted in a VM running in Public Cloud, using [Oracle Cloud Infrastructure (OCI) free tier](https://www.oracle.com/es/cloud/free/).



Vault service is running in `gateway` node, since Vault kubernetes authentication method need access to Kuberentes API, I won't host Vault service in Public Cloud.



## Cluster architecture and hardware



Home lab architecture, showed in the picture below, consist of a Kubernetes cluster of ARM (Rasbperry PI) and x86 (HP elitedesk 800 G3 mini PCs) nodes and a firewall, built with another Raspberry PI, to isolate cluster network from your home network.





  




See further details about the architecture and hardware in the [documentation](https://picluster.ricsanfre.com/docs/home/)



## Official Site



You can browse more information about Pi Cluster Project on https://picluster.ricsanfre.com/. 



The content of this website and the source code to build it (Jekyll static based website) are also stored in this repo: `/docs` folder.



## Usage 



Check out the documentation [Quick Start guide](http://picluster.ricsanfre.com/docs/ansible/) to know how to use and tweak cloud-init files (`/cloud-init` folder), Ansible playbooks (`/ansible` folder) and packaged Kubernetes applications ( `/kubernetes` folder) contained in this repository, so you can use in for your own homelab.



## About the Project



This project was started in June 2021 by Ricardo Sanchez