{"id":13631998,"url":"https://github.com/helmut-hoffer-von-ankershoffen/ceil","last_synced_at":"2025-06-12T04:14:51.050Z","repository":{"id":132152188,"uuid":"188037296","full_name":"helmut-hoffer-von-ankershoffen/ceil","owner":"helmut-hoffer-von-ankershoffen","description":"Helmut Hoffer von Ankershoffen experimenting with auto-provisioned RPi cluster running K8S on bare-metal","archived":false,"fork":false,"pushed_at":"2019-12-09T08:37:10.000Z","size":935,"stargazers_count":48,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-04T11:10:00.491Z","etag":null,"topics":["ansible","cloudflare","ddclient","dhcpcd","dnsmasq","glusterfs","grafana","heketi","helm","hoffer-von-ankershoffen","hypriot","k8s","kadmin","metalb","openvpn","prometheus","rpi","software-engineering","tiller","traefik"],"latest_commit_sha":null,"homepage":"https://helmut.hoffer-von-ankershoffen.me","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/helmut-hoffer-von-ankershoffen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-05-22T12:47:09.000Z","updated_at":"2025-02-22T15:59:38.000Z","dependencies_parsed_at":"2024-01-02T22:30:00.495Z","dependency_job_id":"ad0bfe1d-1580-4027-9f0c-13f9d35268c1","html_url":"https://github.com/helmut-hoffer-von-ankershoffen/ceil","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/helmut-hoffer-von-ankershoffen/ceil","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helmut-hoffer-von-ankershoffen%2Fceil","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helmut-hoffer-von-ankershoffen%2Fceil/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helmut-hoffer-von-ankershoffen%2Fceil/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helmut-hoffer-von-ankershoffen%2Fceil/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/helmut-hoffer-von-ankershoffen","download_url":"https://codeload.github.com/helmut-hoffer-von-ankershoffen/ceil/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helmut-hoffer-von-ankershoffen%2Fceil/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259395458,"owners_count":22850833,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","cloudflare","ddclient","dhcpcd","dnsmasq","glusterfs","grafana","heketi","helm","hoffer-von-ankershoffen","hypriot","k8s","kadmin","metalb","openvpn","prometheus","rpi","software-engineering","tiller","traefik"],"created_at":"2024-08-01T22:02:47.625Z","updated_at":"2025-06-12T04:14:51.020Z","avatar_url":"https://github.com/helmut-hoffer-von-ankershoffen.png","language":"Shell","funding_links":[],"categories":["Shell"],"sub_categories":[],"readme":"# `ceil`: Auto-provisioned RPi cluster running K8S on bare-metal\n\nEnter `make help` to see available commands.\n\nWhy the name? `intval(ceil(M_PI)) === 4` which is the number of k8s nodes of the ceil cluster - flowers to mlande for gifting the name.\n\nSee [branch max](https://github.com/helmuthva/ceil/tree/max) for the Mini PC (amd64) variant.\n\nAuthor: [**Helmut Hoffer von Ankershoffen né Oertel**](https://helmut.hoffer-von-ankershoffen.me/)\n\n## Goals\n\n* Setup auto-provisioned RPi cluster running K8S on bare-metal behind a RPi acting as a router\n* Educate myself on Ansible + RPi + K8S + GitOps for CI/CD/PD from bottom to top\n* Refresh knowledge regarding networking and Python\n* Enhanced PHP/SF4 stack for K8S supporting HPA, progressive deployments and a/b testing\n\n## Tasks\n\n### Phase 0: Hardware\n\n![alt text](https://raw.githubusercontent.com/helmuthva/ceil/master/doc/assets/ceil.jpg \"Ceil Rack\")\n\n- [x] Wire up RPi rack and accessories\n\n### Phase 1: Foundation\n\n- [x] Central CloudOps entrypoint is `make`\n- [x] Flashing of RPis and automatic provisioning with pre-configured base OS\n- [x] Setup and teardown of all steps individually\n- [x] Setup and teardown in one step\n- [x] Setup of k8s cluster on RPis using Ansible inc. weave networking and k8s dashboard\n- [x] Helm/tiller for additional deployments\n- [x] Traefik as ingress inc. Traefik dashboard\n- [x] busybox-http using Traefik as ingress for demos\n- [x] Grafana and prometheus\n\n### Phase 2: Storage and Loadbalancing\n\n- [x] Dynamic volume provisioning using Heketi + GlusterFS spanning thumb drives\n- [x] Enabled persistence for grafana and prometheus\n- [x] MetalLB as LoadBalancer service\n\n### Phase 3: Router\n\n- [x] Act as DHCP client using dhcpcd\n- [x] Act as DHCP \u0026 DNS server for K8S subnet using dnsmasq\n- [x] Act as gateway from wlan0 (WiFi) to eth0 (K8S subnet) using iptables\n- [x] Act as VPN server using OpenVPN\n- [x] Dynamically update domain vpn.ceil.pro (or similar) using ddclient and Cloudflare v4 API\n- [x] Raise Firewall using ufw\n- [x] Act as Docker registry mirror using official docker image `registry:2`\n- [x] Act as private Docker registry\n- [ ] kail and harbor\n- [ ] ngrok\n\n### Phase 4: PiWatch\n\n- [x] Play with [PiTraffic Lights](https://sb-components.co.uk/pi-traffic.html) mounted on top of `ceil-router`\n- [x] Deploy kubewatch to push K8S events to arbitrary webhook\n- [x] Build dockerized Python/FastAPI (ASGI) based webapp [PiWatch](https://github.com/helmuthva/piwatch) triggering PiTraffic as audiovisual event handler for K8S by providing webhook for kubewatch\n- [ ] Refine `PiWatch` to react more fine granular to specific K8S events\n\n### Phase 5: PiPHP\n\n- [x] Deploy custom built base image [arm32v7-docker-php-apache](https://github.com/helmuthva/arm32v7-docker-php-apache) to k8s from private registry provided by router. Further progress of the base image tracked in respective repository.\n- [x] Prepare [Helmuts Helm Chart Repository](https://helmuthva.github.com/helm) hosted on ghpages.\n- [x] Prepare [PiPHP](https://helmuthva.github.com/piphp) docker image based on said base image inc. helm chart and redeploy. Further progress of said app tracked in said repository.\n- [x] Automate build-\u003edeploy workflow inc. helming locally.\n- [ ] Automate full CI/CD workflow with GitHub Actions or similar.\n\n### Phase 6: Auto-Scaling\n- [ ] Autoscaling using HPA and custom metrics\n- [ ] Zero-Scaling using Osiris\n- [ ] Relevant dashboards in grafana\n\n### Phase 7: Mesh-Networking (waiting for ARM images from CNCF et al)\n- [ ] Istio for Mesh-Networking\n- [ ] Visibility tools\n- [ ] Additional tools\n\n### Phase 8: GitOps and Progressive Delivery (waiting for ARM images from CNCF et al)\n\n- [ ] Flagger for Helm using mesh network\n- [ ] Canary deployments using mesh network\n- [ ] ...\n\n### Phase 9: CI and emphemeral test environments (waiting for ARM images from CNCF et al)\n- [ ] Setup CI using JenkinsX\n- [ ] ...\n\n\n### Phase 10: A/B testing (waiting for ARM images from CNCF et al)\n\n- [ ] Using mesh network\n- [ ] ...\n\n### Phase 11: Sharing is caring\n\n- [x] Open source under GPlv3\n- [x] Links to useful material for further studies\n- [ ] GitHub Page\n- [ ] Prepare interactive install script automating the step to manually copy and edit `.tpl` files\n- [ ] Write a series of blog posts\n- [ ] Prepare a workshop presentation\n- [ ] Educate peers in meetups\n\n## Layers and tools\n\n* CloudOps\n  * Workstation: MacBook Pro\n  * Package manager: Homebrew\n  * Flash-Tool for OS of RPis: Hypriot Flash\n  * Entrypoints: `make` and `kubectl` (GitOps in second step)\n* Hardware\n  * SBCs: 5x Raspberry Pi 3B+\n  * Storage: 5x 128GiB SD cards (containers), 5x 128GiB USB ThumbDrives (volumes)\n  * Rack: transparent\n  * Networking: 5-port GBit/s switch + WiFi router connected to router\n  * Power: 6-port USB charger powering switch and RPIs\n  * 4-dir traffic lights with beeper and button: [PiTraffic](https://sb-components.co.uk/pi-traffic.html)\n* Software\n  * OS: Debian, Hypriot distribution\n  * Networking for router: iptables, dhcpcd, dnsmasq, OpenVPN, ddclient, CloudFlare\n  * Configuration management: Ansible\n  * Orchestration: Kubernetes (K8S)\n  * K8S installation: `kadm`\n  * Networking: weave\n  * Persistence: GlusterFS + Heketi for dynamic volume provisioning\n  * Ingresss: Traefik\n  * Loadbalancer: MetaLB\n  * Deployments: helm\n  * Monitoring and Dashboarding: prometheus, grafana\n  * Traffic lights: kubewatch, Python, Flask, PiTraffic, RPi.GPIO\n\n## Install this repository\n\n1) Fork this repository and clone to your workstation\n2) Walk all files with suffix `.tpl`, create a copy in the same directory without said suffix and enter specifics where invited by capital letters\n\n## Provision RPIs\n\n1) Prepare you workstation by installing Ansible, kubectl, helm etc. using homebrew: `make prepare-mac`\n2) Pull the hypriot image (which is not stored  in GitHub): `make pull-image`\n3) Flash RPIs (insert SD cards in your workstation): `make {router,one,two,three,four}-provision`\n4) Insert SD cards into slots of respective RPIs\n5) Insert thumb drives into USB ports of RPIs\n6) Start RPIs by plugging in the USB charger\n\n## Setup router\n\n1) Make a DHCP reservation for `ceil-router` on your home or company WiFi router with IP address `192.168.0.100` -  it will register as `ceil-router` at your WiFi router\n2) Set up a static route to the k8s subnet `11.0.0.0` with `192.168.0.100` as gateway in your company or home wifi router - if this is not achievable use `make workstation-route-add` to add a route on your workstation.\n3) For VPN setup port forwarding (sometimes called \"virtual server\") in your company or home wifi router for port `1194` (or whatever you configured in `router/roles/vpn/defaults/main.yml`) to `192.168.0.100`\n4) Add `192.168.0.100` as the first nameserver for the (WiFi) connection of your workstation using system settings\n5) Reboot `ceil-router` to pickup its IP address via `make router-reboot` - it will register via ZeroConf/Avahi on your workstation as `ceil-router.local`\n6) Check via `make router-check-ip` if the IP address has been picked up\n7) Setup networking services on router using `make router-setup`\n8) Wait for 1 minute than check if the k8s nodes (`ceil-{one,two,three,four}.dev`) have picked up their designated IP addresses from the router in the range `11.0.0.101` to `11.0.0.104`:  `make k8s-check-ip` \n\nNotes:\n- Danger: wipes thumb drive in router\n- It might take some time until the Zeroconf/Avahi distributed the name `ceil-router.local` in your network. You can check by ssh'ing into the router via `make router-ssh`\n- The router will manage / route to the subnet `11.0.0.[0-128]` (`11/25`) the K8S nodes will life in and act as their DHCP and DNS server\n- Furthermore the router acts as an OpenVPN server and updates the IP address of `vpn.ceil.pro` via DDNS\n- After setting up the router wait for a minute to check if the k8s nodes have picked up the designated IPs using `make k8s-check-ip`\n- After the k8s nodes picked up their IP addresses you can ssh into them using `make {one,two,three,four}-ssh`\n- If on your workstation `nslookup ceil-{one,two,three.four}.dev` works but `ping ceil-{one,two,three.four}.dev` does not, reestablish the (WiFi) connection of your workstation\n- If you want to play with the traffic lights mounted on top of the router: `make router-traffic`\n- The last step of the router setup is building [PiWatch](https://github.com/helmuthva/piwatch) which takes ca. 15 minutes for the 1st build\n- Last but not least the router provides a docker registry mirror and private docker registry consumed by the K8S nodes\n\n## Setup K8S and execute all deployments\n\n1) Execute `make setup` to setup K8S inc. persistence and deploy everything at once - takes ca. 45 minutes. \n\nNotes:\n- `ceil-one` is set up as k8s master\n- Danger: wipes thumb drives for setting up GlusterFS.\n- Because of memory constraints the GlusterFS spans `ceil-two` to `ceil-four` but not `ceil-one`\n\nAlternatively you can execute the setup and deploy steps one-by-one as described below\n\n## Interact, open dashboards and UIs\n\n1) Establish proxy to cluster (leave open in separate terminal): `make k8s-proxy` \n2) List nodes: `make nodes-show`\n3) List pods: `make pods-show`\n4) Generate bearer token for accessing K8S dashboard: `make  k8s-dashboard-bearer-token-show`\n5) Access K8S dashboard in your browser and enter token: `make k8s-dashboard-open`\n6) Open Traefik UI in your browser: `make traefik-ui-open`\n8) Show webpage in your browser: `make httpd-open`\n8) Open Prometheus UI in your browser: `make prometheus-open`\n9) Open Grafana dashboards in your browser: `make grafana-open`\n\nNotes:\n- Add the contents of `workstation/etc/hosts` to `/etc/hosts` of your workstation for steps 6 to 9\n\n## Setup K8S inc. persistence and helm/tiller\n\n1) Setup K8S cluster inc. persistence via GlusterFS+Heketi and helm/tiller for later deployments: `make k8s-setup`. \n\nNotes:\n- `ceil-one` is set up as k8s master\n- Danger: wipes thumb drives for setting up GlusterFS.\n- Because of memory constraints the GlusterFS spans `ceil-two` to `ceil-four` but not `ceil-one`\n\n## Deploy\n\n1) Execute all deployments using `make all-deploy` or deploy step by step as documented below.\n2) Interact, open dashboards and UIs as documented above.\n\n## Delete deployments\n\n1) All deployments provide an individual make target for deleting the deployment, e.g. `ngrok-delete`. Execute `make help` to see all commands.\n2) Execute `make all-delete` to delete all deployments at once\n\n## Remove K8S inc. persistence and helm/tiller\n\n1) Execute `make k8s-remove`.\n\n## Teardown\n\n1) Execute `make teardown` to delete all deployments and remove K8S.\n\n## Obstacles \n\n* Examples for setting up K8S on bare metal mostly outdated and/or incomplete or making undocumented assumptions or not using Ansible correctly =\u003e full rewrite\n* Current Kernel of hypriot does not setup pid cgroup which is used by newer K8S for QoS =\u003e downgrade K8S\n* RBAC is rather new and not yet accounted for in deployment procedures of all tools and services =\u003e amend\n* Traefik image of hypriot outdated, dashboard not useable =\u003e use original image given manifest lists \n* Some services do not yet compile docker images for ARM and/or do not use docker manifest lists properly =\u003e google for alternative images or wait for CNCF\n* Most ansible playbooks do not provide a teardown role =\u003e build yourself\n\n## Additional references\n\n* https://medium.com/@evnsio/managing-my-home-with-kubernetes-traefik-and-raspberry-pis-d0330effea9a (ddns, vpn, let's encrypt)\n* https://github.com/luxas/kubeadm-workshop (custom autoscaling, by luxas)\n* http://slides.com/lucask/kubecon-berlin#/18 (multiplatform K8S, by luxas)\n* https://luxaslabs.com/ (slides by luxas)\n* https://medium.com/vescloud/kubernetes-storage-performance-comparison-9e993cb27271 (Kubernetes Storage Performance Benchmark)\n* https://tobru.ch/kubernetes-on-orangepi-arm64/ (unsorted)\n* https://medium.com/@carlosedp/multiple-traefik-ingresses-with-letsencrypt-https-certificates-on-kubernetes-b590550280cf (traefik,let's encrypt)\n* https://medium.com/@carlosedp/building-a-hybrid-x86-64-and-arm-kubernetes-cluster-e7f94ff6e51d (unsorted)\n* https://www.gopeedesignstudio.com/2018/07/13/glusterfs-on-arm/ (glusterfs on arm)\n* https://stefanprodan.com/2018/expose-kubernetes-services-over-http-with-ngrok/ (ngrok, k8s)\n* https://downey.io/blog/how-to-build-raspberry-pi-kubernetes-cluster/ (router)\n* https://downey.io/blog/create-raspberry-pi-3-router-dhcp-server/ (router,dhcp)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhelmut-hoffer-von-ankershoffen%2Fceil","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhelmut-hoffer-von-ankershoffen%2Fceil","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhelmut-hoffer-von-ankershoffen%2Fceil/lists"}