Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sleighzy/raspberry-pi-k3s-homelab
Repository containing random notes on building my Raspberry Pi cluster and K3s deployment
https://github.com/sleighzy/raspberry-pi-k3s-homelab
k3s k3s-cluster k3sup raspberry-pi-4 rpi4
Last synced: 3 months ago
JSON representation
Repository containing random notes on building my Raspberry Pi cluster and K3s deployment
- Host: GitHub
- URL: https://github.com/sleighzy/raspberry-pi-k3s-homelab
- Owner: sleighzy
- Created: 2020-12-21T05:07:25.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2022-01-11T20:00:38.000Z (almost 3 years ago)
- Last Synced: 2024-09-28T10:04:28.923Z (4 months ago)
- Topics: k3s, k3s-cluster, k3sup, raspberry-pi-4, rpi4
- Homepage:
- Size: 3.47 MB
- Stars: 60
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Raspberry Pi K3s Homelab
This repository contains my random notes on building and deploying a [K3s]
Kubernetes cluster using Raspberry Pis. I have a short memory so this makes it
easier to remember why/how I did things...and may help others who travel down
this path.
![k3s-cluster]
- [Raspberry Pi](#raspberry-pi)
- [Ubuntu OS Installation](#ubuntu-os-installation)
- [Using WiFi](#using-wifi)
- [Booting from SSD](#booting-from-ssd)
- [tl;dr](#tl-dr)
- [Troubleshooting](#troubleshooting)
- [Deploying K3s](#deploying-k3s)
- [Cluster Monitoring and Grafana Loki](#cluster-monitoring-and-grafana-loki)
- [Deployed Applications](#deployed-applications)
## Raspberry Pi
[Raspberry Pi] are small single-board computers ideal for home projects,
clustering, and IoT edge devices. They are built using ARM processors and
provide support for numerous input and output devices.
My cluster is built with three [Raspberry Pi 4 Model B]:
- Quad core ARM v8 (64bit) 1.5GHz CPU
- 8Gb RAM
- 2.45GHz and 5GHz Wi-Fi
- Bluetooth
- Gigabit ethernet
- 2 USB 2 and 2 USB 3 ports
- 2x micro-HDMI ports, supports 2 4K monitors
- Micro-SD card for operating system and internal data storage
I am running Ubuntu 20.04.2 LTS 64bit as the operating system.
Additional hardware for each:
- [Kingstone A2000] 1TB M.2 NVMe SSD
- [Orica NVMe M.2] transparent USB-C SSD enclosure
- [Yahboom RGB Cooling HAT] with fan and OLED display
## Ubuntu OS Installation
The [Install Ubuntu on a Raspberry Pi] documentation and installer is extremely
easy to follow to install the Ubuntu OS on your SD card. I chose the recommended
Ubuntu Server 20.04.2 LTS 64bit option as I was more interested in running this
headless, for now, than installing a full desktop.
**Note:** You will need to install Raspberry Pi OS on the SD card and Ubuntu on
the SSD if you follow the instructions for booting your Pi off an SSD and no
longer needing to use an SD card.
### Using WiFi
I found that wifi did not work automatically with Ubuntu, even with the entry in
the `network-config` file. I needed to follow the instructions at
to add
the wifi network settings to the `/etc/netplan/50-cloud-init.yaml` file.
```sh
# update the 50-cloud-init.yaml file with the wifi network settings
$ sudoedit /etc/netplan/50-cloud-init.yaml
$ cat /etc/netplan/50-cloud-init.yaml
network:
ethernets:
eth0:
dhcp4: true
optional: true
version: 2
wifis:
wlan0:
optional: true
access-points:
"my-wifi-ssid":
password: "my-wifi-password"
dhcp4: true
# apply the settings
$ sudo netplan apply
# install net-tools to see adapter ip addresses
$ sudo apt install net-tools
# check that wifi adapter has an ip address
$ ip a
```
## Booting from SSD
SD cards are limited in capacity (in comparison to HDD/SSD with terrabytes of
space) and reported to have a short-lifetime span. An external drive can be
mounted onto the Raspberry Pi for storage, but for raw performance and removing
the dependency on the SD card for the OS an SSD drive can be used to boot and
run the Pi.
I have used an NVMe.m2 drive with a USB-C drive enclosure and Ubuntu to run the
Raspberry Pi as per a number of other blogs and guides.
The instructions on
were excellent and easily followed. I only had a couple of items:
- I used the `stable` release channel for the EEPROM upgrade, Jame's indicates
that this is no longer necessary but I kept for consistency across the Pis as
it worked.
- I needed to follow the additional official Ubuntu docs to update the boot
config so that the Pi would attempt to boot off an SSD drive if no SD card was
found. See
for further information.
### TL;DR
Steps and commands I ran for future reference. Please read Jame's guide in full
as he explains each as well as the manual steps in the automated script.
- Write the Raspberry Pi OS to the SD card using the Raspberry Pi Imager
application
- Write the Ubuntu Server 20.04.2 64bit LTS to the SSD drive, the Raspberry Pi
Imager application will accept the SSD drive as an SD card.
- Boot the Raspberry Pi from the SD card, don't attach the SSD drive yet
Run the below commands to update the EEPROM and set bootloader config to boot
from SSD if no SD card:
```sh
# update to the "stable" channel for firmware releases
# Note: Jame's guide states that this is no longer necessary and he doesn't recommend
# edit the /etc/default/rpi-eeprom-update file and update to FIRMWARE_RELEASE_STATUS="stable"
sudo nano /etc/default/rpi-eeprom-update
# get the latest release
sudo rpi-eeprom-update
# get boot config from the latest EEPROM, this directory /stable/ matches the channel we used
# above, this may need to be /critical/ if this was left unchanged
rpi-eeprom-config --out pieeprom-new.bin --config bootconf.txt /lib/firmware/raspberrypi/bootloader/stable/pieeprom-2021-03-18.bin
# update boot config to change BOOT_ORDER from 0x1 (SD card only)
# to 0xf41 (SSD drive if no SD card)
sed -i -e '/^BOOT_ORDER=/ s/=.*$/=0xf41/' bootconf.txt
# apply new EEPROM and updated boot config
$ sudo rpi-eeprom-update -d -f ./pieeprom-new.bin
*** INSTALLING ./pieeprom-new.bin ***
BOOTFS /boot
EEPROM update pending. Please reboot to apply the update.
sudo reboot
# login again and check the boot loader version
$ vcgencmd bootloader_version
Mar 18 2021 08:54:11
version 1b43d5b6fe5b71c300563afc0548122752a0618b (release)
timestamp 1616057651
update-time 1618219141
capabilities 0x0000001f
```
After rebooting has completed attach the SSD drive to the USB 3 port, one of the
blue ones. Run the below commands to have the partitions setup for booting from
the SSD. Raspberry Pi does not support booting from a compressed kernel, the
steps here will automate the decompressing of the kernel and a script that is
run during upgrades to decompress any new updated kernel.
```sh
# check that the SD card and SSD drive are both mounted,
# the SD card is mmcblk0 and SSD is sda
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 256M 0 part /media/pi/system-boot
└─sda2 8:2 0 931.3G 0 part /media/pi/writable
mmcblk0 179:0 0 116.2G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 116G 0 part /
# unmount the SSD drive from the default location
sudo umount /media/pi/writable
sudo umount /media/pi/system-boot
# mount the SSD drive to the location needed for running the script to
# setup the files for booting from the decompressed kernel
sudo mkdir /mnt/boot
sudo mkdir /mnt/writable
sudo mount /dev/sda1 /mnt/boot
sudo mount /dev/sda2 /mnt/writable
# check mounts as expected
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 256M 0 part /mnt/boot
└─sda2 8:2 0 931.3G 0 part /mnt/writable
mmcblk0 179:0 0 116.2G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 116G 0 part /
# run the automated script
sudo curl https://raw.githubusercontent.com/TheRemote/Ubuntu-Server-raspi4-unofficial/master/BootFix.sh | sudo bash
# unmount the SSD drive
sudo umount /mnt/boot
sudo umount /mnt/writable
# shutdown the Raspberry Pi
sudo shutdown -h now
```
Power off the Raspberry Pi, remove the SD card and then start it back up again.
The Raspberry Pi should check for the SD card, and then after failing will boot
off the SSD drive. When you access the Raspberry Pi you should get the Ubuntu
login prompt. The default credentials are `ubuntu` / `ubuntu`.
**Note:** I needed to follow the instructions in the earlier _Using WiFi_
section of this README file to enable wifi to connect to my network. You'll be
unable to ssh to your Pi over wifi without that. You may need to use your
ethernet connection to access the Raspberry Pi initially until you have set that
up.
I followed the above process for my other Raspberry Pis. As a shortcut you can
reuse the same SD card each time so you only need to run the below command,
without needing to run the prior commands to download the latest EEPROM updates
and applying the boot configuration change to it.
`sudo rpi-eeprom-update -d -f ./pieeprom-new.bin`
### Troubleshooting
If the Raspberry Pi is not booting up from the SSD drive and you are unable to
access it check for the flashing led above the SD card slot. The flashing
pattern may provide more information as to the root cause. See
for the list of warning flash codes. I had an issue whereby there were 4 quick
flashes, a pause, and then another repeated 4 flashes. This means
"`start*.elf not found`" which meant it couldn't locate the files for the kernel
to boot. This was not covered in the online guides but turned out to be that I
needed to update my boot config, as detailed in my instructions above, to set
the `BOOT_ORDER` to `0xf41`. With the default setting of `0x1` it was stuck
constantly retrying the SD card, which had been removed, and wasn't trying the
SSD drive.
To check to see if you have missed any steps above you will need to:
- Power off the Raspberry Pi
- Detach the SSD drive
- Insert the SD card
- Power on and log into the Raspberry Pi OS again as you have booted off the SD
card
Once you've gone through the earlier steps and fixed (hopefully) the problem
then just repeat the steps for removing the SD card and booting up from the SSD
drive.
## Deploying K3s
See the [K3s README file] for instructions on installing K3s on the Raspberry
Pis:
- How to deploy nodes using the standard K3s scripts
- How to deploy nodes using the awesome [k3sup] (pronounced "ketchup") tool
- Using external hard drives for storage with the K3s local-storage class
- Building images with BuildKit for containerd
## Cluster Monitoring and Grafana Loki
See the [Cluster Monitoring README file] for instructions on deploying a
monitoring solution for your Raspberry Pi K3s cluster:
- OS information for the Raspberry Pis, e.g. disk space, temperature, etc.
- Metrics on running pods and resource usage
- Grafana dashboards for viewing metrics
- Deployment of Grafana Loki for aggregating and viewing logs
## Deployed Applications
The table below contains references to my Github repositories and instructions
for installing the applications deployed in my cluster.
| Application | Description |
| ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Traefik 2 Kubernetes Ingress Controller CRD] | Manifest files, instructions, and examples for deploying the Traefik 2 Kubernetes Ingress Controller. This uses the new Kubernetes Custom Resource Definition (CRD) for Ingress Routes in Traefik 2. K3s installs Traefik 1.7 so this is a replacement for that. |
| [K3s Keycloak Deployment] | Deployment of [Keycloak] for authentication and authorization to applications and services. |
| [OpenID Connect Traefik Forward Authentication] | Manifest files and instructions for deploying a Traefik Forward Authentication component to delegate authentication to OpenID Connect Providers such as Github or Google. |
| [MinIO S3] | Files for deploying the Open Source [MinIO] server for S3 object storage. Contains additional documentation and policy files for integrating with Keycloak using OAuth2 / OpenID Connect to authenticate and authorize access to resources. |
| [Docker Registry] | Manifest files and instructions for deploying a private Docker Registry and web user interface |
| [Restic MinIO Backups] | Using [Restic] to backup data to [MinIO] S3 storage |
[cluster monitoring readme file]: monitoring-and-logging/README.md
[docker registry]: https://github.com/sleighzy/k8s-docker-registry
[install ubuntu on a raspberry pi]: https://ubuntu.com/download/raspberry-pi
[k3s]: https://k3s.io/
[k3s-cluster]: ./assets/raspberry-pi-4b-cluster.jpg
[k3s keycloak deployment]: https://github.com/sleighzy/k3s-keycloak-deployment
[k3s readme file]: ./k3s.md
[k3sup]: https://github.com/alexellis/k3sup
[keycloak]: https://www.keycloak.org/
[kingstone a2000]: https://www.kingston.com/en/ssd/a2000-nvme-pcie-ssd
[minio]: https://min.io/
[minio s3]: https://github.com/sleighzy/k3s-minio-deployment
[openid connect traefik forward authentication]:
https://github.com/sleighzy/k3s-traefik-forward-auth-openid-connect
[orica nvme m.2]: https://www.orico.cc/usmobile/article/detail/id/255
[raspberry pi]: https://www.raspberrypi.org/
[raspberry pi 4 model b]:
https://www.raspberrypi.org/products/raspberry-pi-4-model-b/specifications/
[restic]: https://restic.net/
[restic minio backups]: restic-minio-backups.md
[traefik 2 kubernetes ingress controller crd]:
https://github.com/sleighzy/k3s-traefik-v2-kubernetes-crd
[yahboom rgb cooling hat]:
https://category.yahboom.net/collections/home-featured-products/products/rgb-cooling-hat