{"id":13581096,"url":"https://github.com/rayui/scooby","last_synced_at":"2026-01-20T17:34:31.992Z","repository":{"id":188806506,"uuid":"573973431","full_name":"rayui/scooby","owner":"rayui","description":"Pretty fast, pretty simple K3S clusters for Raspberry Pi","archived":false,"fork":false,"pushed_at":"2024-07-25T20:01:09.000Z","size":27,"stargazers_count":14,"open_issues_count":7,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-05T20:46:55.393Z","etag":null,"topics":["bare-metal","baremetal","k3s","k3sup","packer","raspberry-pi","vagrant"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rayui.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-04T02:23:35.000Z","updated_at":"2024-07-25T20:01:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"9e52d3de-f140-44a9-8165-8038da04b5b9","html_url":"https://github.com/rayui/scooby","commit_stats":null,"previous_names":["rayui/scooby"],"tags_count":2,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayui%2Fscooby","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayui%2Fscooby/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayui%2Fscooby/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayui%2Fscooby/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rayui","download_url":"https://codeload.github.com/rayui/scooby/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247445652,"owners_count":20939952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bare-metal","baremetal","k3s","k3sup","packer","raspberry-pi","vagrant"],"created_at":"2024-08-01T15:01:58.065Z","updated_at":"2026-01-20T17:34:31.985Z","avatar_url":"https://github.com/rayui.png","language":"Shell","funding_links":[],"categories":["Shell"],"sub_categories":[],"readme":"# Scooby\n\n## Self-hosting RPi K3S cluster\n\nScooby is a build pipeline for creating bare-metal `K3S` clusters of one or more nodes.\n\nScooby attempts to automate the configuration and deployment of a `K3S` cluster into a single Linux image in a simple, repeatable fashion.\n\nScooby is designed to be easy to work with and requires a bare minimum of hardware and software configuration to get up and running.\n\nScooby uses Vagrant, GitHub and optionally, Amazon S3.\n\nFeatures:\n\n- From commit to bootable image in 15 minutes\n- Versioned deployments\n- Centralized configuration\n- Network booted agents\n- Ready-to-go build pipelines\n- Github asset storage\n- Optional S3 bucket uploader\n\n## Getting started\n\n### Gather required hardware\n\nTo get going, you will need at least one Raspberry Pi 3A or higher, and one local storage device (e.g. USB stick) per Pi.\n\nIf you are using two or more Pis you will also need a physical Ethernet switch or VLAN group, and the master node will require a second ethernet device.\n\nSeveral generic USB2 and USB3 devices using the RTL-8152 chipset have been tested and work well. Driverless Ethernet hats should also work but at the current time have not been tested.\n\nAgent nodes perform well on Pi 3A+ but for performance reasons it is recommended to use a Pi 4B or greater for the master node, and a USB3 SSD for master node storage.\n\nTo create a three-node Scooby cluster a complete hardware list might be:\n\n- 1x Raspberry Pi 4B\n- 1x Raspberry Pi 3B\n- 1x Raspberry Pi 3A\n- 4x port Ethernet switch\n- 1x 1TB USB3 SSD external storage\n- 2x 8GB USB2 keys\n- 1x RTL8152 USB3 to Ethernet adapter\n- 4x Ethernet cables\n\n### Clone the template\n\nClick the `use this template` button above, create a new GitHub repository, and clone it to your local machine.\n\n## Network concepts\n\nThe master node in any Scooby cluster of more than one node has two physical ethernet ports and acts as a network gateway for all traffic originating inside the cluster. A Scooby cluster therefore has a concept of `internal` and `external` networks.\n\nThe `internal` network is the physical network on which the master and agent nodes communicate, and to which the `eth1` device of the master node is physically connected.\n\nThe `external` network is a separate physical network with access to the Internet, and to which the `eth0` device of the master node is physically connected.\n\nAll agent nodes' `eth0` devices are connected to the `internal` network.\n\n## Setting up the Master node\n\nThe first file you will need to edit is `/config`, in the project root. This file contains information about the cluster's base operating system and network settings. There is an example configured with default values that you can uncomment and change to get you going. For example:\n\n`/config`\n\n```\n# Base Raspberry Pi image and SHA\nLC_IMAGE_HREF=\"http://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-01-28/2022-01-28-raspios-bullseye-arm64-lite.zip\"\nLC_IMAGE_SHA=sha256:d694d2838018cf0d152fe81031dba83182cee79f785c033844b520d222ac12f5\n\n# Master node hostname\nLC_HOSTNAME=buffy\n\n# DNS providers\nLC_PRIMARY_DNS=1.0.0.1\nLC_SECONDARY_DNS=\n\n# Master node external network configuration\nLC_EXTERNAL_DEVICE=eth0\nLC_EXTERNAL_IP=192.168.1.64\nLC_EXTERNAL_NET=192.168.1.0/24\nLC_EXTERNAL_GW=192.168.1.1\nLC_EXTERNAL_DOMAIN=\n\n# Master node cluster network configuration\nLC_INTERNAL_DEVICE=eth1\nLC_INTERNAL_IP=192.168.64.1\nLC_INTERNAL_NET=192.168.64.0/24\nLC_INTERNAL_DOMAIN=sunnydale\n```\n\nScooby has been tested with both `bullseye` and `bookworm` releases of `raspios`.\n\n### Static files\n\nAny files placed in the `./server` directory in the project root will be merged into the master node's root filesystem.\n\nFor example, `./server/etc/dnsmasq.d/40-hemeryhigh.conf` would be copied to `/etc/dnsmasq.d/40-hemeryhigh.conf` on the master node file system.\n\nFiles are persistent and available on first boot.\n\n#### Kubernetes Manifests\n\nKubernetes manifests should go in `./server/var/lib/rancher/k3s/server/manifests/` in the project root.\n\n## Setting up the Agents\n\n### Enable Pi network boot\n\nEach agent must be setup for network booting. This part cannot be automated and you should perform it manually on each agent. Fortunately, it needs only to be done once and does not need to be done at all on the Pi 4.\n\nMete Balci has written an excellent guide detailing [how to configure Raspberry Pi for network boot](https://metebalci.com/blog/bare-metal-rpi3-network-boot/) :clap:.\nThe official Raspberry Pi documentation for [debugging network book mode](https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#debugging-network-boot-mode) is also very useful.\n\n### Agent definition\n\nCreate one file for each of the agent nodes in the `./agents` directory. This file contains information about the agent's network settings, storage and `K3S` arguments specific to that node. The name of the file should be the hostname of the agent with an extension of `.agent`.\n\nFor each agent, you will need to provide:\n\n- the agent node's hardware address\n- its IPv4 address\n- a UUID for a partition containing an ext4 filesystem for `K3S` persistent local storage\n- an optional UUID for a swap partition\n- its PXE client ID\n- any extra args for K3s that the agent might need (optional)\n\nFor example:\n\n`./agents/willow.agent`\n\n```\nAGENT_ETHERNET=b8:27:eb:81:1a:52\nAGENT_IP=192.168.64.65\nAGENT_RANCHER_PART_UUID=d5f9e6c2-493c-48da-baf2-0c63dd7a36b1\nAGENT_PXE_ID=c6811a52\nAGENT_K3S_ARGS=\n```\n\n- An incomplete or incorrect agent description will not result in a working agent node.\n\n#### Finding the PXE ID\n\nThe Pi transmits its serial number for PXE boot. Find the serial number of any Raspberry Pi with the following command:\n\n```\n$ cat /sys/firmware/devicetree/base/serial-number\n```\n\n#### Local Storage\n\nAlthough the agents' operating systems are served over the network, `K3S` requires some storage on each agent's local filesystem.\n\nScooby agents expect an ext4 partition on any local storage media, although USB is highly recommended due to the relative volatility of many SD cards.\n\nYou can find the UUID of any partition using `blkid`. To find the partition UUID of /dev/sda1, for example:\n\n```\n$ blkid --match-tag=\"PARTUUID\" /dev/sda1\n```\n\n- Ensure you don't have any bootable physical media in the Pi when you power it up\n- If you have several agents, it can be helpful to write the name of the intended agent on the USB key to ensure the correct key is matched with the correct agent host.\n\n## Building\n\n### Building with Github Actions\n\nBy default, a commit to main will trigger the build job. When the build has completed you will see a zipped asset available to download in the `build` workflow.\n\nYou can enable builds on branch commits by removing the following line\n\n`./github/workflows/branch.yml`\n\n```\n    if: false\n```\n\n### Build locally\n\nTo build Scooby locally, you must first install Vagrant. On Debian-like systems you can do this with the following command:\n\n```\napt install vagrant\n```\n\nThen, execute the following command in the project root:\n\n```\n$ ./scripts/build.sh\n```\n\nThe output disk image will be found in `./images/scooby.img`. The image is typically in the region of 3.5GB.\n\n### Defaults\n\nBuilding with the defaults will produce a deployable image that:\n\n- has only a master node\n- contains no manifests\n- cannot be accessed by SSH\n\n### Secret environment variables\n\nScooby has the following secrets:\n\n`LC_DEFAULT_USER` name of your default user for the cluster (default: spike)  \n`LC_SSH_AUTH_KEY` an SSH auth key to access the cluster nodes (optional)  \n`LC_PACKER_GITHUB_API_TOKEN` your Packer API token (optional)\n\n- If you are building locally, you will need to set the secret environment variables in a file called `.env` in the project root.\n- If you are building using the GHA pipeline, you will need to set these as action secrets in your repository settings.\n\n### AWS S3 secrets\n\nIf you commit to main, Scooby can also attempt to upload the finished image to an AWS S3 bucket.\n\nTo enable S3 upload, create these secrets in your repository settings.\n\n`AWS_ACCESS_KEY_ID` the Key ID for your AWS account  \n`AWS_SECRET_ACCESS_KEY` the Secret Access Key for your AWS account  \n`AWS_BUCKET_S3_URI` the S3 URI for your bucket account  \n`AWS_REGION` the AWS region in which your bucket is located\n\n### Action Runners\n\nFor convenience, Scooby is configured to use macos-latest by default. This is the only GitHub runner that currently provides Vagrant.\n\nA self-hosted runner is recommended if you plan to make frequent changes.\nIf you wish to use a self-hosted runner, you will need to edit `.github/workflows/build.yml` and change `runs-on: macos-latest` to `runs-on: self-hosted`.\n\nYou will then need to follow [these instructions](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners) to host your own runner.\n\n## Deploying\n\n### Clone image\n\nOnce you have built and downloaded the image, use `dd` to transfer it to the master node's storage device. If your master node's storage device appeared as `/dev/sdf`, for example:\n\n```\n$ dd oflag=sync bs=1M if=~/Downloads/scooby.img of=/dev/sdf\n```\n\n### Connect the Pis to the network\n\nA typical Scooby network might look something like this:\n\n```mermaid\ngraph TD;\n  PublicNetwork--\u003eBuffy;\n  Buffy--\u003eXander;\n  Buffy--\u003eWillow;\n  Buffy--\u003eCordy;\n  Buffy--\u003eGiles;\n```\n\nAs explained earlier, the master node acts as a gateway for all network traffic originating inside the cluster that is destined for `external` networks. NAT is enabled for the agents using `iptables-persistent`.\n\nScooby is configured such that master node's `eth0` (the onboard Ethernet device) is the cluster's `external` or public interface, and `eth1` (e.g. the RTL-8152) is the `internal`, or cluster interface. Agent node `eth0` devices should exist on the same network as the master node's `eth1` device.\n\nClients on the external network may access services on the cluster via the master node's `eth0` device. Traefik cannot provide IP addresses by itself so it is recommended to provide them on the master node's `eth0` device with a MetalLB deployment on the master node.\n\nTo enable network boot services for agent nodes, the master node uses `dnsmasq` to provide PXE and DHCP services to the cluster on its `eth1` device. You should therefore not connect the master node's `eth1` device to any network with existing DHCP services.\n\nScooby uses Ethernet only. Wifi is not currently supported.\n\n## Power up the cluster\n\nOnce all your Pis are plugged into the network and all your storage devices are attached, it is time to power up the master node.\n\nA master node called `buffy` would be ready when you see the following in the master node console:\n\n```\nBUFFY WILL PATROL TONIGHT\n```\n\nBe patient; this process can about take 7 minutes on a Pi 4B. Allow the master node a few more minutes following the above message before powering up any agents. `cloud-init` config files are served to the agents by `lighttpd` and it takes a few moments to start responding.\n\nYou can watch the progress of your agent nodes coming online with the following command in a terminal session on the master node:\n\n```\n$ kubectl get nodes -Aw\n```\n\nEventually you should be rewarded with something like:\n\n```\nNAME     STATUS   ROLES                  AGE   VERSION\nbuffy    Ready    control-plane,master   25h   v1.27.4+k3s1\nwillow   Ready    \u003cnone\u003e                 25h   v1.27.4+k3s1\nxander   Ready    \u003cnone\u003e                 24h   v1.27.4+k3s1\ncordy    Ready    \u003cnone\u003e                 25h   v1.27.4+k3s1\n```\n\nCongratulations! Your Raspberry Pi supercomputer is now complete :blush:\n\n## Notes\n\n### First boot\n\nThe master node will appear to go quiet on first boot.\n\nThere will be no console output after `cloud-init` has identified the network devices for about four minutes. This is a known issue, and is caused by `cloud-init` waiting for eth0 to become ready.\n\nCloud-init will eventually find the network interface and continue.\n\n### Pi user\n\nThe default Raspberry Pi user `pi` is disabled for security reasons.\n\n### RTL-8152\n\nRTL-8152 USB to Ethernet devices are a good solution for adding extra connectivity to the Raspberry Pi but they can appear to the operating system as USB storage media when first inserted.\n\nTo ensure the device appears as an Ethernet device and not USB storage, add the following file to your project:\n\n`/server/etc/udev/rules.d/10-modeswitch.rules`\n\n```\nACTION==\"add\", SUBSYSTEM==\"usb\", ENV{ID_VENDOR_ID}==\"0bda\", ENV{ID_MODEL_ID}==\"8152\", RUN+=\"/usr/bin/usb_modeswitch -K -v 0bda -p 8151;/usr/bin/usb_modeswitch -v 0bda -p 8152 -R\"\n```\n\n### MetalLB\n\nUse MetalLB to provide IP addresses for all your K3S services. An example MetalLB manifest in a Scooby deployment might look like the following:\n\n`/server/var/lib/rancher/k3s/server/manifests/metallb.yml`\n\n```\napiVersion: v1\nkind: Namespace\nmetadata:\n  name: metallb-system\n---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n  name: metallb-system\n  namespace: kube-system\nspec:\n  repo: https://charts.bitnami.com/bitnami\n  chart: metallb\n  targetNamespace: metallb-system\n---\napiVersion: metallb.io/v1beta1\nkind: IPAddressPool\nmetadata:\n  name: first-pool\n  namespace: metallb-system\nspec:\n  addresses:\n    - 192.168.1.65-192.168.1.127\n---\napiVersion: metallb.io/v1beta1\nkind: L2Advertisement\nmetadata:\n  name: public-pool\n  namespace: metallb-system\nspec:\n  interfaces: [\"eth0\"]\n  ipAddressPools:\n    - first-pool\n  nodeSelectors:\n    - matchLabels:\n        kubernetes.io/hostname: buffy\n```\n\n### Pihole\n\nYou can easily deploy Pihole to a Scooby cluster with a manifest like this:\n\n`/server/var/lib/rancher/k3s/server/manifests/pihole.yml`\n\n```\napiVersion: v1\nkind: Namespace\nmetadata:\n  name: pihole\n  labels:\n    name: traffic\n---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n  name: pihole\n  namespace: pihole\nspec:\n  chart: pihole\n  targetNamespace: pihole\n  repo: https://mojo2600.github.io/pihole-kubernetes/\n  valuesContent: |-\n    ...\n```\n\nPihole works well for DNS, but load balancers aren't designed to respond to broadcast traffic like DHCP requests.\n\n### DHCP\n\nYou might find that Scooby's dnsmasq instance is an easier solution for serving DHCP to your `external` network than Pihole. Here is an example configuration file that will do that:\n\n`/server/etc/dnsmasq.d/10-external-dhcp.conf`\n\n```\ndomain=magicbox,192.168.1.0/24\ninterface=eth0\n\ndhcp-range=tag:eth0,192.168.1.128,192.168.1.240\ndhcp-option=tag:eth0,option:router,192.168.1.1\ndhcp-option=tag:eth0,6,192.168.1.66\n\ndhcp-host=30:05:5C:8E:ED:C3,192.168.1.10,snyder\ndhcp-host=08:00:20:AA:BB:CC,192.168.1.11,joycey\n```\n\n### OverlayFS\n\nScooby uses OverlayFS for agent filesystems. Under some circumstances you may wish to return an agent to its pristine state and re-deploy it without re-deploying the entire cluster.\n\nTo do this, you will need to SSH into the master node and execute the following as root, changing the hostname to match the agent node in question.\n\n```\n$ exportfs -u faith:/mnt/scooby/agents/faith\n$ umount /mnt/scooby/agents/faith\n$ rm -rf /var/lib/scooby/agents/faith/*\n$ rm -rf /var/lib/scooby/overlay/faith/*\n$ mount -a\n$ exportfs -a\n```\n\n### Accessing devices\n\nRaspberry Pis are often used for IoT applications. Scooby provides a very convenient method of remotely managing IoT devices when used with [Smarter Device Manager](https://gitlab.com/arm-research/smarter/smarter-device-manager) published by [Arm Research](https://www.arm.com/resources/research).\n\nTo access devices such as USB, GPIO pins, V4L and I2C and more from within `K3S`, download the following files from the Smarter Device Manager repo to `./server/var/lib/rancher/k3s/server/manifests/`\n\n- [Raspberry Pi K3S config map](https://gitlab.com/arm-research/smarter/smarter-device-manager/-/raw/master/smarter-device-manager-configmap-rpi.yaml?ref_type=heads\u0026inline=false)\n- [Raspberry Pi K3S daemon set](https://gitlab.com/arm-research/smarter/smarter-device-manager/-/raw/master/smarter-device-manager-ds-with-configmap-rpi-k3s.yaml?ref_type=heads\u0026inline=false)\n\nProvide each agent hosting a device with a `K3S` node label of `smarter-device-manager=enabled`. An agent description file for a node with device access enabled might look like the following:\n\n`./agents/ted`\n\n```\nAGENT_ETHERNET=b8:27:eb:86:53:3c\nAGENT_IP=192.168.64.254\nAGENT_RANCHER_PART_UUID=b357bda8-b641-41eb-9b5d-b3c5486e50e7\nAGENT_PXE_ID=9686533c\nAGENT_K3S_ARGS=\"--node-label 'smarter-device-manager=enabled'\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frayui%2Fscooby","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frayui%2Fscooby","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frayui%2Fscooby/lists"}