{"id":17087065,"url":"https://github.com/ncabatoff/piinit","last_synced_at":"2025-06-11T07:40:02.482Z","repository":{"id":72407516,"uuid":"169911478","full_name":"ncabatoff/piinit","owner":"ncabatoff","description":"Initialize raspberry pi consul/nomad cluster","archived":false,"fork":false,"pushed_at":"2020-01-23T01:28:04.000Z","size":103,"stargazers_count":29,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-26T15:54:07.541Z","etag":null,"topics":["consul","nomad","packer","prometheus","raspberry-pi"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ncabatoff.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-09T20:41:37.000Z","updated_at":"2024-11-28T08:10:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"c1842337-dcbe-4ac8-9353-361d6f3ce131","html_url":"https://github.com/ncabatoff/piinit","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncabatoff%2Fpiinit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncabatoff%2Fpiinit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncabatoff%2Fpiinit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncabatoff%2Fpiinit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ncabatoff","download_url":"https://codeload.github.com/ncabatoff/piinit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248637832,"owners_count":21137538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consul","nomad","packer","prometheus","raspberry-pi"],"created_at":"2024-10-14T13:30:34.611Z","updated_at":"2025-04-12T21:41:46.075Z","avatar_url":"https://github.com/ncabatoff.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# piinit: initialize raspberry pi cluster\n\nThis repo contains code to build a cluster of servers running HashiCorp Nomad \nand Consul, monitored using Prometheus.\n\nThe primary target is low-power ARM single-board computers like the Raspberry Pi.\nDocker is also supported for testing purposes.\n\nUnlike most other non-cloud-based approaches for creating Consul clusters, we use \na single \"immutable\" machine image shared by each of the nodes.  The OS\nisn't modified after burning the image to SD card, except for the data\ndirectories and logs written by the applications themselves.  This is in contrast \nto traditional provisioning/config mgmt solutions like Ansible or Chef.\nIn principle you could even abstain from installing SSH.\n\nThis immutable approach is nice for a variety of reasons, and a nice bonus is\nthat it saves time when initializing the Pi servers, since the image is built \nonly once and then burned to every card.\n\n# Test environment\n\nTo try out a virtual version of the cluster, install Virtualbox and Vagrant,\nthen run\n\n```bash\nvagrant up\n```\n\nThis will create a VM containing all the dependencies, then build and run\ndocker images representing the core nomad/consul and prometheus servers.  It\nwill also setup nomad in client mode on the virtual machine and configure DNS\nresolution to send queries for the .consul domain to the virtual cluster.\n\nTo test that everything is running, go to:\n\n- [prometheus ui](http://localhost:49090/targets)\n- [consul ui](http://localhost:48500/ui)\n- [nomad ui](http://localhost:44646/ui)\n\nNote that although the cluster will be restarted when the VM is rebooted, its\nstate will be wiped.  This is by design, though it's easy enough to add volume\nmappings to docker-launch.sh (and/or /etc/rc.local) if you'd rather \nthe state be persisted.\n\n# Real environment\n\nSummary of steps:\n0. Obtain ARM hardware\n1. Create DNS entries for your RPi MAC addrs on your router\n2. Build packages if you didn't run `vagrant up` above\n3. Setup Packer ARM env\n4. Build ARM OS images\n5. Burn OS images to SD card\n\n## Hardware\n\nBuy 3 ARM based systems, e.g. Raspberry Pi 3B+.  Buy 3 decent MicroSD cards, e.g.\nSandisk Extreme Pro, size is up to you.  You can use slower cards, but I would\nskimp on size before speed.  Note that the Pi probably won't be able to use the\ncard at full speed, but it's still worth it for faster burn times.\n\nIf you don't already have a USB3 Micro SD card reader that supports higher-speed\ncard standards like UHS, get one, again to minimize burn time.\n\n## DNS and assigning hostnames\n\n*Note: Consul handles DNS queries to perform DNS-based service discovery, but \nthat's not what this section is about.*\n\nIn order to use a single OS image across all your cluster's servers, they need\nto use DHCP to get their hostnames.  This means for each server, boot it up,\nget its MAC address, and put that into your DHCP server (typically your router)\nas a static DHCP entry with a fixed IP.  On subsequent boots it will set its\nhostname based on that DHCP entry.\n\n## Build packages\n\n```bash\nmake packages\ncd packages \u0026\u0026 ../pkgbuilder -arches all -config '{\n  \"ConsulSerfEncrypt\": \"S/OHRE9Nc4VmdGtJr11vBA==\", \n  \"CoreServers\": [\"192.168.2.51\", \"192.168.2.52\", \"192.168.2.53\"],\n  \"WifiSsid\": \"YourSSID\",\n  \"WifiPsk\": \"b71288ba03c9197d6afda9f1f67f913c12f41fb9e3585da18c11e68099355e62\"\n}'\n```\n\nReplace ConsulSerfEncrypt and CoreServers with your local values.\n\nWifiSsid and WifiPsk are only needed if any of your systems need WiFi, in which \ncase you should include the package wifi-local in their packer config.  Use the\nwpa-supplicant command line tool to translate your WiFi password into PSK.\n\n### Setup Packer ARM env\n\nTo create the VM used for building your Pi OS image:\n\n```bash\nmkdir -p $GOPATH/src/github.com/ncabatoff\ncd $GOPATH/src/github.com/ncabatoff\ngit clone https://github.com/ncabatoff/packer-builder-arm-image\ncd packer-builder-arm-image\npath/to/piinit/checkout/arm-setup.sh\n```\n\n_Note: You may prefer to use the upstream repo I forked, https://github.com/solo-io/packer-builder-arm-image.\nBe aware however that they seem to be using a bot that automatically merges PRs that merge cleanly, without \ndoing any review.  This makes me uncomfortable enough that I'm not recommending it._\n\n### Build ARM OS images\n\nTo create the OS image for your servers, run:\n\n```bash\npath/to/piinit/checkout/arm-run.sh\n```\n\n#### Customization\n\nBy default arm-run.sh will build an image based on packer-arm.json, which will serve\nfor your core Consul/Nomad servers.  \n\nGiven a single argument arm-run.sh will expect a .json file to give packer.  \nUse packer-arm-mon.json to build a Prometheus server image that can be used on \na Raspberry Pi Zero to monitor your cluster.  It includes a Consul client to \ndiscover all your services.\n\nGiven two arguments arm-run.sh will expect the first to be a .json file to \noverwrite, and the second to be a .jsonnet file used to build the former.\nIt will then give the .json file to packer to build an OS image with.\n\nYou need not use Jsonnet to build your own packer json files, but that's what I\nuse.\n\n### Burning OS images to SD cards\n\nYou should use [Etcher](https://www.balena.io/etcher/) to write the image to\nSD cards.  See `arm-burn.sh` for an example of how to invoke it non-interactively\nfrom the command line.  Make sure to customize it according to your local setup.\n\n## Notes\n\n### How it works\n\n- [Packer](https://packer.io/) is used to create the OS images using\n  - [packer-builder-arm-image](https://github.com/solo-io/packer-builder-arm-image) for ARM Pi images\n  - built-in [Docker builder](https://www.packer.io/docs/builders/docker.html) for AMD64 Docker images\n- cmd/pkgbuilder creates custom .deb files using\n  [go-getter](https://github.com/hashicorp/go-getter) and [nfpm](https://github.com/goreleaser/nfpm), from releases of\n  - [Nomad](https://nomadproject.io)\n  - [Consul](https://consul.io)\n  - [Prometheus](https://prometheus.io)\n  - [node_exporter](https://github.com/prometheus/node_exporter)\n\n### Motivation\n\nMany people have home servers, e.g. to serve media files.  But what about\nredundancy?  You could setup a second server, but then how do you manage failover?\nWhat if you want to run a cronjob regularly, and you don't care where it runs,\nbut you want to ensure it happens even if a server has gone down - and you'd \nprefer it didn't run on more than one server?\n\nRather than reinvent the wheel, it makes sense to use dedicated tools to do this.\n\nConsul gives us reliable service discovery and a distributed KV.  Now you don't\nhave to refer to specific hosts, you can use DNS names that refer to services \nrunning whether the cluster has decided they should run.\n\nNomad gives us a way to run programs - servers and batch jobs - without having \nto specify where.\n\nBoth of these systems require three or more servers to provide redundancy and\navoid split brain inconsistency.  For a couple of hundred dollars you can buy\n3 Raspberry Pis (or similar ARM-based single board computers) to run Nomad\nand Consul.\n\n### Dedicated cluster\n\nYou could certainly run other software on these servers as well.  For the volume\nof data you're likely to have on your home setup, Consul and Nomad need very few\nresources.  Personally I use a couple of NUCs to run everything else, because I'd \nrather have the core Consul/Nomad cluster as reliable and predictable as possible.\n\n### TLS\n\nEventually I plan to add TLS (and probably Vault) to the mix.  For now the goal\nis to get the simplest thing possible running smoothly.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fncabatoff%2Fpiinit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fncabatoff%2Fpiinit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fncabatoff%2Fpiinit/lists"}