{"id":13569961,"url":"https://github.com/rcarmo/raspi-cluster","last_synced_at":"2025-04-13T06:37:27.742Z","repository":{"id":14579415,"uuid":"17295759","full_name":"rcarmo/raspi-cluster","owner":"rcarmo","description":"Notes and scripts for setting up (yet another) Raspberry Pi computing cluster","archived":false,"fork":false,"pushed_at":"2021-05-15T22:04:22.000Z","size":12699,"stargazers_count":256,"open_issues_count":2,"forks_count":23,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-03-26T23:06:31.894Z","etag":null,"topics":["cluster","k3s","kubernetes","raspberry-pi","swarm"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rcarmo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":["rcarmo"]}},"created_at":"2014-02-28T18:38:32.000Z","updated_at":"2025-02-04T19:44:05.000Z","dependencies_parsed_at":"2022-07-16T18:30:33.088Z","dependency_job_id":null,"html_url":"https://github.com/rcarmo/raspi-cluster","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcarmo%2Fraspi-cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcarmo%2Fraspi-cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcarmo%2Fraspi-cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rcarmo%2Fraspi-cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rcarmo","download_url":"https://codeload.github.com/rcarmo/raspi-cluster/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248675351,"owners_count":21143763,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cluster","k3s","kubernetes","raspberry-pi","swarm"],"created_at":"2024-08-01T14:00:46.448Z","updated_at":"2025-04-13T06:37:27.718Z","avatar_url":"https://github.com/rcarmo.png","language":"C","funding_links":["https://github.com/sponsors/rcarmo"],"categories":["C"],"sub_categories":[],"readme":"# raspi-cluster (now with Kubernetes thanks to k3s!)\n\n![Pi 2](https://raw.github.com/rcarmo/raspi-cluster/master/pics/pi2.jpg)\n\n## What?\n\nA while ago I decided to build a small cluster of [Raspberry Pi][rpi] boards. I've since upgraded to Pi 2 boards, and this repository is used for versioning design notes, configuration files and sundry.\n\n## Why?\n\nI wanted something challenging to do in terms of distributed processing, and lacked dedicated hardware to do it. There's a lot to be learned even from simple, unsophisticated solutions, and virtual machines can only do so much.\n\n## How?\n\nThe cluster consists of five nodes: a master and four slaves. The master acts as a gateway, DHCP and NFS server and the slave nodes get their IP address and `/srv/jobs` directory from it.\n\nAll slave nodes are identical -- _completely_ identical, except for hostname and MAC address, and there is no need to log in and configure things manually for each node.\n\nHere's a few more shots of the original version, with the 5-port PSU and the old Model B boards:\n\n![Cabled](https://raw.github.com/rcarmo/raspi-cluster/master/pics/ethernet.jpg)\n![Power cords](https://raw.github.com/rcarmo/raspi-cluster/master/pics/micro_usb.jpg)\n![First assembly](https://raw.github.com/rcarmo/raspi-cluster/master/pics/first_assembly.jpg)\n\nIn retrospect I probably ought to have gone for longer USB cables and moved all of the cabling to the USB side (since it leaves the SD card slot clear), but I also need to be able to see the activity lights, and the Pi isn't exactly designed for easy stacking.\n\nA larger cluster is certainly feasible, but 5 boards is as much as I can power with the PSU I have.\n\n## Hardware\n\nThis is a partial list of the stuff I'm using (Amazon UK affiliate links):\n\n* 5x Raspberry Pi 2 Model B, which replaced the [Raspberry Pi Model B](http://www.amazon.co.uk/Raspberry-Pi-Model-512MB-RAM/dp/B008PT4GGC/ref=as_li_tf_tl?ie=UTF8\u0026tag=thtaofma-21\u0026linkCode=as2\u0026camp=1634\u0026creative=6738) boards (duh!)\n* [7x Class 10 Micro SD Cards](http://www.amazon.co.uk/gp/product/B0036V5DGG/ref=as_li_tf_tl?ie=UTF8\u0026tag=thtaofma-21\u0026linkCode=as2\u0026camp=1634\u0026creative=6738) (2 master cards, 5 for production), which replaced the [Class 10 SD Cards](http://www.amazon.co.uk/gp/product/B003VNKNEG/ref=as_li_tf_tl?ie=UTF8\u0026tag=thtaofma-21\u0026linkCode=as2\u0026camp=1634\u0026creative=6738)\n* [1x TP-Link 5-port Ethernet switch](http://www.amazon.co.uk/TP-Link-TL-SF1005D-100Mbps-Unmanaged-Desktop/dp/B000FNFSPY/ref=as_li_tf_tl?ie=UTF8\u0026tag=thtaofma-21\u0026linkCode=as2\u0026camp=1634\u0026creative=6738) and some ancient cables I had lying around (need to build new stripped down ones)\n* [5x 6 inch micro USB cables](http://www.amazon.co.uk/gp/product/B003YKX6WM/ref=as_li_tf_tl?ie=UTF8\u0026tag=thtaofma-21\u0026linkCode=as2\u0026camp=1634\u0026creative=6738)\n* [1x 5 port USB PSU](http://www.amazon.co.uk/gp/product/B00EKDVGKQ/ref=as_li_tf_tl?ie=UTF8\u0026tag=thtaofma-21\u0026linkCode=as2\u0026camp=1634\u0026creative=6738)\n* 1x ancient Bondi Blue iMac USB keyboard.\n* 1x Custom-printed rack case (see SCAD files in the `3d_models` folder)\n\n## Software\n\nAs a base OS, this cluster spent most of its useful life using [the Ubuntu 16.04 official image for the Pi 2][ub], which worked much better than Raspbian for running Swarm, Docker or Spark (nevertheless, the configuration files in this repo should work in both systems).\n\nUp to the point where I had unfettered access to public cloud resources as part of my work, the cluster ran a mix of Docker Swarm and the occasional [Clojure][clj] program using [Hazelcast][hz] atop JDK 1.8, as well as [Jupyter][jy], which runs very nicely indeed and provided me with an agnostic, notebook-oriented front-end.\n\nEarlier (back in the Rev 1 days) I had set up [Disco][dp] on it and intended to fiddle with MPI, but now I have plenty more ways to parallelize things.\n\nIt's a bit ironic to do some kinds of processing on merely 5GB of aggregated RAM, but my interest was in the algorithms themselves and I never planned on doing something silly like tackling the next Netflix Prize with this -- besides, running things on low-end hardware is often the only way to do proper optimization, and I like the extra challenges.\n\n### List of packages involved so far:\n\n* [k3s][k3s], which is the latest addition and has rendered pretty much everything else below obsolete. See the `k3s` folder for a simple `Makefile` to install it and keep it running.\n* [etcd][etcd], which I was using to store (and distribute) configurations across nodes\n* [Docker][do], which ships with Ubuntu 14.04 and makes it a lot easier to build and tear down environments. Currently trying to get 1.7 to build so I can use `swarm` and other niceties.\n* [OpenVSwitch][ovs], which I'm using for playing around with network topologies\n* [Jupyter][jy], which provides me with a nice web front-end and basic Python parallel computing.\n* [Spark][spark], which has mostly replaced [Disco][dp] for map/reduce jobs.\n* [Dash](https://github.com/rcarmo/dash), a real-time status dashboard (rewritten in [Go][golang], available under the `dashboard` folder here, and still being worked on)\n* A custom daemon that sends out a JSON-formatted multicast packet with system load, CPU usage and RAM statistics (written in raw C, available in `tools`)\n* [ElasticSearch](http://www.elasticsearch.org), which I'm using for storing metrics.\n* Oracle [JDK 8](https://jdk8.java.net/download.html)\n* [leiningen][lein] (which fetches [Hazelcast][hz] and other dependencies for me, via [this library][clj-hz])\n* [Nightcode][nc] as a development environment ([LightTable][lt] doesn't run on ARM, and a lot of my hobby coding these days is actually done on an [ODROID][u2])\n* `distcc` for building binaries slightly faster\n* `dnsmasq` for DHCP and DNS service\n\nThis repository also ships with a very simple monitoring dashboard that looks like this:\n\n![Updated dashboard](https://raw.github.com/rcarmo/raspi-cluster/master/pics/dash.jpg)\n\n## But isn't the [Raspberry Pi][rpi] slow?\n\nWell spotted, young person. It was, and the Pi 2, despite being a marked improvement, isn't exactly a supercomputer. But it's also cheap, and beggars can't be choosers.\n\nNevertheless, the current configuration provides me with 20 ARMv7 cores clocked at 1GHz, and that's nothing to sneeze at.\n\nBut I'm open to [sponsoring][d] so that I can upgrade this to have at least twice as many boards, or move to a set of beefier ARM64 boards. The [NVIDIA Jetson](https://developer.nvidia.com/buy-jetson), in particular, would be great, but five of them aren't easy to come by.\n\n[etcd]: https://github.com/coreos/etcd\n[hz]: http://www.hazelcast.org\n[rpi]: http://www.raspberrypi.org\n[d]: http://the.taoofmac.com/space/site/Donate\n[u2]: http://hardkernel.com/main/products/prdt_info.php?g_code=G135341370451\n[u3]: http://hardkernel.com/main/products/prdt_info.php?g_code=G138733896281\n[clj]: http://www.clojure.org\n[hz]: http://www.hazlecast.org\n[nc]: https://github.com/oakes/Nightcode\n[lt]: http://lighttable.com\n[moebius]: http://moebiuslinux.sourceforge.net\n[lein]: http://leiningen.org\n[clj-hz]: https://github.com/rcarmo/clj-hazelcast\n[dp]: http://discoproject.org\n[golang]: http://www.golang.org\n[spark]: http://spark.apache.org\n[jy]: http://jupyter.org\n[ub]: https://wiki.ubuntu.com/ARM/RaspberryPi\n[do]: http://docker.io\n[ovs]: http://openvswitch.org/\n[k3s]: http://k3s.io","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frcarmo%2Fraspi-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frcarmo%2Fraspi-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frcarmo%2Fraspi-cluster/lists"}