https://github.com/faust64/pi8s
π8s
https://github.com/faust64/pi8s
Last synced: 4 months ago
JSON representation
π8s
- Host: GitHub
- URL: https://github.com/faust64/pi8s
- Owner: faust64
- Created: 2018-07-23T18:57:04.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2021-07-27T06:06:28.000Z (almost 5 years ago)
- Last Synced: 2025-06-24T15:56:08.949Z (12 months ago)
- Language: Jinja
- Size: 54.7 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# k8s on Raspbian
Set of Ansible roles deploying k8s on RPI.
Last tested with raspbian buster 10.3, k8s 1.17.3, on armv7l/v8l.
NOTICE: better look into https://github.com/kubernetes-sigs/kubespray,
if you want some production-ready cluster. RPI is perfectly supported.
The playbook given here is mainly a kubernetes-from-scratch exercise.
## Configuration
Set proper hostnames, enable SSH, install python-apt.
Update `hosts`, `group_vars` & `host_vars`.
## Deployment
Prepare cluster variables, in `group_vars/all.yml`, then deploy cluster:
```
$ ansible-playbook -i hosts ./k8s.yml
```
## Notes
### Multi-Master
Note multi-master deployment is not operational - this could be done with
the following manual steps:
- edit k8s role, do not load join tasks, stop after initializing first
master.
- apply `rpi`/`containers`/`k8s` roles to all masters
- once cluster is initialized on a first master, run the join command from
an other one
- connect etcd on the initial master, add new member (`etcdctl member add`)
- edit `/etc/kubernetes/manifests/etcd.yaml` on initial master, notice that
while our init script should have set a proper URL, the
`initial-cluster-members` param only mentions our initial master: fix it
to include all 3 members - beware, restarting etcd, going from 1 to 3
members: quorum would be broken until our second member joins in, k8s API
would blink, ... that shouldn't take long. though make sure you did run
the firt join command *before* reconfiguring etcd.
- once etcd is back up, we may join our last master, and add it to etcd as well
- generate new kube-apiserver certificates for additional masters
(see `roles/k8s/templates/post-init.j2` -- arguably, consider generating
a unique apiserver certificate whose SAN covers all masters)
- next, scale out `controllers`/`apiserver`/`scheduler` Pods to all masters (see
`roles/k8s/templates/post-init.j2`), consider fixing addresses in
`/etc/kubernetes/(admin|scheduler|controller-manager|kubelet).conf`. Not sure
what's best -- in my case, I have 3 racks of 4 RPIs: kubelets from rack N
would connect to whichever master is on the same rack, masters services in
general rely on locally hosted services. In most cases, you may just want to
use your LB address, some VIP, a DNS record resolving to all masters, ...
Either way, make your re-generate the API server certificate with proper SAN.
### Crio
Note that cri-o runtime does not seem to work with kubernetes, when running
with systemd cgroups driver. Turns out container would fail to start applying
CPU quotas. On raspbian, we should expect those sysctls [to be missing](https://raspberrypi.stackexchange.com/questions/87779/missing-cpu-cfs-period-us-cgroup-subsystem-in-raspbian-stretch-on-raspberry-pi-z).
Docker also complains about those, yet shows it as a warning and still starts
containers - whereas I could not start etcd, initializing my first master with
cri-o. A workaround may be to build your own kernel. Or prefer the cgroupfs
driver -- which, using Raspbian, would require enablig memory cgroups (in
/boot/cmdline.txt).
### Docker Pool
Only applies when using Docker runtime. And even then: not mandatory.
Before first booting Raspbian, having provisionned our flash media, we may
add a separate partition, such as our rootfs would have about 8G of
disk space, while the rest of our SD card can be used as a Volume Group:
```
# fdisk /dev/mmcblk0
p
d
2
n
p
2
[start-offset-as-seen-with-p-command]
+8G
n
p
3
[start-offset-as-seen-with-p-command]
[confirm-next-free-offset-is-fine]
[confirm-last-free-offset-is-fine]
w
```
Then, having booted Raspbian:
```
# apt-get update
# apt-get install lvm2
# pvcreate /dev/mmcblk0p3
# vgcreate data /dev/mmcblk0p3
# lvcreate -n docker -l95%VG data
# lvcreate -n dockermeta -l1%VG data
# lvconvert -y --zero n -c 512K \
--thinpool data/docker \
--poolmetadata data/dockermeda
```
On masters, consider creating a LV hosting etcd.
## Special Thanks
Special thanks to:
* Anne Fernandes
* Dimitris Finas
* Nicolas Fleury-Gobert