https://github.com/squat/onseu2019
ONS EU 2019 talk on connecting Kubernetes clusters with Kilo
https://github.com/squat/onseu2019
Last synced: about 1 year ago
JSON representation
ONS EU 2019 talk on connecting Kubernetes clusters with Kilo
- Host: GitHub
- URL: https://github.com/squat/onseu2019
- Owner: squat
- License: mit
- Created: 2019-09-23T21:45:09.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2019-09-23T21:46:23.000Z (almost 7 years ago)
- Last Synced: 2025-02-08T16:44:33.958Z (over 1 year ago)
- Language: HCL
- Size: 1.41 MB
- Stars: 6
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Open Networking Summit EU 2019
This repository contains the demo code for my Open Networking Summit EU 2019 talk about Kubernetes clusters across clouds using Kilo.
In this demo we will imagine we are a company like Nest that is running object detection processes on video captured by IoT devices.
We will run a web-app in the cloud connected to a GPU-powered image detection and labeling service in a different public cloud provider.
The web-app will stream video from the IoT device over a WireGuard connection to keep the data safe.
Specifically we will:
* create a multi-cloud cluster that spans between DigitalOcean and AWS
* create some GPU workers in AWS
* run the workload that captures video in a device on the edge, e.g. your host capturing video from the webcam, or an edge device running Kubernetes
* peer the workload with the cluster in the cloud
* run a computer vision process on the video captured by the edge workload
* accelerate the computer vision using GPUs in AWS.
## Prerequisites
You will need:
* DigitalOcean and AWS accounts
* Terraform installed
* the Kilo commandline utility `kgctl` installed
* WireGuard installed
* *optional*: an edge device running Kubernetes with WireGuard installed, e.g. k3s on a Raspberry Pi
## Getting Started
Modify the provided `terraform.tfvars` file to suit your project:
```sh
$EDITOR terraform.tfvars
```
## Running
### Exercise 0: create the infrastructure
1. Create the infrastructure using Terraform:
```shell
terraform init
terraform apply --auto-approve
```
2. Annotate the GPU nodes so Kilo knows they are in their own data center:
```shell
for node in $(kubectl get nodes | grep -i ip- | awk '{print $1}'); do kubectl annotate node $node kilo.squat.ai/location="aws"; done
```
3. Install the Kilo manifests:
We'll use a DaemonSet to install the WireGuard and NVIDIA kernel modules on the nodes and install Kilo.
```shell
kubectl apply -f manifests/0/
```
4. Create the local WireGuard link:
```shell
IFACE=wg0
sudo ip link add $IFACE type wireguard
sudo ip a add 10.5.0.4 dev $IFACE
sudo ip link set up dev $IFACE
```
5. Generate a key-pair for the WireGuard link:
```shell
wg genkey | tee privatekey | wg pubkey > publickey
```
6. Create a Kilo Peer on the cluster for the local WireGuard link:
```shell
PEER=squat
cat < peer.ini
sudo wg setconf $IFACE peer.ini
sudo wg set $IFACE private-key privatekey
sudo ip route add 10.4.0.0/16 dev wg0
```
8. Verify we can connect to the cluster's WireGuard endpoint:
```shell
ping 10.4.0.1
```
9. Add routes to the cluster's allowed IPs:
```shell
for ip in $(kgctl showconf peer $PEER | grep AllowedIPs | cut -f 3- -d ' ' | tr -d ','); do
sudo ip route add $ip dev $IFACE
done
```
10. Inspect the cluster's topology:
We'll use the Kilo command line tool to generate a GraphViz graph showing the cluster's topology.
```shell
kgctl graph | circo -Tsvg > cluster.svg && $BROWSER cluster.svg
```
### Exercise 1: connect the IoT camera to the cluster
1. Run the video capture service on the "edge":
Let's run a service producing an MJPEG stream on the local host to simulate an IoT webcam.
```shell
docker run --rm --privileged -p 8080:8080 squat/kubeconeu2019:amd64-latest /mjpeg
```
2. Create a Service to expose the webcam to the cluster:
The object detection application needs to be able to stream video from the webcam.
In order to accomplish this we'll create a Service and Endpoints resource that points to the webcam container running locally.
```shell
cat < cluster.svg && $BROWSER cluster.svg
```
6. Mirror the Raspberry Pi's Service in the cloud cluster:
The object detection application needs to be able to stream video from the webcam.
In order to accomplish this we'll create a Service and Endpoints resource that points to the webcam container running in the k3s cluster.
```shell
cat <