{"id":17888276,"url":"https://github.com/squat/onseu2019","last_synced_at":"2025-04-03T02:42:55.488Z","repository":{"id":73957034,"uuid":"210458231","full_name":"squat/onseu2019","owner":"squat","description":"ONS EU 2019 talk on connecting Kubernetes clusters with Kilo","archived":false,"fork":false,"pushed_at":"2019-09-23T21:46:23.000Z","size":1482,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-08T16:44:33.958Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/squat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-23T21:45:09.000Z","updated_at":"2022-08-17T17:39:55.000Z","dependencies_parsed_at":"2023-03-13T20:17:28.289Z","dependency_job_id":null,"html_url":"https://github.com/squat/onseu2019","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squat%2Fonseu2019","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squat%2Fonseu2019/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squat%2Fonseu2019/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squat%2Fonseu2019/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/squat","download_url":"https://codeload.github.com/squat/onseu2019/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246927809,"owners_count":20856193,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-28T13:36:59.712Z","updated_at":"2025-04-03T02:42:55.461Z","avatar_url":"https://github.com/squat.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Open Networking Summit EU 2019\nThis repository contains the demo code for my Open Networking Summit EU 2019 talk about Kubernetes clusters across clouds using Kilo.\nIn this demo we will imagine we are a company like Nest that is running object detection processes on video captured by IoT devices.\nWe will run a web-app in the cloud connected to a GPU-powered image detection and labeling service in a different public cloud provider.\nThe web-app will stream video from the IoT device over a WireGuard connection to keep the data safe.\n\nSpecifically we will:\n* create a multi-cloud cluster that spans between DigitalOcean and AWS\n* create some GPU workers in AWS\n* run the workload that captures video in a device on the edge, e.g. your host capturing video from the webcam, or an edge device running Kubernetes\n* peer the workload with the cluster in the cloud\n* run a computer vision process on the video captured by the edge workload\n* accelerate the computer vision using GPUs in AWS.\n\n## Prerequisites\nYou will need:\n* DigitalOcean and AWS accounts\n* Terraform installed\n* the Kilo commandline utility `kgctl` installed\n* WireGuard installed\n* *optional*: an edge device running Kubernetes with WireGuard installed, e.g. k3s on a Raspberry Pi\n\n## Getting Started\n\nModify the provided `terraform.tfvars` file to suit your project:\n\n```sh\n$EDITOR terraform.tfvars\n```\n\n## Running\n\n### Exercise 0: create the infrastructure\n\n1. Create the infrastructure using Terraform:\n```shell\nterraform init\nterraform apply --auto-approve\n```\n\n2. Annotate the GPU nodes so Kilo knows they are in their own data center:\n```shell\nfor node in $(kubectl get nodes | grep -i ip- | awk '{print $1}'); do kubectl annotate node $node kilo.squat.ai/location=\"aws\"; done\n```\n\n3. Install the Kilo manifests:\nWe'll use a DaemonSet to install the WireGuard and NVIDIA kernel modules on the nodes and install Kilo.\n```shell\nkubectl apply -f manifests/0/\n```\n\n4. Create the local WireGuard link:\n```shell\nIFACE=wg0\nsudo ip link add $IFACE type wireguard\nsudo ip a add 10.5.0.4 dev $IFACE\nsudo ip link set up dev $IFACE\n```\n\n5. Generate a key-pair for the WireGuard link:\n```shell\nwg genkey | tee privatekey | wg pubkey \u003e publickey\n```\n\n6. Create a Kilo Peer on the cluster for the local WireGuard link:\n```shell\nPEER=squat\ncat \u003c\u003cEOF | kubectl apply -f -\napiVersion: kilo.squat.ai/v1alpha1\nkind: Peer\nmetadata:\n  name: $PEER\nspec:\n  allowedIPs:\n  - 10.5.0.4/32\n  publicKey: $(cat publickey)\n  persistentKeepalive: 10\nEOF\n```\n\n7. Configure the cluster as a peer of the local WireGuard link:\n```shell\nkgctl showconf peer $PEER \u003e peer.ini\nsudo wg setconf $IFACE peer.ini\nsudo wg set $IFACE private-key privatekey\nsudo ip route add 10.4.0.0/16 dev wg0\n```\n\n8. Verify we can connect to the cluster's WireGuard endpoint:\n```shell\nping 10.4.0.1\n```\n\n9. Add routes to the cluster's allowed IPs:\n```shell\nfor ip in $(kgctl showconf peer $PEER | grep AllowedIPs | cut -f 3- -d ' ' | tr -d ','); do\n\tsudo ip route add $ip dev $IFACE\ndone\n```\n\n10. Inspect the cluster's topology:\nWe'll use the Kilo command line tool to generate a GraphViz graph showing the cluster's topology.\n```shell\nkgctl graph | circo -Tsvg \u003e cluster.svg \u0026\u0026 $BROWSER cluster.svg\n```\n\n### Exercise 1: connect the IoT camera to the cluster\n\n1. Run the video capture service on the \"edge\":\nLet's run a service producing an MJPEG stream on the local host to simulate an IoT webcam.\n```shell\ndocker run --rm --privileged -p 8080:8080 squat/kubeconeu2019:amd64-latest /mjpeg\n```\n\n2. Create a Service to expose the webcam to the cluster:\nThe object detection application needs to be able to stream video from the webcam.\nIn order to accomplish this we'll create a Service and Endpoints resource that points to the webcam container running locally.\n```shell\ncat \u003c\u003cEOF | kubectl apply -f -\napiVersion: v1\nkind: Service\nmetadata:\n  name: mjpeg\nspec:\n  ports:\n    - port: 8080\n---\napiVersion: v1\nkind: Endpoints\nmetadata:\n    name: mjpeg\nsubsets:\n  - addresses:\n      - ip: 10.5.0.4\n    ports:\n      - port: 8080\nEOF\n```\n\n3. Run the object detection application:\n```shell\nkubectl apply -f manifests/1/\n```\n\n4. Check out the webapp in a browser!\nNote that we can use the container's Pod IP in the browser to access the workload; we are not port-forwarding!\n```shell\n$BROWSER $(kubectl get pods -o=jsonpath='{range .items[*]}{.metadata.name}{\"\\t\"}{.status.podIP}{\"\\n\"}{end}' | grep kceu | cut -f 2):8080\n```\n\n5. Clean up the manually created Service:\n```shell\nkubectl delete service mjpeg\nkubectl delete endpoints mjpeg\n```\n\n### Exercise 2: connect two Kubernetes clusters\nKilo can also be used as a general purpose multi-cluster tool.\nTo demonstrate this, let's connect another cluster to the one we have running in the cloud.\nIn this exercise, let's imagine that our company chooses to run all of our edge workloads as containers on single node Kubernetes clusters.\nSpecifically, we'll say that our company sells webcams connected to small servers running Kubernetes, e.g. Raspberry Pis running k3s.\nLet's call `KUBECONFIG1` the configuration for the cluster in the cloud and `KUBECONFIG2` the configuration for the edge cluster.\n\n1. Install Kilo on the Raspberry Pi running k3s:\n```shell\nkubectl --kubeconfig $KUBECONFIG2 apply -f manifests/2/kilo-k3s-flannel.yaml\n```\n\n2. Run the webcam container on the edge cluster:\n```shell\nkubectl --kubeconfig $KUBECONFIG2 apply -f manifests/2/mjpeg.yaml\n```\n\n3. *optional*: Mount the camera on a servo and run the servo webapp:\n```shell\nkubectl --kubeconfig $KUBECONFIG2 apply -f manifests/2/servor.yaml\n```\n\n![IoT Webcam](servor.jpg \"webcam mounted on a servo\")\n\n4. Register the two clusters with each other:\n```shell\n# The default Typhoon service CIDR.\nSERVICECIDR1=10.3.0.1/16\n# The WireGuard port used by the cluster.\nWGPORT1=51820\n# The default k3s service CIDR.\nSERVICECIDR2=10.43.0.0/16\n# The WireGuard port used by the edge node.\nWGPORT2=51821\n# Register the nodes in cluster1 as peers of cluster2.\nfor n in $(kubectl --kubeconfig $KUBECONFIG1 get no -o name | cut -d'/' -f2); do\n    # Specify the service CIDR as an extra IP range that should be routable.\n    kgctl --kubeconfig $KUBECONFIG1 --port $WGPORT1 showconf node $n --as-peer -o yaml --allowed-ips $SERVICECIDR1 | kubectl --kubeconfig $KUBECONFIG2 apply -f -\ndone\n# Register the nodes in cluster2 as peers of cluster1.\nfor n in $(kubectl --kubeconfig $KUBECONFIG2 get no -o name | cut -d'/' -f2); do\n    # Specify the service CIDR as an extra IP range that should be routable.\n    kgctl --kubeconfig $KUBECONFIG2 --port $WGPORT2 showconf node $n --as-peer -o yaml --allowed-ips $SERVICECIDR2 | kubectl --kubeconfig $KUBECONFIG1 apply -f -\ndone\n```\n\n5. Inspect the cluster's topology:\nWe'll use the Kilo command line tool to generate a GraphViz graph showing the cluster's topology.\n```shell\nkgctl --kubeconfig $KUBECONFIG1 graph | circo -Tsvg \u003e cluster.svg \u0026\u0026 $BROWSER cluster.svg\n```\n\n6. Mirror the Raspberry Pi's Service in the cloud cluster:\nThe object detection application needs to be able to stream video from the webcam.\nIn order to accomplish this we'll create a Service and Endpoints resource that points to the webcam container running in the k3s cluster.\n```shell\ncat \u003c\u003cEOF | kubectl apply -f -\napiVersion: v1\nkind: Service\nmetadata:\n  name: mjpeg\nspec:\n  ports:\n    - port: 8080\n---\napiVersion: v1\nkind: Endpoints\nmetadata:\n    name: mjpeg\nsubsets:\n  - addresses:\n      - ip: $(kubectl --kubeconfig $KUBECONFIG2 get service mjpeg -o jsonpath='{.spec.clusterIP}') # The cluster IP of the mjep container on the Raspberry Pi.\n    ports:\n      - port: 8080\nEOF\n```\n\n7. Check out the webapp in a browser!\n```shell\n$BROWSER $(kubectl --kubeconfig $KUBECONFIG1 get pods -o=jsonpath='{range .items[*]}{.metadata.name}{\"\\t\"}{.status.podIP}{\"\\n\"}{end}' | grep kceu | cut -f 2):8080\n```\n\n8. Clean up the manually created Service:\n```shell\nkubectl --kubeconfig $KUBECONFIG1 delete service mjpeg\nkubectl --kubeconfig $KUBECONFIG1 delete endpoints mjpeg\n```\n\n### Exercise 3: bonus!\nManually creating a Service in our cluster to mirror every Service that runs on the edge clusters will become hard to manage.\nTo simplify this, we can use a controller fully automate the reflection of Services from one cluster to another, [Service-Reflector](https://github.com/squat/service-reflector). \n\n1. Create an emitter in the edge cluster:\nThis container will publish selected Services for other controllers to mirror.\n```shell\nkubectl --kubeconfig $KUBECONFIG2 apply -f manifests/3/emitter.yaml\n```\n\n2. Register the emitter as a source of the cluster in the cloud:\n```\n# Register the emitter in cluster1 as a source of the reflector in cluster2.\ncat \u003c\u003cEOF | kubectl --kubeconfig $KUBECONFIG1 apply -f -\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: service-reflector\ndata:\n  source-api: http://$(kubectl --kubeconfig $KUBECONFIG2 get service service-reflector -o jsonpath='{.spec.clusterIP}'):8080\nEOF\n```\n\n3. Create a reflector in the cloud cluster:\nThis deployment will mirror the Services published by the configured sources so that they can be natively consumed in the local cluster.\n```shell\nkubectl --kubeconfig $KUBECONFIG1 apply -f manifests/3/reflector.yaml\n```\n\n4. Verify that we can still access our application:\n```shell\n$BROWSER $(kubectl --kubeconfig $KUBECONFIG1 get pods -o=jsonpath='{range .items[*]}{.metadata.name}{\"\\t\"}{.status.podIP}{\"\\n\"}{end}' | grep kceu | cut -f 2):8080\n```\n\n### Clean Up\n1. Finally, use Terraform to clean everything up:\n```shell\nterraform destroy --auto-approve\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsquat%2Fonseu2019","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsquat%2Fonseu2019","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsquat%2Fonseu2019/lists"}