{"id":13776071,"url":"https://github.com/raffaelespazzoli/openshift-sdn-encrypted-tunnel","last_synced_at":"2025-04-09T23:04:40.797Z","repository":{"id":76670596,"uuid":"150999788","full_name":"raffaelespazzoli/openshift-sdn-encrypted-tunnel","owner":"raffaelespazzoli","description":null,"archived":false,"fork":false,"pushed_at":"2018-12-11T12:48:51.000Z","size":19857,"stargazers_count":15,"open_issues_count":1,"forks_count":5,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-24T00:51:27.203Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raffaelespazzoli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-09-30T19:35:32.000Z","updated_at":"2024-05-27T13:03:19.000Z","dependencies_parsed_at":"2024-01-07T22:48:34.172Z","dependency_job_id":"c8bc984a-0324-4539-a91d-e9112e0fa769","html_url":"https://github.com/raffaelespazzoli/openshift-sdn-encrypted-tunnel","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raffaelespazzoli%2Fopenshift-sdn-encrypted-tunnel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raffaelespazzoli%2Fopenshift-sdn-encrypted-tunnel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raffaelespazzoli%2Fopenshift-sdn-encrypted-tunnel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raffaelespazzoli%2Fopenshift-sdn-encrypted-tunnel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raffaelespazzoli","download_url":"https://codeload.github.com/raffaelespazzoli/openshift-sdn-encrypted-tunnel/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248125628,"owners_count":21051770,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T17:01:59.392Z","updated_at":"2025-04-09T23:04:40.773Z","avatar_url":"https://github.com/raffaelespazzoli.png","language":"Shell","funding_links":[],"categories":["\u003ca id=\"01e6651181d405ecdcd92a452989e7e0\"\u003e\u003c/a\u003e工具"],"sub_categories":["\u003ca id=\"ea4dfcd8f33ec1852180c6283b2c8516\"\u003e\u003c/a\u003e未分类"],"readme":"# OpenShift SDN Encrypted Tunnel\n\nOpenShift SDN Ecnrypted tunnel will create a VPN mesh between OpenShift clyster SDNs in such a way that IP packets become routable between pods across SDNs.\n\nThe archietcture of the tunnel is depicted in this diagram:\n\n![vpn mesh](./media/VPNMesh.png)\n\nThe process works as follows\n\nSETUP PHASE:\n\n1. a tun device is created by the tunnel daemon set \n2. the tun is wired to the bridge so that ip packets destined to the CIDR of the other cluster are routed to the tunnel. \n\nTRANSMIT PHASE:\n\n1. a packet is put in the bridge with destination to the CIDR of one of the nodes of the other cluster\n2. the flow rules send the packet to the tunnel\n3. the tunnel daemonset process manages the wired side of the tunnel and sends the UDP-encapsulated and encrypted packet to the correct VIP of the other cluster\n\nRECEIVE PHASE:\n \n1. A UDP encapsulated and encrypted packet is received by the VIP and sent to the corresponding tunnel ds process \n2. the tunnel daemonset process extracts and decrypts the packet from the UDP envelope and puts it in the tun device. \n3. the packet ends up in the bridge. \n4. the bridge examines the destination, which will be local to the node, and delivers the packet immediately. \n\n\nThe routing of the packets works as described in this diagram:\n\n![routing](./media/routing.png)\n\n## Service Proxying\n\nService Proxying is the ability to load balance connection to Kubernetes services to pods. \nIn this case we want to be able to load abalnce conenction to a service of a connected Kubernetes cluster to the pods of this cluster.\nTo do so we use IPVS managed by [kube-router](https://www.kube-router.io/), the architecture is the following:\n\n![service-proxying](./media/service-proxying.png) \n\n## Discovery\n\nDiscovery is the ability to discover Kuberneets service IPs by quering a DNS.\nWe define the convention that each cluster has a domain of this pattern `.cluster.\u003ccluster-name\u003e`, besides the usual `.cluster.local`.\nWe use [coredns](https://coredns.io/) to setup a DNS server with the followign architecture:\n\n![coredns](./media/coredns.png)\n\nPods that need to use that DNS need the following in their template:\n\n```\n  dnsConfig:\n    nameservers:\n      - \u003ccoredns service IP\u003e\n    searches:\n      - svc.cluster.local\n      - cluster.local\n  dnsPolicy: None\n``` \n\n## Installation\n\nThese instructions will help you install the encrypted tunnel, service proxying and discovery on your OpenShift clusters.\n\n### Install wireguard\n\nWireguard needs to be installed in each of the nodes of your clusters.\n\nFor each of your clusters run the following:\n\n```\nansible nodes -i \u003ccluster_inventory\u003e -m shell -a \"curl -Lo /etc/yum.repos.d/wireguard.repo https://copr.fedorainfracloud.org/coprs/jdoss/wireguard/repo/epel-7/jdoss-wireguard-epel-7.repo\"\nansible nodes -i \u003ccluster_inventory\u003e -m shell -a \"wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm\"\nansible nodes -i \u003ccluster_inventory\u003e -m shell -a \"yum install -y epel-release-latest-7.noarch.rpm\"\nansible nodes -i \u003ccluster_inventory\u003e -m shell -a \"yum install -y wireguard-dkms wireguard-tools\"\n\n```\n\n### Run the installation playbook\n\nSee an example of the inventory [here](./ansible/inventory) and customize it for your clusters.\nHere is a minimum inventory:\n```\nclusters:\n- name: \u003ccluster_1_name\u003e\n  url: \u003ccluster_1_master_api_url\u003e\n  username: \u003ccluster_1_username\u003e\n  password: \u003ccluster_1_password\u003e  \n- name: \u003ccluster_2_name\u003e\n  url: \u003ccluster2_master_api_url\u003e\n  username: \u003ccluster_2_username\u003e\n  password: \u003ccluster_2_password\u003e \n```\nOther optional inventory variables are:\n\n| Variable Name  | Default  | Description  |\n|:-:|:-:|:-:|\n| tunnel_mode  | wireguard  | selects the tunnel mode. Currently only `wireguard` is supported.  |\n| namespace  | sdn-tunnel  | namespace in which the sdn-tunnel objects will be created  |\n| appname  | sdn-tunnel  | name and label shared by all the created resources  |\n| tunnel_port  | 5555  | UDP port used to create the the tunnel  |\n| image_sdn_tunnel  | quay.io/raffaelespazzoli/openshift-sdn-tunnel:latest  | image used by the sdn tunnel container  |\n| service_type | LoadBalancer | type of the service used to create the tunnel, supported values are `LoadBalancer` and `NodePort` |\n| coredns_image | quay.io/raffaelespazzoli/coredns:latest | the image to be used for the coredns pod |\n| service_account_name | sdn-tunnel | the service account used by the sdn-tunnel pod |\n| image_kube_router | quay.io/raffaelespazzoli/kube-router:latest | image to be used for the kube-router container |\n\n\n\nRun the playbook:\n\n```\nansible-playbook -i \u003cinventory\u003e ./ansible/playbooks/deploy-wireguard/config.yaml\n```\n\n## Test\n\nWe are going to create two pods in two different clusters and test the connectivity.\n\nLogin to the clusters and save the context in a variable:\n\n```\noc login --username=\u003cuser1\u003e --password=\u003cpwd1\u003e \u003curl1\u003e\nCLUSTER1=$(oc config current-context)\noc login --username=\u003cuser2\u003e --password=\u003cpwd2\u003e \u003curl2\u003e\nCLUSTER2=$(oc config current-context)\n```\n\nCreate the projects and the test pods:\n```\noc --context=$CLUSTER1 new-project test-sdn-tunnel\noc --context=$CLUSTER2 new-project test-sdn-tunnel\noc --context=$CLUSTER1 apply -n test-sdn-tunnel -f ./samples/test-app.yaml\noc --context=$CLUSTER2 apply -n test-sdn-tunnel -f ./samples/test-app.yaml\n```\nAfter a while the pod will be up and running.\nCollect the info needed for the test:\n\n```\nCLUSTER1_NAME=\u003cadd the name for the cluster that you used in the inventory file\u003e\nCLUSTER2_NAME=\u003cadd the name for the cluster that you used in the inventory file\u003e\nPOD1=$(oc --context $CLUSTER1 get pod -n test-sdn-tunnel | grep Running | awk '{print $1}')\nPOD2=$(oc --context $CLUSTER2 get pod -n test-sdn-tunnel | grep Running | awk '{print $1}')\nPOD1_IP=$(oc --context $CLUSTER1 get pod $POD1 -n test-sdn-tunnel -o jsonpath='{.status.podIP}')\nPOD2_IP=$(oc --context $CLUSTER2 get pod $POD2 -n test-sdn-tunnel -o jsonpath='{.status.podIP}')\nSVC1_IP=$(oc --context $CLUSTER1 get svc -n test-sdn-tunnel | grep httpd | awk '{print $3}')\nSVC2_IP=$(oc --context $CLUSTER2 get svc -n test-sdn-tunnel | grep httpd | awk '{print $3}')\nDNS1_IP=$(oc --context $CLUSTER1 get svc -n sdn-tunnel | grep coredns | awk '{print $3}')\nDNS2_IP=$(oc --context $CLUSTER2 get svc -n sdn-tunnel | grep coredns | awk '{print $3}')\n```\n\nLet's test simple pod to pod connectivity:\n\n```\noc --context $CLUSTER1 exec $POD1 -n test-sdn-tunnel -- curl http://$POD2_IP:8080\noc --context $CLUSTER2 exec $POD2 -n test-sdn-tunnel -- curl http://$POD1_IP:8080\n```\n\nLet's test connectivity via the services:\n\n```\noc --context $CLUSTER1 exec $POD1 -n test-sdn-tunnel -- curl http://$SVC2_IP:8080\noc --context $CLUSTER2 exec $POD2 -n test-sdn-tunnel -- curl http://$SVC1_IP:8080\n```\n\nTo test connectivity via name resolution we need to inform the pod on how to resolve the names:\n\n```\noc --context $CLUSTER1 patch deployment httpd -n test-sdn-tunnel -p '{\"spec\":{\"template\":{\"spec\":{\"dnsPolicy\": \"None\", \"dnsConfig\":{\"nameservers\":[\"'$DNS1_IP'\"], \"searches\":[\"svc.cluster.local\",\"cluster.local\"]}}}}}'\noc --context $CLUSTER2 patch deployment httpd -n test-sdn-tunnel -p '{\"spec\":{\"template\":{\"spec\":{\"dnsPolicy\": \"None\", \"dnsConfig\":{\"nameservers\":[\"'$DNS2_IP'\"], \"searches\":[\"svc.cluster.local\",\"cluster.local\"]}}}}}'\n```\n\nThis will cause the pod to be redeployed, so we need to capture the new IPs:\n\n```\nPOD1=$(oc --context $CLUSTER1 get pod -n test-sdn-tunnel | grep Running | awk '{print $1}')\nPOD2=$(oc --context $CLUSTER2 get pod -n test-sdn-tunnel | grep Running | awk '{print $1}')\n```\n\nNow we can test service discovery and connectivity using the service name:\n\n```\noc --context $CLUSTER1 exec $POD1 -n test-sdn-tunnel -- curl http://httpd.test-sdn-tunnel.svc.cluster.$CLUSTER2_NAME:8080\noc --context $CLUSTER2 exec $POD2 -n test-sdn-tunnel -- curl http://httpd.test-sdn-tunnel.svc.cluster.$CLUSTER1_NAME:8080\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraffaelespazzoli%2Fopenshift-sdn-encrypted-tunnel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraffaelespazzoli%2Fopenshift-sdn-encrypted-tunnel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraffaelespazzoli%2Fopenshift-sdn-encrypted-tunnel/lists"}