{"id":16139407,"url":"https://github.com/networkop/meshnet-cni","last_synced_at":"2025-04-09T23:18:45.594Z","repository":{"id":39581868,"uuid":"152452048","full_name":"networkop/meshnet-cni","owner":"networkop","description":"a (K8s) CNI plugin to create arbitrary virtual network topologies","archived":false,"fork":false,"pushed_at":"2024-08-15T23:13:03.000Z","size":44533,"stargazers_count":119,"open_issues_count":22,"forks_count":28,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-09T23:18:40.062Z","etag":null,"topics":["cni","k8s","network","simulation","topology"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/networkop.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-10T16:06:25.000Z","updated_at":"2025-03-26T15:47:54.000Z","dependencies_parsed_at":"2024-06-18T17:09:49.870Z","dependency_job_id":"1bc167df-1c28-4f50-8462-b75de7a08a5b","html_url":"https://github.com/networkop/meshnet-cni","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/networkop%2Fmeshnet-cni","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/networkop%2Fmeshnet-cni/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/networkop%2Fmeshnet-cni/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/networkop%2Fmeshnet-cni/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/networkop","download_url":"https://codeload.github.com/networkop/meshnet-cni/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248125592,"owners_count":21051771,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cni","k8s","network","simulation","topology"],"created_at":"2024-10-09T23:49:00.337Z","updated_at":"2025-04-09T23:18:45.568Z","avatar_url":"https://github.com/networkop.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"[![Actions Status](https://github.com/networkop/meshnet-cni/workflows/CI/badge.svg)](https://github.com/networkop/meshnet-cni/actions)\n[![Docker Pulls](https://img.shields.io/docker/pulls/networkop/meshnet.svg)](https://img.shields.io/docker/pulls/networkop/meshnet)\n[![Go Report Card](https://goreportcard.com/badge/github.com/networkop/meshnet-cni)](https://goreportcard.com/report/github.com/networkop/meshnet-cni)\n[![GoDoc](https://godoc.org/istio.io/istio?status.svg)](https://pkg.go.dev/github.com/networkop/meshnet-cni)\n[![License: BSD](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)\n\n# meshnet CNI \n\n**meshnet** is a (K8s) CNI plugin to create arbitrary network topologies out of point-to-point links with the help of [koko](https://github.com/redhat-nfvpe/koko). Heavily inspired by [Ratchet-CNI](https://github.com/dougbtv/ratchet-cni), [kokonet](https://github.com/s1061123/kokonet) and [Multus](https://github.com/intel/multus-cni).\n\n\n\n## New in version 0.2.0\n\n* Using K8s etcd as datastore via Custom Resources\n* All internal communication now happens over gRPC\n* Support for macvlan links to connect to external resources\n\n\n## Architecture\nThe goal of this plugin is to interconnect pods via direct point-to-point links according to a pre-define topology. To do that, the plugin uses three types of links:\n* **veth** - used to connect two pods running on the same host\n* **vxlan** - used to connected two pods running on different hosts\n  * Optionally users can opt in to use **gRPC** instead, for this use case. Check [Installation](#Installation).\n* **macvlan** - used to connect to external resources, i.e. any physical or virtual device outside of the Kubernetes cluster\n\nTopology information, represented as a list of links per pod, is stored in k8s's etcd datastore as custom resources:\n\n```yaml\napiVersion: networkop.co.uk/v1beta1\nkind: Topology\nmetadata:\n  name: r1\nspec:\n  links:\n  - uid: 1\n    peer_pod: r2\n    local_intf: eth1\n    local_ip: 12.12.12.1/24\n    peer_intf: eth1\n    peer_ip: 12.12.12.2/24\n```\n\nThe plugin configuration file contains a \"chained\" `meshnet` in the list of plugins:\n\n```yaml\n{\n  \"cniVersion\": \"0.3.1\",\n  \"name\": \"kindnet\",\n  \"plugins\": [\n    {\n      \"ipMasq\": false,\n      \"ipam\": {\n        \"dataDir\": \"/run/cni-ipam-state\",\n        \"ranges\": [\n          [\n            {\n              \"subnet\": \"10.244.0.0/24\"\n            }\n          ]\n        ],\n        \"routes\": [\n          {\n            \"dst\": \"0.0.0.0/0\"\n          }\n        ],\n        \"type\": \"host-local\"\n      },\n      \"mtu\": 1500,\n      \"type\": \"ptp\"\n    },\n    {\n      \"capabilities\": {\n        \"portMappings\": true\n      },\n      \"type\": \"portmap\"\n    },\n    {\n      \"name\": \"meshnet\",\n      \"type\": \"meshnet\",\n      \"ipam\": {},\n      \"dns\": {}\n    }\n  ]\n}\n```\n\nThe plugin consists of three main components:\n\n* **datastore** - a k8s native etcd backend cluster storing topology information and runtime pod metadata (e.g. pod IP address and NetNS)\n* **meshnet** - a CNI binary responsible for pod's network configuration\n* **meshnetd** - a daemon responsible for communication with k8s and vxlan (or grpc) link configuration updates\n\n![architecture](arch_v0_2_0.png)\n\nBelow is the order of operation of the plugin from the perspective of kube-node-1:\n\n1. Kubernetes cluster gets populated with the topology information via custom resources\n2. pod-1/pod-2 come up, local kubelet calls the `meshnet` binary for each pod to setup their networking.\n3. Based on the CNI configuration file, Kubelet calls meshnet to set up additional interfaces.\n  \u003e Note that `eth0` is **always** setup by one of the existing CNI plugins. It is used to provide external connectivity to and from the pod\n4. meshnet binary updates the topology data with pod's runtime metadata (namespace filepath and primary IP address).\n5. meshnet binary (via a local meshnet daemon) retrieves the list of `links` and looks up peer pod's metadata to identify what kind of link to setup - veth, vxlan or macvlan.\n6. If the peer is on the same node, it calls koko to setup a `veth` link between the two pods.\n7. If the peer is on the remote node, it does two things:  \n    * 7.1 It calls koko to setup a local `vxlan` link.\n    * 7.2 It makes a gRPC `Update` call to the remote node's meshnet daemon, specifying this link's metadata (e.g. VTEP IP and VNI).\n8. Upon receipt of this information, remote node's `meshnetd` idepmotently updates the local vxlan link, i.e. it creates a new link, updates the existing link if there's a change or does nothing if the link attributes are the same.\n\n## Local Demo\n\nClone this project and build a local 3-node Kubernetes cluster\n\n```sh\nmake up\n```\n\nBuild the meshnet-cni docker image\n\n```sh\nmake docker\n```\n\nInstall meshnet-cni plugin\n\n```sh\n# install meshnet with VXLAN link\nmake install\n# or install meshnet with gRPC link\nmake grpc=1 install\n```\n\nVerify that meshnet is up and `READY`\n\n```sh\nkubectl get daemonset -n meshnet\n```\n\nInstall a 3-node test topology\n\n```sh\nkubectl apply -f tests/3node.yml\n```\n\nCheck that all pods are running\n\n```sh\nkubectl get pods -l test=3node                                                                  \nNAME   READY   STATUS    RESTARTS   AGE\nr1     1/1     Running   0          40m\nr2     1/1     Running   0          40m\nr3     1/1     Running   0          40s\n```\n\nTest connectivity between pods\n\n```sh\nkubectl exec r1 -- ping -c 1 12.12.12.2\nkubectl exec r2 -- ping -c 1 23.23.23.3\nkubectl exec r3 -- ping -c 1 13.13.13.1\n```\n\nCleanup\n\n```sh\nkubectl delete --grace-period=0 --force -f tests/3node.yml\n```\n\nDestroy the local kind cluster\n\n```sh\nmake down\n```\n\n\n## Installation\n\nThe following manifest will create all that's required for meshnet plugin to function, i.e.:\n\n* A `meshnet` namespace\n* A Custom Resource Definition for network topologies\n* A set of RBAC rules to allow meshnet to interact with new custom resources\n* A daemonset with meshnet plugin and configuration files\n\n```sh\n# to install meshnet with VXLAN link\nkubectl apply -k manifests/base\n# to install meshnet with gRPC link\nkubectl apply -k manifests/overlays/grpc-link\n```\n\n#### Interaction with existing resources\n\nMeshnet plugin was designed to work alongside any other existing or future Kubernetes resources that may not require any special topology to be set up for them. Every pod coming up will have its first interface setup by an existing CNI plugin (e.g. flannel, weave, calico) and will only have additional interfaces connected if there's a matching custom `Topology` resource.\n\nDuring the initial installation process, meshnet will try to insert itself into the list of CNI plugins. For example, assuming the following configuration is present in `/etc/cni/net.d/weave.conf`:\n\n```\n{\n  \"cniVersion\": \"0.2.0\",\n  \"name\": \"weave\",\n  \"type\": \"weave-net\",\n}\n```\nMeshnet will convert the above to conflist and produce the file `/etc/cni/net.d/00-meshnet.conflist` with the following content:\n\n```\n{\n  \"cniVersion\": \"0.2.0\",\n  \"name\": \"weave\",\n  \"plugins\": [\n    {\n      \"cniVersion\": \"0.2.0\",\n      \"name\": \"weave\",\n      \"type\": \"weave-net\"\n    },\n    {\n      \"name\": \"meshnet\",\n      \"type\": \"meshnet\",\n      \"ipam\": {},\n      \"dns\": {}\n    }\n  ]\n}\n```\n\n### Customising installation paths\n\nIn some cases, Kubernetes distros may store CNI configuration files and binaries in non-standard directories and override them with `--cni-bin-dir` and `--cni-conf-dir` flags. In order to install meshnet into the right directories, create a new overlay under `manifests/overlays` and patch the `cni-dir` or `cni-bin` volumes with the correct location. See [kops overlay](manifests/overlays/kops) for an example.\n\n### Resilient topologies\n\nIf you need to have Pods restarted and re-scheduled by the kube-controller, it's possible to deploy them as StatefulSets with replica number = 1. See [this example](/tests/2node-sts.yml).\n\n### Examples\n\nInside the `tests` directory there are 4 manifests with the following test topologies:\n\n* A simple point-to-point 2-node topology\n* A 3-node topology connected as a triangle\n* A 5-node topology connected as [quincunx](https://en.wikipedia.org/wiki/Quincunx)\n* A 2-node topology with 2nd node connected to a macvlan interface\n\n#### Use k8s-topo to orchestrate network topologies\n\nLogin the K8s master node and\n\n```\ngit clone https://github.com/networkop/k8s-topo.git \u0026\u0026 cd k8s-topo\n```\n\nDeploy k8s-topo pod\n\n```\nkubectl create -f manifest.yml\n```\n\nConnect to the k8s-topo pod\n\n```\nkubectl exec -it k8s-topo sh\n```\n\nCreate a random 20-node network topology\n\n```\n./examples/builder/builder 20 0\nTotal number of links generated: 19\n```\n\nCreate the topology inside K8s\n\n```\nk8s-topo --create examples/builder/random.yml \n```\n\nOptionally, you can generate a D3.js network topology graph \n\n```\nk8s-topo --graph examples/builder/random.yml \n```\n\nView the generated topology graph at `http://\u003cany_k8s_cluster_node_ip\u003e:32000`\n\nVerify that the topology has been deployed (from the master node)\n\n```\nkubectl get pods -o wide | grep qrtr\nqrtr-1     1/1     Running   0          11s     10.233.65.231   node3   \u003cnone\u003e\nqrtr-10    1/1     Running   0          11s     10.233.65.234   node3   \u003cnone\u003e\nqrtr-11    1/1     Running   0          10s     10.233.66.246   node4   \u003cnone\u003e\n```\n\nLogin the first node and verify connectivity to every other loopback \n\n```\n$ qrtr-1\n/ # for i in `seq 0 20`; do echo \"192.0.2.$i =\u003e\"  $(ping -c 1 -W 1 192.0.2.$i|grep loss); done\n192.0.2.0 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.1 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.2 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.3 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.4 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.5 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.6 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.7 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.8 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.9 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.10 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.11 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.12 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.13 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.14 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.15 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.16 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.17 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.18 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n192.0.2.19 =\u003e 1 packets transmitted, 1 packets received, 0% packet loss\n```\n\nDestroy the topology\n\n```\nk8s-topo --destroy examples/builder/random.yml \n```\n\n\n## Troubleshooting\n\nThere are two places to collect meshnet logs:\n\n1. Meshnet daemon logs can be collected outside of the Kubernetes cluster. For example, the below command will collect logs from all meshnet daemons using [stern](https://github.com/wercker/stern)\n\n\n```\nstern meshnet -n meshnet\n```\n\n2. Meshnet plugin (binary) logs can be collected on the respective Kubernetes nodes, e.g.\n\n```\nroot@kind-worker:/# journalctl -u kubelet\n```\n\n---\n\nEach POD is supposed to run an `init-wait` container that waits for the right number of interface to be connected before passing the ball to the main container. However, sometimes, PODs restart resulting in the missing interfaces inside the main container process, since they may have been added *AFTER* the process that reads the container interface list (e.g. qemu-kvm for VM-based containers). This is the procedure I use to identify the cause of the failure:\n\n1. Identify which POD is at fault. This will most likely be the incorrect number of interfaces.\n2. Identify which interface is missing or was added last.\n3. Identify the correlation between the pair of containers interconnected by the missing interface\n4. Look for the peer container's failures using `kubectl get events --sort-by=.metadata.creationTimestamp'`\n5. Identify which k8s node this POD is running on `kubectl get pods acme-scs1001-a -o yaml  | grep node`\n6. On that node check the `journalctl` for any errors associated with the POD\n\n\n\n\n[dynamic-client]: https://www.oreilly.com/library/view/programming-kubernetes/9781492047094/ch04.html\n[dynclient-example]: https://github.com/kubernetes/client-go/blob/611184f7c43ae2d520727f01d49620c7ed33412d/examples/dynamic-create-update-delete-deployment/main.go\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetworkop%2Fmeshnet-cni","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnetworkop%2Fmeshnet-cni","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetworkop%2Fmeshnet-cni/lists"}