{"id":13839317,"url":"https://github.com/jetstack/cni-migration","last_synced_at":"2025-07-11T03:31:53.923Z","repository":{"id":52712013,"uuid":"258055088","full_name":"jetstack/cni-migration","owner":"jetstack","description":"A CLI to migrate the CNI on a Kubernetes cluster from Canal (Calico + Flannel) to Cilium, live with no downtime.","archived":true,"fork":false,"pushed_at":"2020-11-16T12:24:23.000Z","size":153,"stargazers_count":74,"open_issues_count":3,"forks_count":13,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-11-21T04:30:58.168Z","etag":null,"topics":["calico","canal","ciliuim","cni","flannel","kubernetes","migration"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jetstack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-23T00:44:27.000Z","updated_at":"2024-08-21T13:25:07.000Z","dependencies_parsed_at":"2022-09-14T00:11:29.934Z","dependency_job_id":null,"html_url":"https://github.com/jetstack/cni-migration","commit_stats":null,"previous_names":["joshvanl/cni-migration"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jetstack/cni-migration","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jetstack%2Fcni-migration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jetstack%2Fcni-migration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jetstack%2Fcni-migration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jetstack%2Fcni-migration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jetstack","download_url":"https://codeload.github.com/jetstack/cni-migration/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jetstack%2Fcni-migration/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264721317,"owners_count":23653918,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["calico","canal","ciliuim","cni","flannel","kubernetes","migration"],"created_at":"2024-08-04T17:00:18.870Z","updated_at":"2025-07-11T03:31:53.635Z","avatar_url":"https://github.com/jetstack.png","language":"Go","readme":"cni-migration is a CLI tool for migrating a Kubernetes cluster's CNI solution\nfrom Flannel (Canal) to Cilium. The tool works by running both CNIs at the same\ntime using [multus-cni](https://github.com/intel/multus-cni/). All pods are\nupdated to attach a network interface from both CNIs, and then migrate each node\nto only running Cilium. This ensures that all pods are able to communicate to\nboth networks at all times during the migration.\n\n## How\n\nThe following are the steps taken to migrate the CNI. During and after each\nstep, the inter-pod communication is regularly tested using\n[knet-stress](https://github.com/joshvanl/knet-stress), which will send a HTTP\nrequest to all other knet-stress instances on all nodes. This proves a\nbi-directional network connectivity across cluster.\n\n1. This step involves installing both CNIs on all nodes and labelling the nodes\n   accordingly.\n\n- Label all nodes with `node-role.kubernetes/canal-cilium=true` and\n  patch the canal DaemonSet to have a node selector on this label.\n- Label all nodes with `node-role.kubernetes/cni-priority-canal=true`.\n- Deploy two knet-stress DaemonSets that run two knet-stress instances on each\n  node.\n- Deploy two DeamonSet instances of Cilium.\n  - The first has a node selector on the label\n    `node-role.kubernetes/cilium-canal=true` and writes its CNI config\n    to `99-cilium.conf`. This then runs on all nodes.\n  - The second has a node selector on the label\n    `node-role.kubernetes/cilium=true` and writes its CNI config\n    to `00-cilium.conf`. This will not run until a node is being migrated.\n- Deploy twp DaemonSet instances of Multus.\n  - Deploy multus DaemonSet with the node\n    selector`node-role.kubernetes/cni-priority-canal=true`. This has a static\n    config that uses the Flannel CNI config for the main Pod IP network interface, and\n    the Cilium as an extra network interface attached. The resulting CNI config is\n    written to `00-multus.conflist`. This CNI config will be chosen by the Kubelet\n    until the node has been migrated.\n  - Deploy multus DaemonSet with the node\n    selector`node-role.kubernetes/cni-priority-cilium=true`. This multus is the\n    same as the previous however swaps the primary Pod IP to that of Cilium\n    rather than Flannel.\n\n2. This step ensures that all workloads on the cluster are running with network\n   interfaces from both CNIs. The \"sbr\" Channing CNI is used to the at the\n   default route inside each pod is Cilium, however the Pod IP remains that of\n   the range of Flannel.\n\n- Roll all nodes in the cluster one by one. This step ensures that every pod\n  in the cluster is reassigned an IP, meaning that all pods will have a\n  network interface from both CNIs applied using multus.\n- Check knet-stress connectively after every node roll.\n- At this stage, all pods on the cluster have both CNI network interfaces\n  attached. All nodes are running the two CNIs which are controlled by multus.\n\n3. This step will reverse the order of priority of CNIs, so that Cilium becomes\n   the primary Pod IP, with an extra Flannel network interface attached.\n\n- Relabel and roll all the nodes on the on the cluster with the label\n  `node-role.kubernetes/cni-priority-cilium=true`. This will change the priority\n   of the CNI on each cluster to Cilium and have each Pod IP be in Cilium's range.\n- Check knet-stress connectively after every node roll.\n- At this stage, all pods on the cluster have both CNI network interfaces\n  attached, however the Pod IP is not in Cilium's range, rather than Flannel.\n\n4. This step is iterative by performing the same operation on all nodes until\n   they have all been migrated.\n\n- First, the selected node is drained, tainted, and has all pods deleted on it.\n  This node removes the label `node-role.kubernetes/cilium-canal=true`.\n  The taint added uses the label `node-role.kubernetes/cilium=true` which\n  terminates the first Cilium DaemonSet, replaced with the second. This second\n  Cilium DaemonSet writes its CNI config to `00-cilium.conf` which puts it as\n  the first CNI config to be selected and used by Kubelet, making this node now\n  only use Cilium CNI, rather than multus (Cilium _and_ Canal).\n- The node is untainted which allows workloads to be re-scheduled to it,\n  which will have only Cilium CNI network interfaces attached. These pods should\n  still be reachable by all other pods in the cluster.\n- The node has the label `node-role.kubernetes/migrated=true` added which\n  signals that this node has been migrated.\n\n5. After migrating all nodes, we now do a simple clean up of old resources.\n\n- The now unused and non-scheduled Multus, Canal, and first Cilium DaemonSets\n  are deleted.\n\nThe cluster should now be fully migrated from Canal to Cilium CNI.\n\n## Requirements\n\nThe following requirements apply in order to run the migration.\n\n### Firewall\n\n- Cilium uses Geneve as a backend mode and as such, needs the port 6081 over UDP\n  to communicate across nodes. This must be opened before migration.\n  *Note*: Cilium can not run in VXLAN mode since it has not been possible to\n  run two separate VXLAN interfaces on each host (one for Flannel and one for\n  Cilium).\n- All Kubernetes NetworkPolices will remain active and applied during, and after\n  the migration, being compatible with Cilium. No action needed.\n\n### Images\n\n- docker.io/cilium/cilium:v1.7.2\n- docker.io/cilium/operator:v1.7.2\n- nfvpe/multus:v3.4.1\n- gcr.io/jetstack-josh/knet-stress:cli (preferably a private image is built from\n  source and used)\n\n## Configuration\n\nThe cni-migration tool has input configuration file (default `--config\nconifg.yaml`), that holds options for the migration.\n\n### labels\n\nThis holds options on which label keys and shared value should be used for each\nsignal of steps:\n\n```yaml\n  canal-cilium: node-role.kubernetes.io/canal-cilium\n  cni-priority-canal: node-role.kubernetes.io/priority-canal\n  cni-priority-cilium: node-role.kubernetes.io/priority-cilium\n  rolled: node-role.kubernetes.io/rolled\n  cilium: node-role.kubernetes.io/cilium\n  migrated: node-role.kubernetes.io/migrated\n  value: \"true\" # used as the value to each label key\n```\n\n### paths\n\nThe file paths for each manifest bundle:\n\n```yaml\n  cilium: ./resources/cilium.yaml\n  multus: ./resources/multus.yaml\n  knet-stress: ./resources/knet-stress.yaml\n```\n\n### preflightResources\n\nList of resources that must exist before beginning the migration.\n\n```yaml\n  daemonsets:\n    knet-stress:\n    - knet-stress\n    - knet-stress-2\n  deployments:\n  statefulsets:\n```\n\n### watchedResources\n\nList of resources which must be ready when checked throughout the migration\nbefore continuing:\n\n```yaml\n  daemonsets:\n    kube-system:\n    - canal\n    - cilium\n    - cilium-migrated\n    - kube-multus-canal\n    - kube-multus-cilium\n    - kube-controller-manager\n    - kube-scheduler\n    knet-stress:\n    - knet-stress\n    - knet-stress-2\n  deployments:\n  statefulsets:\n```\n\n### cleanUpResources\n\nList of resources which will be removed after completing the migration\nsuccessfully:\n\n```yaml\n  daemonsets:\n    kube-system:\n    - canal\n    - cilium\n    - kube-multus-canal\n    - kube-multus-cilium\n    knet-stress:\n    - knet-stress\n    - knet-stress-2\n  deployments:\n  statefulsets:\n```\n","funding_links":[],"categories":["NetWork"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjetstack%2Fcni-migration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjetstack%2Fcni-migration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjetstack%2Fcni-migration/lists"}