{"id":13928961,"url":"https://github.com/Nordix/ipam-node-annotation","last_synced_at":"2025-07-19T10:32:34.929Z","repository":{"id":103062223,"uuid":"543090309","full_name":"Nordix/ipam-node-annotation","owner":"Nordix","description":"An IPAM CNI-plugin that reads ranges from an annotation in the K8s node object","archived":false,"fork":false,"pushed_at":"2023-02-11T15:27:27.000Z","size":69,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-30T05:43:16.342Z","etag":null,"topics":["cni-plugin","ipam"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Nordix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-09-29T11:41:57.000Z","updated_at":"2023-09-27T09:49:46.000Z","dependencies_parsed_at":"2023-09-25T07:09:40.948Z","dependency_job_id":null,"html_url":"https://github.com/Nordix/ipam-node-annotation","commit_stats":{"total_commits":16,"total_committers":1,"mean_commits":16.0,"dds":0.0,"last_synced_commit":"a003a19599df12a9ad588a89897901818377b451"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nordix%2Fipam-node-annotation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nordix%2Fipam-node-annotation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nordix%2Fipam-node-annotation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nordix%2Fipam-node-annotation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Nordix","download_url":"https://codeload.github.com/Nordix/ipam-node-annotation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226588902,"owners_count":17655802,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cni-plugin","ipam"],"created_at":"2024-08-07T18:02:00.995Z","updated_at":"2024-11-26T17:31:12.934Z","avatar_url":"https://github.com/Nordix.png","language":"Shell","funding_links":[],"categories":["others"],"sub_categories":[],"readme":"# IPAM CNI-plugin - kube-node and node-annotation\n\n`Kube-node` and `node-annotation` are IPAM [CNI-plugins](\nhttps://github.com/containernetworking/cni) that assigns IP addresses\nto PODs based on the Kubernetes node object. `Node-annotation` is an\nexperimental plugin written with shell scripts. It is kept as a\nreference. `Kube-node` is written in go and should be prefered.\n\nFor addresses on the main Kubernetes network, `eth0` in PODs, the\naddress ranges (subnets) are taken from `node.spec.podCIDRs`. This is\nautomatically generated if the `kube-controller-manager` is started\nwith `--allocate-node-cidrs=true`. Example:\n\n```bash\n# kubectl get node vm-002 -o json | jq .spec.podCIDRs\n[\n  \"11.0.1.0/24\",\n  \"1100:0:0:1::/64\"\n]\n```\n\n```json\n{\n  \"name\": \"k8snet\",\n  \"cniVersion\": \"1.0.0\",\n  \"isDefaultGateway\": true,\n  \"ipam\": {\n    \"type\": \"kube-node\",\n    \"kubeconfig\": \"/etc/kubernetes/kubeconfig\",\n    \"dataDir\": \"/run/container-ipam-state/k8snet\"\n  }\n}\n```\n\nA few things should be noted. `Kube-node` must be able to read node\nobjects from the K8s API-server, so it must have a working\n`kubeconfig`. This may be omitted if the kubeconfig can be found in\nother ways, for instance using the `$KUBECONFIG` environment variable.\n\n`Kube-node` delegates address allocation to the [host-local](\nhttps://www.cni.dev/plugins/current/ipam/host-local/) ipam, so the `dataDir`\nshould be set to a directory that is cleared on node reboot (e.g. a\n`tmpfs`).\n\n\n\n## Limit IPv4 address allocation\n\nIn large clusters IPv4 addresses may become a [limiting resource](\nhttps://github.com/kubernetes/kubernetes/issues/109814#issuecomment-1138840270).\n`Kube-node` can be configured to only assign IPv4 address to some namespaces.\n\n```json\n{\n  \"name\": \"k8snet\",\n  \"cniVersion\": \"1.0.0\",\n  \"isDefaultGateway\": true,\n  \"ipam\": {\n    \"type\": \"kube-node\",\n    \"kubeconfig\": \"/etc/kubernetes/kubeconfig\",\n    \"dataDir\": \"/run/container-ipam-state/k8snet\",\n    \"ipv4-namespaces\": [\n        \"old-application\"\n    ]\n  }\n}\n```\n\n**WARNING**: If this is used, Kubernetes must be configured to use\nIPv6 addresses for access to the API-server and as default for\nservices. This is done by specifying an IPv6 address for the\n`--advertise-address` and set an IPv6 address first in\n`--service-cluster-ip-range` when the `kube-apiserver` is started.\n\n\n\n## Secondary networks\n\nif `kube-node` is used to assign addresses on interfaces which are\n*not* the Kubernetes network, for instance interfaces added with [Multus](\nhttps://github.com/k8snetworkplumbingwg/multus-cni), the address\nranges (subnets) can be specified in an annotation on the node object.\n\n```\nkubectl annotate node vm-002 kube-node.nordix.org/net1=172.20.2.0/24,fd00::2:0:0/96\n```\n\n```json\n{\n  \"name\": \"net1\",\n  \"cniVersion\": \"1.0.0\",\n  \"isDefaultGateway\": false,\n  \"ipam\": {\n    \"type\": \"kube-node\",\n    \"kubeconfig\": \"/etc/kubernetes/kubeconfig\",\n    \"dataDir\": \"/run/container-ipam-state/net1\",\n    \"annotation\": \"kube-node.nordix.org/net1\"\n  }\n}\n```\nNote that `isDefaultGateway` is set to \"false\" for secondary networks.\n\n\n## Build\n\n```\n./build.sh      # Print help\n./build.sh binaries\n```\n\n\n\n\n## The node-annotation IPAM\n\nThe `node-annotation` is an IPAM [CNI-plugin](\nhttps://github.com/containernetworking/cni) that reads the addresses\nfrom an annotation in the Kubernetes [node object](\nhttps://kubernetes.io/docs/concepts/architecture/nodes/).\n\nExample:\n```json\n{\n    \"name\": \"mynet\",\n    \"type\": \"ipvlan\",\n    \"master\": \"eth0\",\n    \"ipam\": {\n        \"type\": \"node-annotation\",\n        \"annotation\": \"example.com/ipvlan-ranges\"\n    }\n}\n```\nAnd in the node object:\n```\n# kubectl get node $(hostname) -o json | jq .metadata.annotations\n{\n  \"example.com/ipvlan-ranges\": \"ranges: [[{\\\"subnet\\\": \\\"172.16.5.0/24\\\"}]]\",\n  \"node.alpha.kubernetes.io/ttl\": \"0\",\n  \"volumes.kubernetes.io/controller-managed-attach-detach\": \"true\"\n}\n```\n\nThe `node-annotation` ipam is a wrapper for the [host-local](\nhttps://www.cni.dev/plugins/current/ipam/host-local/) ipam (by\ndefault). *Anything* in the annotation will be inserted in a\n`host-local` config. In the figure everything in bold is taken from\nthe annotation.\n\n\u003cimg src=\"node-annotation-plugin.svg\" width=\"85%\" /\u003e\n\nThis gives you the freedom to use any `host-local` configuration.\n\nFor now `node-annotation` is implemented as a shell script. It is\nintended mainly for testing.\n\n\n### Usage\n\n`node-annotation` shall be installed in the cni-bin directory, usually\n\"/opt/cni/bin\". `node-annotation` must be able to get the K8s node\nobjects using `kubectl get nodes -o json` and analyze with [jq](\nhttps://stedolan.github.io/jq/).\n\nConfiguration is in `json` format and is read from\n`/etc/cni/node-annotation.conf` or `$NODE_ANNOTATION_CFG`. Example;\n\n```json\n{\n   \"kubeconfig\": \"/etc/kubernetes/kubeconfig\",\n   \"nextipam\": \"/opt/cgi/bin/host-local\"\n}\n```\n\n`kubeconfig` is needed unless `$KUBECONFIG` is defined. `nextipam` is\noptional and may be used to chain with another plugin than `host-local`.\n\n**NOTE**; a \"key\" must only contain characters that are valid in a\n  shell script variable. That means no dash (-).\n\n\n\n### Manual Testing\n\nThe `node-annotation` script can be invoked with a parameter to test\nsome things on a cluster.\n\n```\n# /opt/cni/bin/node-annotation -h\n\n node-annotation --\n\n   An IPAM CNI-plugin that uses annotations on the K8s node object\n   https://github.com/Nordix/ipam-node-annotation\n\n Commands;\n\n   ipam\n     Act as an ipam CNI-plugin. This is the default command\n   error_quit [msg]\n     Print an error in standard CNI json format and quit\n   my_node\n     Print the own node object\n   get_annotation [--node=node] \u003cannotation\u003e\n     Print the value of the annotation in the K8s node object\n\n# kubectl annotate node $(hostname) example.com/ipvlan-ranges=\"\\\"ranges\\\": [\n  { \\\"subnet\\\": \\\"4000::16.0.0.0/120\\\" },\n  { \\\"subnet\\\": \\\"16.0.0.0/24\\\" }\n]\"\n# /opt/cni/bin/node-annotation get_annotation example.com/ipvlan-ranges\n\"ranges\": [\n  [{ \"subnet\": \"4000::16.0.0.0/120\" }],\n  [{ \"subnet\": \"16.0.0.0/24\" }]\n]\n```\n\nA test of the entire function;\n```\n# cat \u003e test.cfg \u003c\u003cEOF\n{\n    \"name\": \"mynet\",\n    \"type\": \"ipvlan\",\n    \"master\": \"eth0\",\n    \"ipam\": {\n        \"type\": \"node-annotation\",\n        \"annotation\": \"example.com/ipvlan-ranges\"\n    }\n}\nEOF\n# export NODE_ANNOTATION_CFG=./nacfg\n# cat \u003e $NODE_ANNOTATION_CFG \u003c\u003cEOF\n{\n  \"nextipam\": \"cat\"\n}\nEOF\ncat test.cfg | /opt/cni/bin/node-annotation\n{\n  \"name\": \"mynet\",\n  \"type\": \"ipvlan\",\n  \"master\": \"eth0\",\n  \"ipam\": {\n    \"type\": \"cat\",\n    \"ranges\": [\n      {\n        \"subnet\": \"4000::16.0.0.0/120\"\n      },\n      {\n        \"subnet\": \"16.0.0.0/24\"\n      }\n    ]\n  }\n}\n```\n\n\n### Kind\n\nIf you have [Multus](https://github.com/k8snetworkplumbingwg/multus-cni)\nrunning in [kind](https://kind.sigs.k8s.io/) there are some steps to make\n`node-annotation` work.\n\nFirst copy the `node-annotation` to \"/opt/cni/bin\" on all workers.\n\nOnly the controller have a kubeconfig at \"/etc/kubernetes/admin.conf\".\nCreate one at the workers and configure `node-annotation` to use it;\n\n```\nkind get kubeconfig --internal | \\\n  docker exec -i kind-worker tee /etc/kubernetes/kubeconfig \u003e /dev/null\necho \"{ \\\"kubeconfig\\\": \\\"/etc/kubernetes/kubeconfig\\\" }\" | docker exec -i kind-worker \\\n  tee /etc/cni/node-annotation.conf \u003e /dev/null\n# (repeat for all workers)\n```\n\nThe Multus kubeconfig (/etc/cni/net.d/multus.d/multus.kubeconfig)\ncan't be used due to restrictions.\n\nNow you can annotate the workers and create a NAD. Example;\n```\nkubectl annotate node worker example.com/bridge=\"\\\"ranges\\\": [\n  [{ \\\"subnet\\\":\\\"4000::16.0.0.0/120\\\", \\\"rangeStart\\\":\\\"4000::16.0.0.0\\\" , \\\"rangeEnd\\\":\\\"4000::16.0.0.7\\\"}],\n  [{ \\\"subnet\\\":\\\"16.0.0.0/24\\\", \\\"rangeStart\\\":\\\"16.0.0.0\\\" , \\\"rangeEnd\\\":\\\"16.0.0.7\\\"}]\n]\"\ncat | kubectl apply -f - \u003c\u003cEOF\napiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: meridio-bridge\nspec:\n  config: '{\n    \"cniVersion\": \"0.4.0\",\n    \"type\": \"bridge\",\n    \"bridge\": \"cbr2\",\n    \"isGateway\": true,\n    \"ipam\": {\n      \"type\": \"node-annotation\",\n      \"annotation\": \"example.com/bridge\"\n    }\n  }'\nEOF\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNordix%2Fipam-node-annotation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNordix%2Fipam-node-annotation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNordix%2Fipam-node-annotation/lists"}