{"id":29646346,"url":"https://github.com/johnlam90/aws-multi-eni-controller","last_synced_at":"2026-05-10T03:17:51.735Z","repository":{"id":292940546,"uuid":"978473283","full_name":"johnlam90/aws-multi-eni-controller","owner":"johnlam90","description":"A Kubernetes controller that automatically creates and attaches AWS Elastic Network Interfaces (ENIs) to nodes based on node labels.","archived":false,"fork":false,"pushed_at":"2025-07-02T21:17:33.000Z","size":23694,"stargazers_count":1,"open_issues_count":15,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-02T22:27:18.492Z","etag":null,"topics":["aws","cni","cni-plugin","crd","elastic-network-interface","eni","kubernetes","multus"],"latest_commit_sha":null,"homepage":"https://johnlam90.github.io/aws-multi-eni-controller/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/johnlam90.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-06T03:32:43.000Z","updated_at":"2025-06-09T20:35:33.000Z","dependencies_parsed_at":"2025-06-05T04:59:33.318Z","dependency_job_id":"86938bf5-575a-40b3-b7a5-63843c1b5066","html_url":"https://github.com/johnlam90/aws-multi-eni-controller","commit_stats":null,"previous_names":["johnlam90/aws-multi-eni-controller"],"tags_count":46,"template":false,"template_full_name":null,"purl":"pkg:github/johnlam90/aws-multi-eni-controller","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnlam90%2Faws-multi-eni-controller","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnlam90%2Faws-multi-eni-controller/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnlam90%2Faws-multi-eni-controller/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnlam90%2Faws-multi-eni-controller/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/johnlam90","download_url":"https://codeload.github.com/johnlam90/aws-multi-eni-controller/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnlam90%2Faws-multi-eni-controller/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266413577,"owners_count":23924766,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cni","cni-plugin","crd","elastic-network-interface","eni","kubernetes","multus"],"created_at":"2025-07-22T02:39:01.329Z","updated_at":"2026-05-10T03:17:51.701Z","avatar_url":"https://github.com/johnlam90.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS Multi-ENI Controller for Kubernetes\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Go Report Card](https://img.shields.io/badge/Go%20Report-A%2B-brightgreen?logo=go)](https://github.com/johnlam90/aws-multi-eni-controller/actions/workflows/go-report.yml)\n[![Go](https://img.shields.io/badge/Go-1.23+-00ADD8.svg)](https://go.dev/)\n[![Helm](https://img.shields.io/badge/Helm-v3-0F1689.svg)](https://helm.sh)\n[![Version](https://img.shields.io/badge/Version-v1.3.5-blue.svg)](https://github.com/johnlam90/aws-multi-eni-controller/releases)\n[![GitHub Pages](https://img.shields.io/badge/GitHub%20Pages-Active-brightgreen)](https://johnlam90.github.io/aws-multi-eni-controller/)\n[![OpenSSF Best Practices](https://img.shields.io/badge/OpenSSF-Best%20Practices-brightgreen)](https://www.bestpractices.dev/)\n\n## Overview\n\nThe AWS Multi-ENI Controller is a Kubernetes controller **purpose-built for Multus CNI deployments on AWS**. It automatically creates and attaches AWS Elastic Network Interfaces (ENIs) to nodes based on node labels, enabling Multus CNI to provide multiple network interfaces to pods without complex infrastructure templates.\n\nBy bridging the gap between AWS networking and Kubernetes multi-network capabilities, this controller solves the challenge of dynamically provisioning and managing ENIs for Multus CNI. It follows the Kubernetes operator pattern and provides a declarative way to manage ENIs through custom resources, making it ideal for workloads that require multiple network interfaces, such as networking plugins, security tools, or specialized applications.\n\n## Architecture\n\nThe AWS Multi-ENI Controller consists of two main components:\n\n1. **NodeENI Controller**: Watches for NodeENI custom resources and nodes with matching labels. When a node matches the selector in a NodeENI resource, the controller creates an ENI in the specified subnet with the specified security groups and attaches it to the node at the specified device index.\n\n2. **ENI Manager**: A DaemonSet that runs on nodes with matching labels and automatically brings up secondary interfaces when they're attached.It can also set mtu,bind dpdk interfaces and advertise sriov dpdk resources to the ec2 worker node.\n\n### System Architecture Diagram\n\nThe following diagram illustrates the overall architecture and interaction between components:\n\n![AWS Multi-ENI Controller Architecture](docs/diagrams/arch.svg)\n\n\u003e **Note**: This diagram is automatically generated from `docs/diagrams/arch.drawio.svg` using the [VS Code Draw.io extension](https://marketplace.visualstudio.com/items?itemName=hediet.vscode-drawio). To edit the diagram, open the `.drawio.svg` file in VS Code.\n\nThis diagram shows how the AWS Multi-ENI Controller integrates with Kubernetes and AWS to provide multiple network interfaces for pods using Multus CNI.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"docs/diagrams/controller-flow.svg\" alt=\"AWS Multi-ENI Controller Architecture Diagram\" /\u003e\n\u003c/div\u003e\n\n### ENI Lifecycle\n\n```mermaid\nsequenceDiagram\n    participant User\n    participant NodeENI as NodeENI Resource\n    participant Controller as NodeENI Controller\n    participant AWS as AWS EC2 API\n    participant Node as Kubernetes Node\n    participant ENIManager as ENI Manager DaemonSet\n\n    User-\u003e\u003eNodeENI: Create NodeENI resource\n    NodeENI-\u003e\u003eController: Watch event\n    Controller-\u003e\u003eAWS: Create ENI\n    AWS--\u003e\u003eController: ENI created\n    Controller-\u003e\u003eAWS: Attach ENI to node\n    AWS--\u003e\u003eController: ENI attached\n    Controller-\u003e\u003eNodeENI: Update status\n    ENIManager-\u003e\u003eNode: Detect new ENI\n    ENIManager-\u003e\u003eNode: Configure interface\n\n    Note over User,ENIManager: Deletion Flow\n\n    User-\u003e\u003eNodeENI: Delete NodeENI resource\n    NodeENI-\u003e\u003eController: Watch event\n    Controller-\u003e\u003eAWS: Detach ENI\n    AWS--\u003e\u003eController: ENI detached\n    Controller-\u003e\u003eAWS: Delete ENI\n    AWS--\u003e\u003eController: ENI deleted\n    Controller-\u003e\u003eNodeENI: Remove finalizer\n```\n\n## Prerequisites\n\nBefore deploying the AWS Multi-ENI Controller, ensure you have:\n\n- Kubernetes cluster running on AWS (e.g., EKS)\n- kubectl configured to access your cluster\n- Helm 3.0+ (for Helm installation)\n- IAM permissions for EC2 ENI operations\n- **Amazon Linux 2023 Support**: The controller includes full IMDSv2 support for AL2023 nodes\n\n### Required IAM Permissions\n\nThe controller requires the following IAM permissions:\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"ec2:CreateNetworkInterface\",\n        \"ec2:DeleteNetworkInterface\",\n        \"ec2:DetachNetworkInterface\",\n        \"ec2:AttachNetworkInterface\",\n        \"ec2:DescribeInstances\",\n        \"ec2:DescribeNetworkInterfaces\",\n        \"ec2:DescribeSubnets\",\n        \"ec2:DescribeSecurityGroups\",\n        \"ec2:ModifyNetworkInterfaceAttribute\"\n      ],\n      \"Resource\": \"*\"\n    }\n  ]\n}\n```\n\n### IMDSv2 Support\n\nThe AWS Multi-ENI Controller includes comprehensive support for **Instance Metadata Service Version 2 (IMDSv2)**, ensuring compatibility with both Amazon Linux 2 and Amazon Linux 2023 nodes:\n\n- **Amazon Linux 2023**: Full support for nodes with `HttpTokens: required` (IMDSv2 enforcement)\n- **Amazon Linux 2**: Backward compatibility with both `HttpTokens: optional` and `HttpTokens: required`\n- **Automatic Hop Limit Configuration**: Automatically configures EC2 instance metadata hop limit for containerized environments\n- **Strict IMDSv2 Enforcement**: Configurable IMDSv1 fallback prevention for enhanced security\n- **Timeout \u0026 Retry**: Optimized timeout and retry settings for reliable credential retrieval\n\nFor detailed information about IMDSv2 implementation and automatic configuration, see [IMDSv2 Support Documentation](docs/imdsv2-support.md).\n\n## Installation\n\n### Option 1: Install with Helm (Recommended)\n\n```bash\n# Install the latest version\nhelm install aws-multi-eni oci://ghcr.io/johnlam90/charts/aws-multi-eni-controller --version 1.3.5 \\\n  --namespace eni-controller-system --create-namespace\n\n# With custom values\nhelm install aws-multi-eni oci://ghcr.io/johnlam90/charts/aws-multi-eni-controller --version 1.3.5 \\\n  --namespace eni-controller-system --create-namespace \\\n  --set awsRegion=us-east-1 \\\n  --set nodeSelector.ng=multi-eni\n```\n\n\u003e **Important**: Always specify the `--namespace eni-controller-system` flag and the `--create-namespace` flag when installing the chart to ensure all resources are created in the correct namespace.\n\n### Option 2: Install with YAML Manifests\n\n```bash\n# Clone the repository\ngit clone https://github.com/johnlam90/aws-multi-eni-controller.git\ncd aws-multi-eni-controller\n\n# Apply the CRDs\nkubectl apply -f deploy/crds/networking.k8s.aws_nodeenis.yaml\n\n# Apply the controller manifests\nkubectl apply -f deploy/controller.yaml\nkubectl apply -f deploy/eni-manager.yaml\n```\n\n## Usage Examples\n\n### Basic NodeENI Resource\n\n```yaml\napiVersion: networking.k8s.aws/v1alpha1\nkind: NodeENI\nmetadata:\n  name: multus-eni-config\nspec:\n  nodeSelector:\n    ng: multi-eni\n  subnetID: subnet-0f59b4f14737be9ad  # Use your subnet ID\n  securityGroupIDs:\n  - sg-05da196f3314d4af8  # Use your security group ID\n  deviceIndex: 2\n  mtu: 9001  # Optional: Set MTU for jumbo frames\n  deleteOnTermination: true\n  description: \"Multus ENI for secondary network interfaces\"\n```\n\n### Label Your Nodes\n\n```bash\nkubectl label node your-node-name ng=multi-eni\n```\n\n### Verify ENI Creation\n\n```bash\nkubectl get nodeeni multus-eni-config -o yaml\n```\n\n### Using Subnet Names Instead of IDs\n\n```yaml\napiVersion: networking.k8s.aws/v1alpha1\nkind: NodeENI\nmetadata:\n  name: multus-eni-subnet-name\nspec:\n  nodeSelector:\n    ng: multi-eni\n  subnetName: my-subnet-name  # Subnet with this Name tag will be used\n  securityGroupIDs:\n  - sg-05da196f3314d4af8\n  deviceIndex: 2\n```\n\n### Multiple Subnets Configuration\n\n```yaml\napiVersion: networking.k8s.aws/v1alpha1\nkind: NodeENI\nmetadata:\n  name: multi-subnet-nodeeni\nspec:\n  nodeSelector:\n    ng: multi-eni\n  subnetNames:\n  - multus-test-subnet-1\n  - multus-test-subnet-2\n  securityGroupNames:\n  - multus-test-sg\n  deviceIndex: 2\n  mtu: 9001\n  deleteOnTermination: true\n```\n\n## Configuration Options\n\n### MTU Configuration\n\nThe controller supports configuring custom MTU values for ENIs, which is useful for enabling jumbo frames (9001 bytes) or other specialized network configurations:\n\n```yaml\napiVersion: networking.k8s.aws/v1alpha1\nkind: NodeENI\nmetadata:\n  name: jumbo-frames-eni\nspec:\n  nodeSelector:\n    ng: multi-eni\n  subnetID: subnet-0f59b4f14737be9ad\n  securityGroupIDs:\n  - sg-05da196f3314d4af8\n  deviceIndex: 2\n  mtu: 9001  # Set MTU to 9001 for jumbo frames\n```\n\n### DPDK Integration\n\nFor high-performance networking applications, the controller supports binding interfaces to DPDK drivers:\n\n```yaml\napiVersion: networking.k8s.aws/v1alpha1\nkind: NodeENI\nmetadata:\n  name: dpdk-enabled-eni\nspec:\n  nodeSelector:\n    ng: multi-eni\n  subnetID: subnet-0f59b4f14737be9ad\n  securityGroupIDs:\n  - sg-05da196f3314d4af8\n  deviceIndex: 2\n  dpdkEnabled: true  # Enable DPDK binding\n```\n\n### Controller Concurrency\n\nControl how many NodeENI resources can be reconciled in parallel:\n\n```yaml\n# In Helm values.yaml\ncontroller:\n  maxConcurrentReconciles: 10  # Default: 5\n  maxConcurrentENICleanup: 5   # Default: 3\n```\n\n## Multus CNI Integration\n\nThe AWS Multi-ENI Controller was **specifically designed** to solve the challenges of using [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) in AWS environments. This section provides comprehensive guidance on integrating these two technologies.\n\n### Understanding Multus CNI and AWS Multi-ENI Controller\n\n### How They Work Together\n\n```mermaid\nflowchart TB\n    subgraph AWS[\"AWS Cloud\"]\n        VPC[\"VPC\"]\n        Subnets[\"Multiple Subnets\"]\n        ENIs[\"Elastic Network Interfaces\"]\n    end\n\n    subgraph K8S[\"Kubernetes Cluster\"]\n        subgraph Node[\"Worker Node\"]\n            kubelet[\"Kubelet\"]\n            CNI[\"Container Network Interface\"]\n            Multus[\"Multus CNI Plugin\"]\n            Pods[\"Pods with Multiple Interfaces\"]\n            Interfaces[\"eth0, eth1, eth2...\"]\n        end\n\n        subgraph Control[\"Control Plane\"]\n            API[\"Kubernetes API\"]\n            Controller[\"AWS Multi-ENI Controller\"]\n            NAD[\"NetworkAttachmentDefinition\"]\n            NodeENI[\"NodeENI Resources\"]\n        end\n\n        subgraph DaemonSet[\"ENI Manager DaemonSet\"]\n            Manager[\"ENI Manager\"]\n        end\n    end\n\n    %% Connections\n    Controller --\u003e|\"Creates/Attaches\"| ENIs\n    ENIs --\u003e|\"Attached to\"| Node\n    Manager --\u003e|\"Configures\"| Interfaces\n    Multus --\u003e|\"Uses\"| Interfaces\n    Multus --\u003e|\"References\"| NAD\n    Pods --\u003e|\"Uses\"| Multus\n    NodeENI --\u003e|\"Defines\"| Controller\n    NAD --\u003e|\"References\"| Interfaces\n\n    %% Styling with improved contrast\n    classDef aws fill:#FF9900,stroke:#232F3E,color:#000000,stroke-width:2px;\n    classDef k8s fill:#326CE5,stroke:#0B2161,color:#FFFFFF,stroke-width:2px;\n    classDef controller fill:#00A896,stroke:#004D40,color:#FFFFFF,stroke-width:2px;\n    classDef node fill:#F8F8F8,stroke:#2E2E2E,color:#000000,stroke-width:2px;\n    classDef interface fill:#E1F5FE,stroke:#0277BD,color:#000000,stroke-width:2px;\n    classDef resource fill:#E8F5E9,stroke:#2E7D32,color:#000000,stroke-width:2px;\n\n    class AWS,VPC,Subnets aws;\n    class K8S,Control,DaemonSet k8s;\n    class Controller,Manager controller;\n    class Node,kubelet,CNI,Multus,Pods node;\n    class Interfaces,ENIs interface;\n    class NodeENI,NAD,API resource;\n```\n\n1. **AWS Multi-ENI Controller** watches for NodeENI resources and creates/attaches ENIs to nodes\n2. **ENI Manager DaemonSet** configures the network interfaces on the nodes\n3. **Multus CNI** uses these interfaces to provide additional networks to pods\n4. **NetworkAttachmentDefinition** resources define how pods connect to these additional networks\n\n### Deployment Sequence\n\n1. Deploy the AWS Multi-ENI Controller\n2. Create NodeENI resources to provision ENIs on nodes\n3. Deploy Multus CNI\n4. Create NetworkAttachmentDefinition resources referencing the ENIs\n5. Deploy pods that use the additional networks\n\n## Example Configurations\n\n### 1. Basic Multus Integration\n\nFirst, create a NodeENI resource to provision an ENI:\n\n```yaml\napiVersion: networking.k8s.aws/v1alpha1\nkind: NodeENI\nmetadata:\n  name: multus-eni-config\nspec:\n  nodeSelector:\n    ng: multi-eni\n  subnetID: subnet-0f59b4f14737be9ad\n  securityGroupIDs:\n  - sg-05da196f3314d4af8\n  deviceIndex: 2  # This will create eth2 on the node\n  mtu: 9001\n  deleteOnTermination: true\n  description: \"Multus ENI for secondary network interfaces\"\n```\n\nThen, create a NetworkAttachmentDefinition that uses this interface:\n\n```yaml\napiVersion: k8s.cni.cncf.io/v1\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: secondary-network\nspec:\n  config: '{\n    \"cniVersion\": \"0.3.1\",\n    \"type\": \"ipvlan\",\n    \"master\": \"eth2\",\n    \"mode\": \"l2\",\n    \"ipam\": {\n      \"type\": \"host-local\",\n      \"subnet\": \"192.168.1.0/24\",\n      \"rangeStart\": \"192.168.1.200\",\n      \"rangeEnd\": \"192.168.1.250\"\n    }\n  }'\n```\n\nFinally, deploy pods that use this network:\n\n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  name: multinet-pod\n  annotations:\n    k8s.v1.cni.cncf.io/networks: secondary-network\nspec:\n  containers:\n  - name: multinet-container\n    image: nginx\n    resources:\n      requests:\n        cpu: 100m\n        memory: 128Mi\n```\n\n### 2. Multi-Subnet Configuration\n\nFor applications that need to connect to multiple subnets:\n\n```yaml\n# First, create NodeENI resources for different subnets\napiVersion: networking.k8s.aws/v1alpha1\nkind: NodeENI\nmetadata:\n  name: multus-eni-subnet1\nspec:\n  nodeSelector:\n    ng: multi-eni\n  subnetID: subnet-0f59b4f14737be9ad  # First subnet\n  securityGroupIDs:\n  - sg-05da196f3314d4af8\n  deviceIndex: 2  # This will create eth2\n  mtu: 9001\n---\napiVersion: networking.k8s.aws/v1alpha1\nkind: NodeENI\nmetadata:\n  name: multus-eni-subnet2\nspec:\n  nodeSelector:\n    ng: multi-eni\n  subnetID: subnet-abcdef1234567890  # Second subnet\n  securityGroupIDs:\n  - sg-05da196f3314d4af8\n  deviceIndex: 3  # This will create eth3\n  mtu: 9001\n```\n\nThen create NetworkAttachmentDefinitions for each subnet:\n\n```yaml\napiVersion: k8s.cni.cncf.io/v1\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: subnet1-network\nspec:\n  config: '{\n    \"cniVersion\": \"0.3.1\",\n    \"type\": \"ipvlan\",\n    \"master\": \"eth2\",\n    \"mode\": \"l2\",\n    \"ipam\": {\n      \"type\": \"host-local\",\n      \"subnet\": \"10.1.0.0/24\",\n      \"rangeStart\": \"10.1.0.100\",\n      \"rangeEnd\": \"10.1.0.200\"\n    }\n  }'\n---\napiVersion: k8s.cni.cncf.io/v1\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: subnet2-network\nspec:\n  config: '{\n    \"cniVersion\": \"0.3.1\",\n    \"type\": \"ipvlan\",\n    \"master\": \"eth3\",\n    \"mode\": \"l2\",\n    \"ipam\": {\n      \"type\": \"host-local\",\n      \"subnet\": \"10.2.0.0/24\",\n      \"rangeStart\": \"10.2.0.100\",\n      \"rangeEnd\": \"10.2.0.200\"\n    }\n  }'\n```\n\nDeploy a pod that connects to both networks:\n\n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  name: multi-subnet-pod\n  annotations:\n    k8s.v1.cni.cncf.io/networks: subnet1-network,subnet2-network\nspec:\n  containers:\n  - name: multi-subnet-container\n    image: nginx\n```\n\n### 3. DPDK Integration with Multus\n\nFor high-performance networking applications:\n\n```yaml\n# Create a NodeENI with DPDK enabled\napiVersion: networking.k8s.aws/v1alpha1\nkind: NodeENI\nmetadata:\n  name: dpdk-eni\nspec:\n  nodeSelector:\n    ng: multi-eni\n  subnetID: subnet-0f59b4f14737be9ad\n  securityGroupIDs:\n  - sg-05da196f3314d4af8\n  deviceIndex: 2\n  dpdkEnabled: true\n  dpdkDriver: \"vfio-pci\"\n  dpdkResourceName: \"intel.com/intel_sriov_netdevice\"\n```\n\nCreate a NetworkAttachmentDefinition for DPDK:\n\n```yaml\napiVersion: k8s.cni.cncf.io/v1\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: dpdk-network\nspec:\n  config: '{\n    \"cniVersion\": \"0.3.1\",\n    \"type\": \"host-device\",\n    \"device\": \"0000:00:06.0\",\n    \"vlan\": 1000,\n    \"ipam\": {\n      \"type\": \"host-local\",\n      \"subnet\": \"192.168.1.0/24\",\n      \"rangeStart\": \"192.168.1.200\",\n      \"rangeEnd\": \"192.168.1.250\"\n    }\n  }'\n```\n\nDeploy a pod that uses DPDK:\n\n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  name: dpdk-pod\n  annotations:\n    k8s.v1.cni.cncf.io/networks: dpdk-network\nspec:\n  containers:\n  - name: dpdk-container\n    image: dpdk-app:latest\n    resources:\n      limits:\n        intel.com/intel_sriov_netdevice: 1\n```\n\n### 4. Real-World Use Cases\n\n#### Network Isolation for Multi-Tenant Applications\n\n```yaml\n# Create NodeENI resources for tenant-specific subnets\napiVersion: networking.k8s.aws/v1alpha1\nkind: NodeENI\nmetadata:\n  name: tenant-a-eni\nspec:\n  nodeSelector:\n    ng: multi-tenant\n  subnetID: subnet-tenant-a\n  securityGroupIDs:\n  - sg-tenant-a\n  deviceIndex: 2\n---\napiVersion: networking.k8s.aws/v1alpha1\nkind: NodeENI\nmetadata:\n  name: tenant-b-eni\nspec:\n  nodeSelector:\n    ng: multi-tenant\n  subnetID: subnet-tenant-b\n  securityGroupIDs:\n  - sg-tenant-b\n  deviceIndex: 3\n```\n\nCreate NetworkAttachmentDefinitions for each tenant:\n\n```yaml\napiVersion: k8s.cni.cncf.io/v1\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: tenant-a-network\n  namespace: tenant-a\nspec:\n  config: '{\n    \"cniVersion\": \"0.3.1\",\n    \"type\": \"ipvlan\",\n    \"master\": \"eth2\",\n    \"mode\": \"l2\",\n    \"ipam\": {\n      \"type\": \"host-local\",\n      \"subnet\": \"10.10.0.0/24\"\n    }\n  }'\n---\napiVersion: k8s.cni.cncf.io/v1\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: tenant-b-network\n  namespace: tenant-b\nspec:\n  config: '{\n    \"cniVersion\": \"0.3.1\",\n    \"type\": \"ipvlan\",\n    \"master\": \"eth3\",\n    \"mode\": \"l2\",\n    \"ipam\": {\n      \"type\": \"host-local\",\n      \"subnet\": \"10.20.0.0/24\"\n    }\n  }'\n```\n\n#### Network Security Appliances\n\nFor deploying network security appliances that need to inspect traffic:\n\n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  name: network-firewall\n  annotations:\n    k8s.v1.cni.cncf.io/networks: ingress-network,egress-network\nspec:\n  containers:\n  - name: firewall\n    image: network-firewall:latest\n    securityContext:\n      capabilities:\n        add: [\"NET_ADMIN\"]\n```\n\n### 5. Best Practices for Multus CNI with AWS Multi-ENI Controller\n\n#### Device Index Management\n\nMaintain consistent device indices across your cluster:\n\n```yaml\n# Standard device index allocation\n# eth0: Primary ENI (managed by AWS)\n# eth1: Reserved for AWS CNI (if using)\n# eth2: First Multus network\n# eth3: Second Multus network\n# eth4: Third Multus network\n```\n\n#### Network Attachment Definition Naming Conventions\n\nUse a consistent naming convention for NetworkAttachmentDefinitions:\n\n```yaml\napiVersion: k8s.cni.cncf.io/v1\nkind: NetworkAttachmentDefinition\nmetadata:\n  # Format: \u003cpurpose\u003e-\u003csubnet\u003e-\u003cindex\u003e\n  name: app-subnet1-eth2\nspec:\n  config: '{ ... }'\n```\n\n#### Resource Requests for Multi-Network Pods\n\nAlways specify resource requests for pods with multiple networks:\n\n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  name: multi-net-pod\n  annotations:\n    k8s.v1.cni.cncf.io/networks: network1,network2\nspec:\n  containers:\n  - name: app\n    image: app:latest\n    resources:\n      requests:\n        cpu: 200m\n        memory: 256Mi\n      limits:\n        cpu: 500m\n        memory: 512Mi\n```\n\n#### Monitoring ENI Attachments\n\nCreate a monitoring solution to track ENI attachments:\n\n```bash\n# Example script to monitor ENI status\nkubectl get nodeeni -o custom-columns=NAME:.metadata.name,NODES:.status.attachments[*].nodeID,ENIs:.status.attachments[*].eniID,STATUS:.status.attachments[*].status\n```\n\n## Troubleshooting\n\n### Common Issues with AWS Multi-ENI Controller\n\n1. **ENI not being created**:\n   - Check controller logs: `kubectl logs -n eni-controller-system deployment/eni-controller`\n   - Verify node labels: `kubectl get nodes --show-labels | grep multi-eni`\n   - Check AWS permissions for ENI creation\n\n2. **ENI not being deleted**:\n   - Check finalizers on NodeENI: `kubectl get nodeeni -o yaml`\n   - Verify AWS permissions for ENI deletion\n\n3. **Interface not coming up**:\n   - Check ENI Manager logs: `kubectl logs -n eni-controller-system daemonset/eni-manager`\n   - Verify ENI Manager is running on the node\n\n4. **MTU not being applied**:\n   - Ensure MTU is set in the NodeENI resource\n   - Check ENI Manager logs for MTU configuration issues\n\n### Multus CNI Integration Issues\n\n1. **Pods can't access secondary networks**:\n   - Verify the NetworkAttachmentDefinition references the correct interface name\n   - Check Multus logs: `kubectl logs -n kube-system daemonset/kube-multus-ds`\n   - Verify the interface is up on the node: `ip link show eth2`\n   - Check pod annotations are correct: `kubectl get pod \u003cpod-name\u003e -o yaml | grep annotations -A5`\n\n2. **Wrong IP assignment on secondary networks**:\n   - Check IPAM configuration in the NetworkAttachmentDefinition\n   - Verify subnet configuration matches the actual ENI subnet\n   - Check for IP conflicts with other pods\n\n3. **DPDK binding issues**:\n   - Verify DPDK kernel modules are loaded: `lsmod | grep vfio`\n   - Check SRIOV device plugin is running: `kubectl get pods -n kube-system | grep sriov`\n   - Verify PCI address is correct: `ls -l /sys/bus/pci/devices/`\n\n4. **Multus CNI and AWS Multi-ENI Controller version compatibility**:\n   - Ensure Multus CNI version is compatible (v3.7+ recommended)\n   - Check for any warnings in Multus logs about interface detection\n\n### Debugging Commands\n\n```bash\n# Check NodeENI status\nkubectl get nodeeni -o wide\n\n# Check NetworkAttachmentDefinition\nkubectl get net-attach-def -A\n\n# Check Multus logs\nkubectl logs -n kube-system daemonset/kube-multus-ds\n\n# Check ENI Manager logs\nkubectl logs -n eni-controller-system daemonset/eni-manager\n\n# Check interfaces on a node\nkubectl debug node/\u003cnode-name\u003e -it --image=busybox -- ip addr\n\n# Check pod network interfaces\nkubectl exec -it \u003cpod-name\u003e -- ip addr\n\n# Verify DPDK binding status\nkubectl exec -it \u003cpod-name\u003e -- dpdk-devbind.py --status\n```\n\n### Common Error Messages and Solutions\n\n| Error Message | Possible Cause | Solution |\n|---------------|----------------|----------|\n| `failed to find plugin \"ipvlan\" in path` | Multus CNI plugin not installed correctly | Verify Multus CNI installation and CNI plugins |\n| `error getting interface \"eth2\": no such network interface` | ENI not attached or interface not up | Check ENI Manager logs and node interface status |\n| `failed to allocate for range 0: no IP addresses available in range set` | IPAM exhausted or misconfigured | Adjust IPAM range or check for leaked IPs |\n| `cannot allocate resource intel.com/intel_sriov_netdevice` | SRIOV device plugin issue or resource exhausted | Check SRIOV device plugin status and resource allocation |\n\n## Complete Multus CNI Integration Workflow\n\nThe following diagram illustrates the complete workflow of AWS Multi-ENI Controller with Multus CNI:\n\n```mermaid\nsequenceDiagram\n    participant User as User/Operator\n    participant K8sAPI as Kubernetes API\n    participant Controller as AWS Multi-ENI Controller\n    participant AWS as AWS EC2 API\n    participant ENIManager as ENI Manager DaemonSet\n    participant Node as Kubernetes Node\n    participant Multus as Multus CNI\n    participant Pod as Pod with Multiple Networks\n\n    User-\u003e\u003eK8sAPI: 1. Deploy AWS Multi-ENI Controller\n    User-\u003e\u003eK8sAPI: 2. Deploy Multus CNI\n    User-\u003e\u003eK8sAPI: 3. Create NodeENI resource\n    K8sAPI-\u003e\u003eController: 4. NodeENI watch event\n    Controller-\u003e\u003eAWS: 5. Create ENI\n    AWS--\u003e\u003eController: 6. ENI created\n    Controller-\u003e\u003eAWS: 7. Attach ENI to node\n    AWS--\u003e\u003eController: 8. ENI attached\n    Controller-\u003e\u003eK8sAPI: 9. Update NodeENI status\n    ENIManager-\u003e\u003eNode: 10. Detect new ENI\n    ENIManager-\u003e\u003eNode: 11. Configure interface (eth2)\n    User-\u003e\u003eK8sAPI: 12. Create NetworkAttachmentDefinition\n    User-\u003e\u003eK8sAPI: 13. Deploy pod with network annotation\n    K8sAPI-\u003e\u003eNode: 14. Schedule pod\n    Node-\u003e\u003eMultus: 15. Setup pod networking\n    Multus-\u003e\u003ePod: 16. Configure primary interface (eth0)\n    Multus-\u003e\u003ePod: 17. Configure secondary interface (net1)\n    Pod--\u003e\u003eUser: 18. Pod running with multiple networks\n\n    Note over User,Pod: Complete Integration Flow\n```\n\n### End-to-End Integration Steps\n\n1. **Deploy AWS Multi-ENI Controller**: Install the controller in your cluster\n2. **Deploy Multus CNI**: Install Multus CNI in your cluster\n3. **Create NodeENI Resources**: Define which nodes should get ENIs and from which subnets\n4. **Controller Creates/Attaches ENIs**: The controller automatically creates and attaches ENIs to nodes\n5. **ENI Manager Configures Interfaces**: The ENI Manager brings up the interfaces on the nodes\n6. **Create NetworkAttachmentDefinitions**: Define how pods should use these interfaces\n7. **Deploy Pods with Network Annotations**: Specify which additional networks pods should use\n8. **Multus Configures Pod Networking**: Multus sets up additional interfaces in the pods\n\n## Documentation\n\nFor more detailed information, please refer to the following documentation:\n\n- [Architecture](docs/architecture.md) - Detailed architecture and workflow\n- [Deployment](docs/deployment.md) - Comprehensive deployment options\n- [Configuration](docs/configuration.md) - Advanced configuration options\n- [Multus CNI Integration](docs/multus-integration.md) - Detailed guide for Multus CNI integration\n- [Troubleshooting](docs/troubleshooting.md) - Detailed troubleshooting guide\n- [DPDK Integration](docs/dpdk.md) - DPDK setup and configuration\n\n## Contributing\n\nContributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for details.\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnlam90%2Faws-multi-eni-controller","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjohnlam90%2Faws-multi-eni-controller","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnlam90%2Faws-multi-eni-controller/lists"}