AWS Multi-ENI Controller for Kubernetes

A Kubernetes controller that automatically creates and attaches AWS Elastic Network Interfaces (ENIs) to nodes based on node labels. Primarily designed to simplify Multus CNI deployments on AWS, this controller enables attaching multiple network interfaces to pods without complex infrastructure templates. It's ideal for workloads that require multiple network interfaces, such as networking plugins, security tools, or specialized applications.

Overview

The AWS Multi-ENI Controller consists of two main components:

1. NodeENI Controller: Watches for NodeENI custom resources and nodes with matching labels. When a node matches the selector in a NodeENI resource, the controller creates an ENI in the specified subnet with the specified security groups and attaches it to the node at the specified device index.

2. ENI Manager: A DaemonSet that runs on nodes with matching labels and automatically brings up secondary interfaces when they're attached.

When a node no longer matches the selector or when the NodeENI resource is deleted, the controller automatically detaches and deletes the ENI, ensuring proper cleanup of AWS resources.

Features

• Multus CNI Integration: Simplifies deploying Multus CNI on AWS by automating ENI management
• Dynamic ENI Management: Automatically creates and attaches ENIs to nodes based on labels
• Multi-Subnet Support: Can attach ENIs from different subnets to the same or different nodes
• Subnet Flexibility: Supports both subnet IDs and subnet names (via AWS tags)
• Security Group Flexibility: Supports both security group IDs and names
• MTU Configuration: Configure custom MTU settings for ENIs (e.g., jumbo frames)
• Proper Cleanup: Uses finalizers to ensure ENIs are properly detached and deleted when no longer needed
• Parallel ENI Cleanup: Efficiently cleans up multiple ENIs in parallel for improved performance
• Configurable Concurrency: Control the number of concurrent operations for better scaling
• Cross-Distribution Compatibility: Works across all Linux distributions with different interface naming patterns
• AWS SDK v2 Integration: Uses the latest AWS SDK Go v2 for improved performance and features
• Optimized Image: Lightweight container image (22MB) for fast deployments
• Helm Support: Easy deployment with Helm charts and OCI registry
• Library Support: Can be used as a Go library for programmatic ENI management
• No Infrastructure Changes: Works with vanilla EKS/Kubernetes clusters without complex IaC templates

Quick Start

Prerequisites

• Kubernetes cluster running on AWS (e.g., EKS)
• kubectl configured to access your cluster
• Helm 3.0+ (for Helm installation)
• IAM permissions for EC2 ENI operations

Install with Helm (Recommended)

# Install the latest version
helm install aws-multi-eni oci://ghcr.io/johnlam90/charts/aws-multi-eni-controller --version 1.3.0 \
  --namespace eni-controller-system --create-namespace

# With custom values
helm install aws-multi-eni oci://ghcr.io/johnlam90/charts/aws-multi-eni-controller --version 1.3.0 \
  --namespace eni-controller-system --create-namespace \
  --set awsRegion=us-east-1 \
  --set nodeSelector.ng=multi-eni

Important: Always specify the --namespace eni-controller-system flag and the --create-namespace flag when installing the chart to ensure all resources are created in the correct namespace.

Create a NodeENI Resource

apiVersion: networking.k8s.aws/v1alpha1
kind: NodeENI
metadata:
  name: multus-eni-config
spec:
  nodeSelector:
    ng: multi-eni
  subnetID: subnet-0f59b4f14737be9ad  # Use your subnet ID
  securityGroupIDs:
  - sg-05da196f3314d4af8  # Use your security group ID
  deviceIndex: 2
  mtu: 9001  # Optional: Set MTU for jumbo frames
  deleteOnTermination: true
  description: "Multus ENI for secondary network interfaces"

Label Your Nodes

kubectl label node your-node-name ng=multi-eni

Verify ENI Creation

kubectl get nodeeni multus-eni-config -o yaml

Integration with Multus CNI

The AWS Multi-ENI Controller was primarily built to simplify the use of Multus CNI with AWS EC2 instances. Multus CNI enables attaching multiple network interfaces to Kubernetes pods, but it requires those interfaces to exist on the node first.

Why Use AWS Multi-ENI Controller with Multus

Traditional approaches to provisioning secondary ENIs for Multus have significant drawbacks:

• Complex Infrastructure as Code: Typically requires complex CloudFormation, CDK, Terraform, or Crossplane templates
• Static Configuration: ENIs are often statically defined at node creation time
• Difficult Maintenance: Updating or changing ENI configurations requires node replacement
• Manual Cleanup: ENIs may be left orphaned when nodes are terminated

The AWS Multi-ENI Controller solves these problems by:

1. Dynamic ENI Provisioning: Automatically creates and attaches ENIs to nodes based on labels
2. Declarative Configuration: Uses Kubernetes-native NodeENI resources
3. No Infrastructure Changes: Works with vanilla EKS/Kubernetes clusters
4. Automatic Cleanup: Properly detaches and deletes ENIs when no longer needed

How It Works with Multus

1. Install AWS Multi-ENI Controller: Deploy the controller to your cluster
2. Create NodeENI Resources: Define which nodes should get which ENIs
3. Install Multus CNI: Deploy Multus to your cluster
4. Create NetworkAttachmentDefinition: Configure Multus to use the secondary interfaces

Example NetworkAttachmentDefinition for Multus using a secondary interface:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: secondary-network
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "macvlan",
    "master": "eth2",
    "mode": "bridge",
    "ipam": {
      "type": "host-local",
      "subnet": "192.168.1.0/24",
      "rangeStart": "192.168.1.200",
      "rangeEnd": "192.168.1.250"
    }
  }'

Benefits of This Approach

• Simplified Operations: No need for complex IaC templates or custom AMIs
• Kubernetes-Native: Manage network interfaces using Kubernetes resources
• Flexible Deployment: Add or remove secondary interfaces without rebuilding nodes
• Multi-Subnet Support: Attach ENIs from different subnets to the same node
• Consistent Device Indices: Maintain predictable interface naming across nodes
• Automatic MTU Configuration: Configure jumbo frames or other MTU settings

Example Use Cases

• Network Isolation: Separate application traffic from control plane traffic
• Multi-VPC Connectivity: Connect pods to multiple VPCs
• Network Security Appliances: Deploy firewalls, IDS/IPS, or other security tools
• Specialized Networking: Support for SR-IOV, DPDK, or other high-performance networking
• Legacy Network Integration: Connect to non-Kubernetes networks

Configuration Options

Using Subnet Names Instead of IDs

apiVersion: networking.k8s.aws/v1alpha1
kind: NodeENI
metadata:
  name: multus-eni-subnet-name
spec:
  nodeSelector:
    ng: multi-eni
  subnetName: my-subnet-name  # Subnet with this Name tag will be used
  securityGroupIDs:
  - sg-05da196f3314d4af8
  deviceIndex: 2

Using Security Group Names Instead of IDs

apiVersion: networking.k8s.aws/v1alpha1
kind: NodeENI
metadata:
  name: multus-eni-sg-name
spec:
  nodeSelector:
    ng: multi-eni
  subnetID: subnet-0f59b4f14737be9ad
  securityGroupNames:
  - my-security-group  # Security group with this name will be used
  deviceIndex: 2

Multiple Subnets Configuration

apiVersion: networking.k8s.aws/v1alpha1
kind: NodeENI
metadata:
  name: multi-subnet-nodeeni
spec:
  nodeSelector:
    ng: multi-eni
  subnetNames:
  - multus-test-subnet-1
  - multus-test-subnet-2
  securityGroupNames:
  - multus-test-sg
  deviceIndex: 2
  mtu: 9001
  deleteOnTermination: true

MTU Configuration

The controller supports configuring custom MTU values for ENIs, which is useful for enabling jumbo frames (9001 bytes) or other specialized network configurations:

apiVersion: networking.k8s.aws/v1alpha1
kind: NodeENI
metadata:
  name: jumbo-frames-eni
spec:
  nodeSelector:
    ng: multi-eni
  subnetID: subnet-0f59b4f14737be9ad
  securityGroupIDs:
  - sg-05da196f3314d4af8
  deviceIndex: 2
  mtu: 9001  # Set MTU to 9001 for jumbo frames

Scaling and Performance

The controller includes configuration options to optimize performance in larger deployments:

Controller Concurrency

Control how many NodeENI resources can be reconciled in parallel:

# In Helm values.yaml
controller:
  maxConcurrentReconciles: 10  # Default: 5

ENI Cleanup Concurrency

Control how many ENIs can be cleaned up in parallel:

# In Helm values.yaml
controller:
  maxConcurrentENICleanup: 5  # Default: 3

Troubleshooting

Common Issues

1. ENI not being created:

   • Check controller logs: kubectl logs -n eni-controller-system deployment/eni-controller
   • Verify node labels: kubectl get nodes --show-labels | grep multi-eni
   • Check AWS permissions for ENI creation

2. ENI not being deleted:

   • Check finalizers on NodeENI: kubectl get nodeeni -o yaml
   • Verify AWS permissions for ENI deletion

3. Interface not coming up:

   • Check ENI Manager logs: kubectl logs -n eni-controller-system daemonset/eni-manager
   • Verify ENI Manager is running on the node

4. MTU not being applied:

   • Ensure MTU is set in the NodeENI resource
   • Check ENI Manager logs for MTU configuration issues

Using as a Library

The AWS Multi-ENI Controller can also be used as a Go library for programmatic ENI management:

go get github.com/johnlam90/aws-multi-eni-controller

Basic Example

package main

import (
    "context"
    "log"
    "time"

    "github.com/go-logr/logr"
    "github.com/go-logr/zapr"
    "github.com/johnlam90/aws-multi-eni-controller/pkg/lib"
    "go.uber.org/zap"
)

func main() {
    // Create a logger
    zapLog, _ := zap.NewDevelopment()
    logger := zapr.NewLogger(zapLog)

    // Create a context with timeout
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
    defer cancel()

    // Create an ENI manager
    eniManager, err := lib.NewENIManager(ctx, "us-east-1", logger)
    if err != nil {
        log.Fatalf("Failed to create ENI manager: %v", err)
    }

    // Create an ENI
    options := lib.ENIOptions{
        SubnetID:           "subnet-12345678",
        SecurityGroupIDs:   []string{"sg-12345678"},
        Description:        "Example ENI",
        DeviceIndex:        1,
        DeleteOnTermination: true,
    }

    eniID, err := eniManager.CreateENI(ctx, options)
    if err != nil {
        log.Fatalf("Failed to create ENI: %v", err)
    }

    log.Printf("Created ENI: %s", eniID)
}

For more details, see the Library Documentation.

Documentation

• Architecture - Detailed architecture and workflow
• Deployment - Comprehensive deployment options
• Configuration - Advanced configuration options
• Troubleshooting - Detailed troubleshooting guide
• Library Usage - Using as a Go library

Website

Visit our project website for more information, detailed documentation, and usage examples.

Architecture

The AWS Multi-ENI Controller follows the Kubernetes operator pattern with two main components:

1. NodeENI Controller: Manages the lifecycle of ENIs based on NodeENI resources
2. ENI Manager: Brings up secondary interfaces on nodes

ENI Lifecycle

The following diagram illustrates the complete lifecycle of an ENI managed by the controller:

The diagram shows the flow from NodeENI creation through ENI creation, attachment, configuration, and eventual cleanup when the NodeENI is deleted or updated.

For a code-based version of this diagram (using Mermaid.js), see ENI Lifecycle Diagram.

Component Architecture

The AWS Multi-ENI Controller consists of several components working together. To see a visual representation of the system architecture, check out the Architecture Diagram.

To understand the detailed relationships between the key components of the AWS Multi-ENI Controller, see the Class Diagram.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for details.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.