Skip to content

Conversation

@neddp
Copy link
Member

@neddp neddp commented Dec 12, 2025

Problem

When deploying BOSH instances with multiple network interfaces (using different nic_group configurations) and an Elastic IP, the CPI fails during create_vm with:

CPI error 'Unknown' with message 'There are multiple interfaces attached to instance 'i-xxxxx'. 
Please specify an interface ID for the operation instead.'

Root Cause

The associate_address AWS API call was using the instance_id parameter, which according to AWS documentation only works when "the instance must have exactly one attached network interface."
This happens in the scenario where we use multiple NICs (nic_groups) and VIP address.

Solution

Modified network_configurator.rb to use the network_interface_id parameter (pointing to the primary network interface with device_index 0) instead of instance_id when associating Elastic IPs. This approach works for both single and multiple network interface scenarios.

AWS Documentation Reference

From the AWS EC2 AssociateAddress API Documentation:

InstanceId parameter:

"The ID of the instance. The instance must have exactly one attached network interface. You can specify either the instance ID or the network interface ID, but not both."

NetworkInterfaceId parameter:

"The ID of the network interface. If the instance has more than one network interface, you must specify a network interface ID.

You can specify either the instance ID or the network interface ID, but not both."

Changes

Code Changes

  • Updated configure_vip method in lib/cloud/aws/network_configurator.rb:
    • Added describe_instances call to retrieve instance network interfaces
    • Finds the primary network interface (device_index 0)
    • Uses network_interface_id parameter instead of instance_id for associate_address
    • Works correctly for instances with single or multiple network interfaces

Test Coverage

Unit Tests (spec/unit/network_configurator_spec.rb)

Added 3 new test cases to validate multi-NIC VIP scenarios:

  1. Single NIC with VIP - Validates that the primary NIC (device_index 0) is used for Elastic IP association
  2. Two NICs with VIP (secondary first) - Ensures the primary NIC is correctly identified when it's not first in the array
  3. Two NICs with VIP (primary first) - Validates finding the primary NIC regardless of array order

Test Helpers Added:

  • create_nic_mock - Creates network interface mocks with proper device_index
  • mock_describe_instances - Mocks the AWS describe_instances response structure
  • setup_vip_mocks - Sets up common VIP association mocks

All unit tests pass.

Integration Tests (spec/integration/manual_network_spec.rb)

Added comprehensive integration test: "creates VM with multiple NICs and associates Elastic IP to primary NIC"

Test validates:

  1. VM is created with 2 network interfaces (using nic_group 0 and 1)
  2. Both NICs are properly attached (device_index 0 and 1)
  3. Elastic IP is associated using network_interface_id (not instance_id)
  4. Elastic IP is specifically associated with the primary NIC (device_index 0)
  5. The Elastic IP is accessible via the primary NIC's association

Test setup:

  • Uses same subnet for both NICs (AWS allows multiple NICs in same subnet)
  • Thread-safe IP allocation prevents conflicts
  • Works with existing CI environment variables (no new environment variables required)

# API call will fail in that case.

# AWS requires network_interface_id when the instance has multiple network interfaces.
network_interfaces = ec2.client.describe_instances(instance_ids: [instance.id]).reservations.first.instances.first.network_interfaces
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a scenario where the describe_instances call could fail? for example a timeout or 503 error? Should it maybe also be in a retry block?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

2 participants