Docker v28 update causes `failed to listen on TCP socket: address already in use`

[anoadragon453]

From May 5th, we started noticing failed to listen on TCP socket: address already in use errors in our Complement jobs on Synapse, for example: https://github.com/element-hq/synapse/actions/runs/14858385920/job/41717289741#step:7:1191

  2025/05/06 11:30:21 Sharing [SERVER_NAME=hs1 SYNAPSE_LOG_TESTING=1 SYNAPSE_COMPLEMENT_USE_WORKERS= SYNAPSE_COMPLEMENT_DATABASE=sqlite] host environment variables with container
  2025/05/06 11:30:22 fed.hs_with_application_service.hs1 : failed to deployBaseImage: Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint complement_fed.hs_with_application_service.hs1 (198c5615feb31d0f6a3fd3bb3cd990010b944a0daed9243503292e556aee3c4a): failed to listen on TCP socket: address already in use
  2025/05/06 11:30:22 ============================================
  
  
  2025/05/06 11:30:22 fed.hs_with_application_service.hs1 : Server logs:
  2025/05/06 11:30:22 ============== fed.hs_with_application_service.hs1 : END LOGS ==============
  
  
      federation_room_join_test.go:586: OldDeploy: Failed to construct blueprint: ConstructBlueprintIfNotExist(hs_with_application_service): failed to ConstructBlueprint: errors whilst constructing blueprint hs_with_application_service: [Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint complement_fed.hs_with_application_service.hs1 (198c5615feb31d0f6a3fd3bb3cd990010b944a0daed9243503292e556aee3c4a): failed to listen on TCP socket: address already in use]

This appears to have coincided with GitHub updating the ubuntu24.04 runner, which included an update to the Docker Client and Server components:

- Docker Client 26.1.3
- Docker Server 26.1.3
⬇️
- Docker Client 28.0.4
- Docker Server 28.0.4

We found moby/moby#49935 where a user was experiencing a similar networking-related issue in v28. This comment implies that this isn't considered a regression, and applications should update to account for the breaking change.

• Before moby 28.0.0, for a given compose config, interface names in containers were always assigned to the same network endpoints.
• In moby 28.0.0:
  • That changed unexpectedly, interface names became unpredictable by-default.
  • New option com.docker.network.endpoint.ifname was added, making it possible to explicitly assign an interface name.
  • New option GwPriority was added to make it possible to determine which network provides a container's default gateway.

[anoadragon453]

moby/moby#49518 might also be relevant?

[kegsay]

So it turns out #772 was the root cause here. On v26 this wasn't an issue, but they have clearly refactored the networking code in v28 which is why this now fails.

When this happens, we keep the container dangling (as it's ContainerStart which causes the error, not ContainerCreate) thus we subsequently see The container name "/complement_crypto_dirty_hs1" is already in use by container "ef141595c2188aa4d0ece3aa5f74fdb43a7ba276cf95a00076ad387881d7b24a". You have to remove (or rename) that container to be able to reuse that name.

I've ended up just doing PublishAllPorts as only homerunner used the port-binding IP override option which is what introduced this regression in the first place.