Bug: `Reaper.delete_instance()` raises 409 and keeps the dead reaper when ryuk was killed from outside

[PHcz]

Describe the bug

When ryuk has been killed from outside (docker kill, an OOM kill, a Docker Desktop restart), Reaper.delete_instance() raises APIError 409 and leaves Reaper holding the dead reaper. After that, Reaper.get_instance() returns the dead instance instead of creating a new one.

The reaper container is created with auto_remove=True, so Docker starts removing it as soon as it dies. delete_instance() calls Reaper._container.stop(), whose remove(force=True) arrives while that removal is still running, and Docker answers:

409 Client Error: Conflict ... removal of container <id> is already in progress

delete_instance() suppresses only docker.errors.NotFound, so the 409 propagates. The socket has already been closed and Reaper._socket set to None, but the exception skips the two resets after it: Reaper._container and Reaper._instance still point at the dead reaper. get_instance() only creates a reaper when _instance is None, so it keeps returning the dead one, and the session has no reaper and no way back to one.

How we got there: a pytest session whose reaper was removed mid-run, with the session's containers still running. After ryuk's ACK the reaper link carries no data, so a lost reaper shows up on Reaper._socket as end-of-file within milliseconds. We check for that before each test and replace the reaper with delete_instance() + get_instance(). The replacement is where this surfaced: 30 of 40 SIGKILLs in one harness, and 10 of 10 in the script below. delete_instance() also runs from atexit, and it hits the same 409 there when the process exits during the removal. The script below does, and ends with Exception ignored in atexit callback.

To Reproduce

"""Reaper.delete_instance() after ryuk was killed from outside: 409, and stale state."""

import selectors
import uuid

import docker
import docker.errors
import testcontainers.core.container as tcc
from testcontainers.core.config import testcontainers_config as c
from testcontainers.core.container import DockerContainer, Reaper

# Publish ryuk's 8080 on 127.0.0.1 only (host port still kernel-assigned), just to keep the
# experiment's Docker-socket-holding ryuk off the LAN. Not part of the issue.
_orig_with_exposed_ports = DockerContainer.with_exposed_ports


def _loopback(self, *ports):
    out = _orig_with_exposed_ports(self, *ports)
    if self.image == c.ryuk_image:
        for p in ports:
            self.ports[str(p)] = ("127.0.0.1", None)
    return out


DockerContainer.with_exposed_ports = _loopback

tcc.SESSION_ID = f"repro-{uuid.uuid4()}"
reaper = Reaper.get_instance()
ryuk_id = Reaper._container.get_wrapped_container().id

client = docker.from_env()
client.api.kill(ryuk_id)  # what `docker kill` or an OOM kill does from outside
with selectors.DefaultSelector() as sel:  # the kill shows on the reaper link as end-of-file
    sel.register(Reaper._socket, selectors.EVENT_READ)
    sel.select(5)

try:
    Reaper.delete_instance()
except docker.errors.APIError as exc:
    print("delete_instance() raised", exc.status_code, exc.explanation)
print("_socket:", Reaper._socket)
print("_instance is still the dead reaper:", Reaper._instance is reaper)
print("get_instance() returns the dead reaper:", Reaper.get_instance() is reaper)

Output (the first four lines were the same in 10 of 10 runs of a loop version):

delete_instance() raised 409 removal of container 4dfcbdd146b5512c26004bbf887100f12d8cb0e3e67dad883e902485659fa24a is already in progress
_socket: None
_instance is still the dead reaper: True
get_instance() returns the dead reaper: True
Exception ignored in atexit callback <bound method Reaper.delete_instance of <class 'testcontainers.core.container.Reaper'>>:
Traceback (most recent call last):
  ...
docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.55/containers/<id>?v=True&link=False&force=True: Conflict ("removal of container <id> is already in progress")

Suggested fix

Treat "removal already in progress" like "already gone", and reset the class state whatever stop() does. For example:

    @classmethod
    def delete_instance(cls) -> None:
        container = Reaper._container
        try:
            if Reaper._socket is not None:
                Reaper._socket.close()
            if container is not None and container._container is not None:
                try:
                    container.stop()
                except docker.errors.NotFound:
                    pass
                except docker.errors.APIError as exc:
                    if exc.status_code != 409:  # 409: auto_remove is already removing it
                        raise
        finally:
            Reaper._socket = None
            Reaper._container = None
            Reaper._instance = None

One more thing for anyone recreating a reaper right after this: the new container reuses the name testcontainers-ryuk-{SESSION_ID}, which is only free once the old container is fully removed. We measured the old container still listed up to 0.29 s after the kill, and still listed in 4 of 20 checks made with no wait at all. So we wait for its id to disappear before calling get_instance(). We saw no name conflict with that wait in place (80+ replacements), and did not test without it. #1093 covers a related name collision from another path.

Runtime environment

• testcontainers-python 4.15.0 (Reaper is unchanged on main), ryuk 0.8.1 (the default), docker-py 7.2.0, Python 3.13.13
• Docker Desktop 4.88.1, Engine 29.7.2 (API 1.55), linuxkit 7.0.12, linux/arm64, 12 CPUs
• macOS 27.0 (26A428), Darwin 27.0.0 arm64