ROCm CI: prune build intermediates during make, retain final image tag

[jiridanek]

Parent: #2491

Idea

During make, podman accumulates dangling layers from multi-stage builds (ROCm pytorch is especially heavy). Clean up during the build without deleting the tagged output image.

Approach

• Add cleanup in the Makefile / podman build path after each stage or on build failure:
  • podman container prune -f
  • remove untagged/dangling images
  • explicitly retain ${OUTPUT_IMAGE} / the matrix job's final tag
• Optionally prune only if disk usage on storage path exceeds a threshold (coordinate with monitor_resources.py if ISSUE#3965: modernize resource monitoring and remove obsolete LVM overlay opendatahub-io/notebooks#3967 is backported)

Why not post-make prune

#2489 showed that podman system prune -af after make deletes the built image before testcontainers, image tests, OpenShift tests, and check-payload. Any cleanup must not touch the final artifact.

Scope

• Makefile image macro / podman build invocation
• ROCm pytorch targets first (rocm-jupyter-pytorch, rocm-runtime-pytorch)
• Generalize if pattern works for other large images

Acceptance criteria

• Peak disk during make is lower on ROCm pytorch jobs
• Post-build tests still find the local image (no ImageNotFound)
• Template test + push matrix ROCm amd64 legs pass

[jiridanek]

@coderabbitai They are mostly separate images consuming extra space, but with an important caveat: container storage deduplicates identical layers, so the true overhead is less than the sum of all their reported sizes.

Here is the evidence from your own output.

Same IMAGE ID = just an alias (no extra space)

Look at these three lines:

ghcr.io/.../build-cache  e4a935a0a952...  f8d69719d2f45  19.6GB
ghcr.io/.../build-cache  778b8418a652...  f8d69719d2f45  19.6GB
ghcr.io/.../workbench-images  jupyter-trustyai...  f8d69719d2f45  19.6GB

All three share the exact same IMAGE ID (f8d69719d2f45). These are just multiple names for the same image. They reference the exact same layers on disk. Deleting one tag does not free space; deleting the last tag frees the image.

Different IMAGE ID = distinct image (extra space)

Now look at these:

ghcr.io/.../build-cache  7f51fe66ba48...  c8c2bc7d531c8  11.3GB
ghcr.io/.../build-cache  6c99961b7dc5...  c9c3bd8ea609b  11.3GB
ghcr.io/.../build-cache  d3a1ac7fe0a5...  48fce85c59071  9.69GB

These have different IMAGE IDs from the final image (f8d69719d2f45) and from each other. They are actual separate images in storage. They are not just "names for layers inside the final image" — they are intermediate build results that Podman/Buildah committed to disk as cacheable images.

How much extra space?

Container storage is content-addressable. If a layer exists in both the final image and a cache image, it is stored once and referenced by both. So the "extra" space is not 19.6 + 11.3 + 11.3 + .... Instead, it is the sum of layers unique to each cache image.

From your list, you have roughly:

• 1 final image at ~19.6 GB
• ~50 build-cache images ranging from 1 GB to 19.6 GB
• Many cache images at ~3.16 GB, ~2.71 GB, ~2.62 GB, etc.

If you add up all the reported sizes, you get a huge number. But because of layer sharing, the real disk usage is much smaller. The only way to know the exact overhead is to inspect the layer store.

Bottom line

Question	Answer
Are they just aliases for final image layers?	No. Only the ones matching the final image's IMAGE ID are aliases. The rest are independent images.
Do they take extra space?	Yes. The ones with different IMAGE IDs are stored as separate images. But shared layers are deduplicated, so the overhead is less than the sum of reported sizes.
Can I safely delete them?	Yes, but you lose build cache. If you delete the build-cache images, future builds will have to re-execute RUN steps instead of reusing layers. The final image continues to work.

If you want to reclaim space without losing the final image, podman image prune or podman rmi targeting the build-cache repo will remove the intermediate cache images while preserving the final workbench-images image.

[jiridanek]

Status: ⚪ Superseded — recommend closing without implementing

No prune-during-make logic exists in the Makefile or build-notebooks-TEMPLATE.yaml on rhoai-2.25 — this idea was never implemented literally. However, the ROCm pytorch ENOSPC problem this issue was meant to mitigate was actually fixed by a different, more targeted approach in #2647 / tracked in #2493: folding de-vendor-torch.sh into the same USER 1001 heredoc as uv pip install, so the giant vendored ROCm blobs never get committed as their own layer in the first place. Push builds for rocm-jupyter-pytorch and rocm-runtime-pytorch now complete at ~58–59% peak disk (previously 100% / ENOSPC).

Since the root cause is now addressed by avoiding the intermediate layers rather than pruning them after the fact, and the general concern (post-make prune deleting the tagged output image, per #2489) is a non-issue as long as no such prune step exists — I'd recommend closing this as not needed, revisiting only if ENOSPC resurfaces on other large images (CUDA, etc.) where the same-RUN de-vendor trick doesn't apply.