Backport main #3967: drop LVM overlay, use /mnt/containers/storage

[jiridanek]

Parent: #2491

Source

opendatahub-io/notebooks#3967 (ISSUE#3965), merged on main 2026-06-29. Not on rhoai-2.25 or rhds/cherry-pick-qol-rhoai-2.25.

What changes on main

• Delete ci/cached-builds/gha_lvm_overlay.sh
• Set graphroot = /mnt/containers/storage in ci/cached-builds/storage.conf
• Prepare /mnt/containers/storage in build-notebooks-TEMPLATE.yaml instead of LVM mount step
• Add scripts/monitor_resources.py (disk/memory on /, /mnt, storage path)
• Simplify free-up-disk-space composite action

rhoai-2.25 today

• Still runs gha_lvm_overlay.sh + home-dir storage (~/.local/share/containers/storage)
• Comment in overlay script notes root_reserve_mb=2048 was insufficient for CUDA builds

Work

1. Cherry-pick or manually port ISSUE#3965: modernize resource monitoring and remove obsolete LVM overlay opendatahub-io/notebooks#3967 onto rhoai-2.25
2. Resolve conflicts with RHDS-specific workflow differences (subscription jobs, AIPCC login, PRODUCT)
3. Run test-build-notebooks-template.yaml on jupyter-minimal and a ROCm target
4. Validate full push matrix ROCm amd64 legs

Acceptance criteria

• No gha_lvm_overlay.sh step in build-notebooks-TEMPLATE.yaml
• Podman uses /mnt/containers/storage on GHA runners
• Resource monitoring available in workflow logs
• No regression on arm64 (note: arm64 runners may lack /mnt scratch — verify main's handling)

[jiridanek]

Status: ✅ Resolved

Verified against current rhoai-2.25 HEAD (166024004fd):

• ci/cached-builds/gha_lvm_overlay.sh — absent (removed)
• ci/cached-builds/storage.conf — graphroot = "/mnt/containers/storage" ✅ (matches main)
• scripts/monitor_resources.py — present
• Workflows migrated to ubuntu-26.04 / ubuntu-26.04-arm

Landed in #2647 (RHAIENG-6546, merged 2026-07-30), which cherry-picked ISSUE#3965 (e4cf912dfb), backported ci/logging_config.py so monitor_resources.py stops erroring, dropped codeserver swap, and migrated remaining ubuntu-24.04/ubuntu-latest workflows to ubuntu-26.04 (this last part overlaps with #2496's ask).

Closing this out — all acceptance criteria met.