fix(tests): probe / instead of /api for OpenShift port-forward readiness

[jiridanek]

Summary

Three commits, in order:

1. fix(tests): probe / instead of /api for OpenShift port-forward readiness — restores a previously-working fix that got silently reverted, fixing the recurring RStudio/cuda-rstudio test_image_run_on_openshift timeout tracked in [rhoai-2.25] Flaky test_image_run_on_openshift: rstudio/cuda-rstudio (amd64) hit 'Connecting to pod succeeds' timeout on nearly every push #2684.
2. ci(test-containers): add a job to run the openshift-marked container tests — adds openshift-container-tests to the fast, PR-scoped self-test workflow (test-containers.yaml), so this exact code path gets exercised on every PR instead of only in the full, expensive image-build workflow. This is what would have caught both regressions described in commit 1 immediately instead of weeks later. See test-containers.yaml self-test doesn't exercise the openshift-marked tests (ImageDeployment/test_image_run_on_openshift) opendatahub-io/notebooks#4257 for the equivalent tracking issue on main.
3. fix(ci): disable zstd:chunked partial pulls in openshift-container-tests — fixes a failure the new job in commit 2 immediately hit in this PR's own CI (3 consecutive runs): podman pull of the codeserver image failed with 501 Unsupported client range, because that image's layers happen to be zstd:chunked-compressed (a known buildah bug means whether a given build's layers end up zstd:chunked vs. plain gzip depends on --cache-from/--cache-to cache hit/miss, not on the image itself) and install-podman-action's storage.conf enables partial/ranged fetches for such layers, which ghcr.io's blob endpoint doesn't support. Disabled for this job only.

Commit 1: RStudio readiness probe fix

Restores a previously-working fix that got silently reverted, fixing the recurring RStudio/cuda-rstudio test_image_run_on_openshift timeout tracked in #2684.

• RHAIENG-6036 (08f4fe374, merged 2026-07-11) fixed this correctly: probe / with a persistent requests.Session() (cookies are required across RStudio's multi-hop redirect chain), because RStudio doesn't support /api the way Jupyter and code-server do.
• The hermetic-codeserver backport (da9540c47) later silently reverted this hunk back to a bare 30s/localhost/no-session probe (almost certainly because that backport branch was cut before 08f4fe374 merged).
• A same-day follow-up (74873588f) noticed only the regressed timeout and re-fixed it by copying main's /api-only probe — but main dropped RStudio entirely, so that path never actually worked here.

Evidence

Confirmed against real ghcr.io rhoai-2.25 images (jupyter-minimal, rstudio, codeserver) pulled and run directly:

rstudio /api  -> ConnectionError: Failed to establish a new connection to 127.0.0.1:8888 (Connection refused)

RStudio's nginx redirects /api to a hardcoded http://127.0.0.1:8888/api/ — its own internal container port, not the locally-forwarded port — so it's unreachable through any port-forward/proxy.

/ (session-based, redirects followed) returns 200 for all three:

jupyter-minimal 200 http://127.0.0.1:<port>/login?next=%2Flab
rstudio         200 http://127.0.0.1:<port>/rstudio/
codeserver      200 http://127.0.0.1:<port>/codeserver/?folder=/opt/app-root/src

(a bare GET without a persistent cookie jar redirect-loops on RStudio's /, which is why the session matters, not just the path.)

Commit 2: add openshift-container-tests to the self-test workflow

.github/workflows/test-containers.yaml only ran pytest -m 'not openshift and not cuda and not rocm', explicitly skipping test_image_run_on_openshift. That's exactly the code path that regressed twice without anyone noticing (commit 1's history). Adds a job that reuses find-images's matrix and mirrors build-notebooks-TEMPLATE.yaml's own provisioning steps (install-podman-action + provision-k8s) to run pytest -m 'openshift and not cuda and not rocm' against each pulled image in fast, PR-scoped CI.

Commit 3: fix a pull failure the new job surfaced

The new job in commit 2 failed 3/3 times in this PR's own CI on codeserver specifically:

Error: unable to copy from source docker://ghcr.io/.../codeserver...:
partial pull of blob sha256:...: read zstd:chunked manifest: fetching
partial blob: received unexpected HTTP status: 501 Unsupported client range

codeserver's layers happened to be pushed as zstd:chunked (the other 4 images in the matrix are plain gzip) — this is a known buildah issue where --cache-to/--cache-from used to ignore the configured compression_format for cache-derived (hit) layers, so which images end up zstd:chunked vs. gzip depends on build-cache hit/miss, not on the image itself. install-podman-action's storage.conf enables partial/ranged pulls for zstd:chunked layers, and ghcr.io's blob endpoint returns 501 for the resulting ranged request instead of falling back to a full pull. Disabled enable_partial_images for this job only (not the shared ci/cached-builds/storage.conf, which other build jobs still benefit from).

Test plan

• ruff check / pyright clean on the changed files
• Manually verified / + session works for jupyter-minimal, rstudio, and codeserver images pulled from ghcr.io (rhoai-2.25 build)
• actionlint / YAML-valid on test-containers.yaml
• CI: openshift-container-tests passes for all 5 images (rstudio/cuda-rstudio no longer time out; codeserver's partial-pull failure fixed) after commit 3

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Improved port-forward readiness detection by checking the service root endpoint.
  • Added redirect handling and stricter validation for successful HTTP responses.
  • Refined readiness timeouts to provide more reliable startup behavior.

• Tests

  • Added automated OpenShift container test coverage across supported images.
  • Added automated result collection and reporting to improve test visibility.
  • Expanded validation across multiple container and cluster configurations.

[coderabbitai]

📝 Walkthrough

Walkthrough

The Kubernetes readiness check now probes the root URL and uses shared timeout constants. The container workflow adds matrix-based OpenShift tests with Podman, Kubernetes provisioning, GHCR authentication, JUnit output, and conditional Codecov uploads.

Changes

Container test execution

Layer / File(s)	Summary
Readiness probe and timeout handling tests/containers/kubernetes_utils.py	The readiness probe requests the root URL, follows redirects, requires HTTP 200, uses a five-second timeout, and uses TestFrameConstants.TIMEOUT_2MIN for the wait.
OpenShift container test workflow .github/workflows/test-containers.yaml	The workflow adds matrix-based OpenShift container tests. It configures rootful Podman, disables unsupported partial image pulls, provisions Kubernetes, authenticates with GHCR, runs marked tests, and uploads JUnit results to Codecov.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• red-hat-data-services/notebooks#2447: Extends the same container test workflow.
• red-hat-data-services/notebooks#2488: Modifies the same port-forward readiness probe.
• red-hat-data-services/notebooks#2674: Modifies port-forward readiness and connection timeouts.

Suggested reviewers: daniellutz, dibryant

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the primary OpenShift readiness probe fix.
Description check	✅ Passed	The description clearly explains the fixes, rationale, affected workflows, failure evidence, and test results, although it omits the template checklists.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/rhoai-2.25-openshift-rstudio-probe

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Superseded by newer run: https://github.com/red-hat-data-services/notebooks/actions/runs/30716162558

CI status [antigravity]

Run: Build Notebooks (pr) #30715956755 — 2/2 complete · 1 passed · 1 skipped
Last updated: 2026-08-01T19:59:35Z

No workbench image jobs ran; all matrix jobs were skipped.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/containers/kubernetes_utils.py`:
- Around line 277-285: Update the readiness flow around _ready and Wait.until so
a single requests.Session is created before polling, reused by every _ready
invocation, and closed after Wait.until completes, including when the wait
raises. Add a regression test covering cookie-dependent readiness to verify
repeated polls use the same session and eventually succeed.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 952a7542-eba9-4c99-99ba-7bb7103c49b0

📥 Commits

Reviewing files that changed from the base of the PR and between 7cc120e and 651d32e.

📒 Files selected for processing (1)

• tests/containers/kubernetes_utils.py

[github-actions]

Superseded by newer run: https://github.com/red-hat-data-services/notebooks/actions/runs/30716163834

CI status [antigravity]

Run: Build Notebooks (push) #30716162558 — 2/2 complete · 2 skipped
Last updated: 2026-08-01T20:04:27Z

Workflow completed with skipped jobs.

[github-actions]

Superseded by newer run: https://github.com/red-hat-data-services/notebooks/actions/runs/30716642515

CI status [antigravity]

Run: Build Notebooks (pr) #30716163834 — 2/2 complete · 1 passed · 1 skipped
Last updated: 2026-08-01T20:04:51Z

No workbench image jobs ran; all matrix jobs were skipped.

[github-actions]

📋 Review Summary

This pull request correctly addresses the RStudio/cuda-rstudio timeout issue on OpenShift by reverting to a session-based root (/) probe instead of probing /api, which fails due to RStudio's hardcoded internal redirect behavior. The changes are clean, targeted, and well-tested via the expanded workflow matrix.

🔍 General Feedback

• The fix correctly accounts for multi-hop redirects and cookies required by RStudio's session initialization.
• The new openshift-container-tests job cleanly integrates dynamic image matrix discovery with robust result reporting via Codecov.

I did not find any issues requiring inline comments.

[github-actions]

Superseded by newer run: https://github.com/red-hat-data-services/notebooks/actions/runs/30716643675

CI status [antigravity]

Run: Build Notebooks (push) #30716642515 — 2/2 complete · 2 skipped
Last updated: 2026-08-01T20:17:38Z

Workflow completed with skipped jobs.

[github-actions]

CI status [antigravity]

Run: Build Notebooks (pr) #30716643675 — 2/2 complete · 1 passed · 1 skipped
Last updated: 2026-08-01T20:18:05Z

No workbench image jobs ran; all matrix jobs were skipped.

[jiridanek]

Follow-up: why only codeserver hit the zstd:chunked partial-pull failure

For the record, since this came up in review discussion — investigated why codeserver was the only image (of the 5 in the matrix) that hit the 501 Unsupported client range failure fixed in the third commit.

Not a different build pipeline. All 5 images build through the identical GHA workflow with the same podman build --cache-from/--cache-to invocation. The difference is a known buildah bug: --cache-from/--cache-to doesn't respect the configured compression_format for cache-derived (unchanged/reused) layers — those keep whatever compression they already had, while genuinely new/changed layers get freshly compressed with the currently-configured zstd:chunked format. So whether a given image's layers end up zstd:chunked vs. plain gzip is a function of build-cache hit/miss on that particular run, not something inherent to the image. See podman-container-tools/buildah#4613 and podman-container-tools/buildah#5586.

Verified, not just theorized:

• Confirmed via skopeo inspect --raw that the failing blob (sha256:44607f3c..., 87MB, zstd:chunked-annotated) is not an unchanged layer inherited from the base image registry.access.redhat.com/ubi9/python-312:latest — that base image's own layers are all plain gzip (80MB/17MB/229MB/70MB), none matching. So this blob was genuinely produced/recompressed during this repo's own build.
• codeserver's other manifest layers do match blobs shared with the base image chain (e.g. 211952189db..., 9b722470c...) and stayed plain gzip — consistent with unchanged/reused content keeping its original compression, while the one layer that's actually new to this build (presumably the code-server/VS Code bundle itself) came out zstd:chunked.
• Couldn't find a literal "cache hit/miss" log line for --cache-from/--cache-to to nail this down further (podman doesn't print one for this caching mode the way it does for classic per-RUN-layer caching) — so this is strong circumstantial evidence, not a smoking gun, but it rules out the alternative explanation (different pipeline) and is consistent with the cache-hit/miss explanation.

cc @coderabbitai Either way, the fix (disabling enable_partial_images for this job) is correct regardless of which image ends up zstd:chunked on any given run, since that's effectively non-deterministic per the above.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@jiridanek Thanks for the detailed investigation. The evidence rules out a distinct codeserver build pipeline. It also explains why the failure can move between images when cache reuse changes.

Disabling enable_partial_images for the OpenShift container test job is the correct job-level mitigation. It does not depend on a specific image or a deterministic layer compression result.

🐇🔍

---

✏️ Learnings added

Learnt from: jiridanek
URL: https://github.com/red-hat-data-services/notebooks/pull/2685

Timestamp: 2026-08-01T20:26:53.528Z
Learning: In `.github/workflows/test-containers.yaml`, all container images use the same GitHub Actions build workflow and `podman build --cache-from/--cache-to` configuration. Buildah cache behavior can cause reused layers to retain their existing compression while newly produced layers use the configured `zstd:chunked` compression. Therefore, the image that contains a `zstd:chunked` layer can vary between builds. Keep `enable_partial_images` disabled for the OpenShift container test job because GHCR blob endpoints can reject ranged `zstd:chunked` requests with `501 Unsupported client range`.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

_{You are interacting with an AI system.}

[ide-developer]

Reviewed all three commits.

1. RStudio readiness probe fix — restores the RHAIENG-6036 fix that got silently reverted by the codeserver-backport merge. The root cause (RStudio's nginx redirects /api to a hardcoded http://127.0.0.1:8888/api/, unreachable through any port-forward) is well-documented in #2684 and verified empirically against real pulled images (jupyter-minimal, rstudio, codeserver), not just inferred from logs.
2. New openshift-container-tests job — closes the coverage gap that let the above regression go unnoticed for weeks; sensibly reuses the existing find-images matrix and the same provisioning building blocks (install-podman-action + provision-k8s) the main build workflow already relies on.
3. Disable partial zstd:chunked pulls for this job — real failure caught by the new job in its own first CI runs (codeserver 3/3), root-caused to a 501 Unsupported client range on ghcr.io's blob endpoint for ranged fetches, scoped narrowly to this job rather than touching the shared storage.conf.

All CI checks are green, including all 5 openshift: matrix legs. Nice find-and-fix loop — the new test job earning its keep on the very PR that introduces it is a good sign.

Approving.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ide-developer
Once this PR has been reviewed and has the lgtm label, please assign jstourac for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment