[rhoai-2.25] fix(ci): loosen jupyter-minimal/datascience probe timeouts for ppc64le/s390x - #2674
Conversation
livenessProbe (5s delay, 5s period, 3 failures = ~20s grace) kills the container via SIGTERM before jupyter lab finishes binding port 8888 under QEMU emulation on ppc64le, producing a permanent CrashLoopBackOff and failing the 600s `kubectl wait --for=condition=ready` in scripts/test_jupyter_with_papermill.sh. This is pre-existing on rhoai-2.25 and affects every push/PR, not just this repo's changes (see #2641 CI run 30664191953). Loosen livenessProbe to 30/10/6 (~90s grace) and readinessProbe to 45/10/6, matching main's already-proven fix. This is a partial cherry-pick: the full upstream commit (70bf817) bundles this fix with an unrelated hermetic build for jupyter/pytorch+llmcompressor that conflicts heavily with rhoai-2.25's diverged Dockerfile/Tekton state, so only the statefulset.yaml hunk is applied here. (cherry picked from commit 70bf817)
…4le CI Same CrashLoopBackOff root cause as jupyter-minimal: the livenessProbe (5s delay, 5s period, 3 failures) kills the container via SIGTERM before jupyter lab + datascience deps finish binding port 8888 under QEMU emulation on ppc64le. Datascience needs even more headroom than minimal (heavier deps, and jupyter + papermill share the pod cgroup during ppc64le CI tests), so it also needs a memory bump. Loosen livenessProbe to 120/10/6 and readinessProbe to 90/10/6, and raise resources.limits/requests.memory from 2Gi to 4Gi, matching main's already-proven fix. This is a partial cherry-pick assembled from two upstream commits, since each bundles this fix with unrelated work that conflicts heavily with rhoai-2.25's diverged state: - 70bf817 introduced the initial 30/10/6 liveness + 90/10/6 readiness loosening (bundled with an unrelated hermetic build for jupyter/pytorch+llmcompressor). - b76e061 raised liveness further to 120/10/6 and bumped memory to 4Gi (bundled with an unrelated prefetch-input centralization refactor). (cherry picked from commit 70bf817) (cherry picked from commit b76e061)
… pods
ImageDeployment.deploy() is used by tests/containers/workbenches/*.py
(the "OpenShift container tests" pytest suite) across every image and
architecture. Its post-readiness HTTP connect check
(Wait.until("Connecting to pod succeeds", 1, 30, ...)) only allowed
30s, which is too short once jupyter is genuinely slow to bind under
QEMU emulation (observed as
tests.containers.kubernetes_utils.WaitError: Timeout after 30 s
waiting for Connecting to pod succeeds on jupyter-minimal s390x,
#2641 CI run 30664191953).
This check has no k8s liveness/readiness probe backing it (the
Deployment this class creates defines none), so it can't restart-loop
the pod like the StatefulSet-based tests do -- it just times out.
Raise the timeout 30s -> 120s, switch localhost -> 127.0.0.1, probe
/api explicitly, and add a 10s per-request timeout, matching main's
already-proven fix (itself the result of incrementally raising this
same constant over time: 30 -> 60 -> 120, each time for the same
non-native-architecture slowness).
This is a partial cherry-pick: main's commits
f784977 (raised 30->60) and
8660870 (raised 60->120, plus the
localhost/api/timeout changes) each bundle this fix with unrelated
work (an s390x arch-detection removal, and a code-server nginx
open-redirect fix respectively) that conflicts with rhoai-2.25's
diverged state, so this applies the resulting hunk directly rather
than picking either commit whole.
📝 WalkthroughWalkthroughThe PR increases Jupyter probe delays, intervals, and failure thresholds. It increases datascience container memory to 4Gi. Deployment tests wait longer for pod connectivity and use a bounded request to ChangesStartup tolerance
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Each commit message documents which upstream commit(s) it's drawn from and why a literal -x cherry-pick wasn't possible (each upstream commit bundles the fix with unrelated, conflicting work). This is new |
|
Tip For best results, initiate chat on the files or code changes.
The three commits preserve the required upstream CI fixes without importing unrelated changes that conflict with the release branch. PR ✏️ Learnings added
You are interacting with an AI system. |
jiridanek
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml (1)
36-53: 🩺 Stability & Availability | 🔵 Trivial | 🏗️ Heavy liftUse
startupProbefor slow startup.Both StatefulSets encode the startup budget in liveness and readiness settings. This delays healthy pods from becoming ready and delays recovery or traffic removal after post-start failures. A
startupProbegates the other probes only during initialization, which matches the PR objective. (v1-32.docs.kubernetes.io)
jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml#L36-L53: add a startup probe for the measured emulated startup budget, then reduce the readiness delay and failure window.jupyter/minimal/ubi9-python-3.12/kustomize/base/statefulset.yaml#L36-L52: add the same startup protection with the minimal-image budget, then restore prompt post-start health detection.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml` around lines 36 - 53, The liveness and readiness probes currently carry the slow-startup budget; add a startupProbe to both jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml lines 36-53 and jupyter/minimal/ubi9-python-3.12/kustomize/base/statefulset.yaml lines 36-52, using each image’s measured startup budget. Then reduce readiness initial delays and failure windows in both StatefulSets so post-start failures are detected promptly, while retaining the existing probe endpoints and liveness behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml`:
- Around line 36-53: The liveness and readiness probes currently carry the
slow-startup budget; add a startupProbe to both
jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml lines 36-53
and jupyter/minimal/ubi9-python-3.12/kustomize/base/statefulset.yaml lines
36-52, using each image’s measured startup budget. Then reduce readiness initial
delays and failure windows in both StatefulSets so post-start failures are
detected promptly, while retaining the existing probe endpoints and liveness
behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 93aad417-0f1d-4347-baf4-2e3c0cb89f04
📒 Files selected for processing (3)
jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yamljupyter/minimal/ubi9-python-3.12/kustomize/base/statefulset.yamltests/containers/kubernetes_utils.py
CI status [antigravity]Run: Build Notebooks (pr) #30690916729 — 9/9 complete · 8 passed · 1 failed Failures so far
Likely root causes
Suggested next steps
|
|
Checked the `jupyter-datascience-ubi9-python-3.12 · linux/ppc64le` failure on this PR's pull_request run — it's not related to this PR's changes. It fails at the `make jupyter-datascience-ubi9-python-3.12` build step itself (before the statefulset.yaml / probe-timeout changes in this PR are even exercised): Root cause: This is persistent, not a flaky CDN issue — it'll block every ppc64le build of |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: ide-developer The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
2 participants
Description
Backports a fix for pre-existing, branch-wide CI breakage on
rhoai-2.25:jupyter-minimal-ubi9-python-3.12(ppc64le, s390x) andjupyter-datascience-ubi9-python-3.12(ppc64le) fail on essentially every push/PR (confirmed on the last 14+ unrelated runs, e.g. PR #2641 run 30664191953).Root cause: under QEMU emulation (ppc64le/s390x, since GHA runners are amd64),
jupyter labtakes far longer to bind port 8888 than the hardcoded CI timeouts/probes assume:scripts/test_jupyter_with_papermill.sh→ thejupyter/{minimal,datascience}/ubi9-python-3.12/kustomize/base/statefulset.yamlStatefulSet. ItslivenessProbe(5sdelay,5speriod,3failures ≈ 20s grace) kills the container via SIGTERM before jupyter finishes starting, producing a permanentCrashLoopBackOff(exitCode 143, observedrestartCount: 9) until the outer 600skubectl wait --for=condition=readygives up.tests/containers/kubernetes_utils.py'sImageDeployment.deploy()instead (no k8s probes at all there), whose own hardcoded app-level check —Wait.until("Connecting to pod succeeds", 1, 30, ...)— only allows 30s, too short for the same reason (WaitError: Timeout after 30 s ... Remote end closed connection).This exact problem was already diagnosed and fixed on
main, but never backported here. This PR is a partial cherry-pick of the relevant hunks from 3 upstream commits (each bundles the fix with unrelated work — a hermetic build, a prefetch-input refactor, a code-server nginx security fix — that conflicts heavily withrhoai-2.25's diverged state, so a literalcherry-pick -xof any of them fails outright):70bf81778f3b7cc3a76ffe155d983c57e444ba86— introduced the initial probe loosening for bothjupyter-minimalandjupyter-datascience.b76e061c47c0d1b822b9da5e60169eb83f6b5ffa— raisedjupyter-datascience's liveness further (30→120s) and bumped its memory limit 2Gi→4Gi (comment: "Jupyter + papermill share the pod cgroup; keep headroom for datascience tests on ppc64le CI").f784977a25/86608705582326196e7b078fea6648b9f156295d— raised thekubernetes_utils.pyconnect-wait timeout30s → 60s → 120sover time for the same non-native-architecture slowness, plus made the check itself more robust (127.0.0.1, explicit/apipath, per-requesttimeout=10).Each commit in this PR documents its exact upstream source(s) in the commit message trailer.
Not changed: the other 6 jupyter images with kustomize StatefulSets (
trustyai,pytorch,tensorflow,rocm/pytorch,rocm/tensorflow,pytorch+llmcompressor) still have tight/moderate probes onmaintoo — they haven't needed loosening because they aren't observed failing, so no changes proposed for those here.Per the merge-criteria note below about
odh/notebooks→rhds/notebookssync: this PR intentionally goes the other direction (main→rhoai-2.25) becauserhoai-2.25diverged frommainbefore these fixes landed upstream, and the normal sync doesn't run against release branches.How Has This Been Tested?
main's current committed state directly (git show main:<path>) rather than reconstructing values from memory — the twostatefulset.yamlfiles are now byte-identical tomainexcept for one pre-existing, unrelated line (--ServerApp.tornado_settings, removed by a separate, unrelated commit onmain).python3 -c "import yaml; yaml.safe_load(...)") on bothstatefulset.yamlfiles.ast.parse) onkubernetes_utils.py; confirmed the pre-existing pyright diagnostics in that file are all in unrelated code, not touched by this change.jupyter-minimal-ubi9-python-3.12(ppc64le, s390x) andjupyter-datascience-ubi9-python-3.12(ppc64le) jobs should go green here where they've been failing on every recent push/PR torhoai-2.25.Self checklist (all need to be checked):
make test(gmakeon macOS) before asking for reviewDockerfile.konfluxfiles should be done inodh/notebooksand automatically synced torhds/notebooks. For Konflux-specific changes, modifyDockerfile.konfluxfiles directly inrhds/notebooksas these require special attention in the downstream repository and flow to the upcoming RHOAI release. — N/A here, see note above: this intentionally backports frommaintorhoai-2.25directly since the branch diverged before these fixes landed.Merge criteria:
Summary by CodeRabbit