[rhoai-2.25] RHAIENG-6036: fix trustyai k8s papermill probe timing

[jiridanek]

Summary

Align TrustyAI StatefulSet health probes with pytorch (15/10) so Elyra/Jupyter can finish starting before liveness kills the pod.

Problem: On Build Notebooks k8s papermill (make_test.py), the trustyai pod CrashLoopBackOffs: liveness initialDelaySeconds: 5 probes port 8888 before Jupyter listens (~12s+ with Elyra). Papermill then fails with container not found ("notebook").

Evidence: #2499 CI run 29087867712 — testcontainers 16 passed; k8s phase failed on probe kills (events captured in make_test.py finally dumps).

Fix: jupyter/trustyai/ubi9-python-3.12/kustomize/base/statefulset.yaml only:

• liveness: initialDelaySeconds 5→15, periodSeconds 5→10
• readiness: initialDelaySeconds 10→15, periodSeconds 5→10

Matches jupyter/pytorch/ubi9-python-3.12/kustomize/base/statefulset.yaml on main. Not a cherry-pick -x from main trustyai (trustyai statefulset on main still has 5/10).

Out of scope (already fixed elsewhere)

Symptom	Fix	Status
/opt/app-root/share 755:0 + spinner	#2451	RHAIENG-6041 — testcontainers green on tip
Papermill CSV 404	#2499	Merged
Codeserver mysql test	#2446	RHAIENG-6039 — testcontainers green on tip
Codeserver check-payload FIPS	#2500	Separate PR

Test plan

• jupyter-trustyai-ubi9-python-3.12 amd64 Build Notebooks: k8s papermill (make_test.py) passes
• Testcontainers phase remains green (unchanged by this PR)

Local repro

k3s on 100.105.65.25: Quay trustyai image + #2499 branch — papermill passes with probe 15s patch (/tmp/test-trustyai-quay-k8s.log).

Summary by CodeRabbit

• Bug Fixes
  • Improved service health-check timing by increasing liveness and readiness probe initial delays and intervals to better accommodate startup.
  • Preserved existing health-check endpoints, probe methods/targets, and success/failure thresholds.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f9543a9c-fe7e-4b4c-b4b5-1776883706a9

📥 Commits

Reviewing files that changed from the base of the PR and between 293a861 and 973cd54.

📒 Files selected for processing (1)

• jupyter/trustyai/ubi9-python-3.12/kustomize/base/statefulset.yaml

---

📝 Walkthrough

Walkthrough

The StatefulSet liveness and readiness probes now wait longer before starting and run at longer intervals. Their endpoints, ports, schemes, and success/failure thresholds remain unchanged.

Changes

StatefulSet probe timing

Layer / File(s)	Summary
Health probe timing jupyter/trustyai/ubi9-python-3.12/kustomize/base/statefulset.yaml	Liveness and readiness probe delays increase to 15 seconds, and polling periods increase to 10 seconds while probe behavior remains otherwise unchanged.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: daniellutz

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly matches the main change: adjusting TrustyAI k8s papermill probe timing.
Description check	✅ Passed	The PR description covers the summary, problem, fix, test plan, and local repro, with only minor template gaps.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/rhoai-2.25-ws2-codeserver-trustyai

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

tests/containers/workbenches/jupyterlab/jupyterlab_datascience_test.py (1)

88-93: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Skip guard for code-server images looks correct.

The dual-variant label check (-code-server- for ODH, -codeserver- for RHOAI/RHDS Konflux) is well-documented and consistent with the pattern in conftest.py. The removal of the conditional mysql-connector-python installation block is safe since code-server images now never reach that code path.

As per coding guidelines, consider adding @allure.issue("RHOAIENG-140") to link this test to the Jira issue referenced in the skip message.

♻️ Optional: add allure issue decorator

+    `@allure.issue`("RHOAIENG-140")
     `@allure.description`("Check that mysql client functionality is working with SASL plain auth.")
     def test_mysql_connection(self, tf: TestFrame, datascience_image: Image, subtests):

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/containers/workbenches/jupyterlab/jupyterlab_datascience_test.py`
around lines 88 - 93, Add the `@allure.issue`("RHOAIENG-140") decorator to the
affected JupyterLab datasource test function, alongside its existing test
decorators, so the test is linked to the Jira issue referenced by the
code-server skip guard.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/containers/workbenches/jupyterlab/jupyterlab_datascience_test.py`:
- Around line 88-93: Add the `@allure.issue`("RHOAIENG-140") decorator to the
affected JupyterLab datasource test function, alongside its existing test
decorators, so the test is linked to the Jira issue referenced by the
code-server skip guard.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a3c9f052-48ee-4f80-9bb9-264502e392f2

📥 Commits

Reviewing files that changed from the base of the PR and between ab66e21 and d8115e6f24b2c8a3241326ce7bda9f5f1c192db4.

📒 Files selected for processing (5)

• jupyter/trustyai/ubi9-python-3.12/Dockerfile.cpu
• jupyter/trustyai/ubi9-python-3.12/Dockerfile.konflux.cpu
• jupyter/trustyai/ubi9-python-3.12/kustomize/base/statefulset.yaml
• scripts/check-payload/config.toml
• tests/containers/workbenches/jupyterlab/jupyterlab_datascience_test.py

[jiridanek]

this may not be actually necessary, maybe the timeouts were a rare flake?

[github-actions]

Superseded by newer run: https://github.com/red-hat-data-services/notebooks/actions/runs/30348232918

CI status [antigravity]

Run: Build Notebooks (push) #30348228985 — 1/1 complete · 1 skipped
Last updated: 2026-07-28T09:50:05Z

Workflow completed with skipped jobs.

[jiridanek]

Rebased onto current rhoai-2.25 tip (8711362) + probe-only cherry-pick (973cd5490).

TrustyAI kind flake: exit 137 / CrashLoop under aggressive probes (initialDelaySeconds 5/10). Aligns liveness/readiness to 15 / 10 (probe-only; no memory change).

Recent evidence:

• RHAIENG-4594: chore(deps): fix CVE-2026-40192 in Pillow [rhoai-2.25] #2630 job 90187214189
• [rhoai-2.25] RHAIENG-287: fix(Dockerfiles): wrap remaining RUN && chains in bash heredocs #2485 job 89981963970

Please approve + lgtm so we can land this flake fix.

[github-actions]

CI status [antigravity]

Run: Build Notebooks (pr) #30348232918 — 2/2 passed
Last updated: 2026-07-28T10:06:58Z

All matrix jobs completed successfully.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: atheo89

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [atheo89]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment