Use `startupProbe` for slow startup

[jiridanek]

• [rhoai-2.25] fix(ci): loosen jupyter-minimal/datascience probe timeouts for ppc64le/s390x red-hat-data-services/notebooks#2674 (review)

🧹 Nitpick comments (1)

jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml (1)

36-53: 🩺 Stability & Availability | 🔵 Trivial | 🏗️ Heavy lift

Use startupProbe for slow startup.

Both StatefulSets encode the startup budget in liveness and readiness settings. This delays healthy pods from becoming ready and delays recovery or traffic removal after post-start failures. A startupProbe gates the other probes only during initialization, which matches the PR objective. (v1-32.docs.kubernetes.io)

• jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml#L36-L53: add a startup probe for the measured emulated startup budget, then reduce the readiness delay and failure window.
• jupyter/minimal/ubi9-python-3.12/kustomize/base/statefulset.yaml#L36-L52: add the same startup protection with the minimal-image budget, then restore prompt post-start health detection.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml` around
lines 36 - 53, The liveness and readiness probes currently carry the
slow-startup budget; add a startupProbe to both
jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml lines 36-53
and jupyter/minimal/ubi9-python-3.12/kustomize/base/statefulset.yaml lines
36-52, using each image’s measured startup budget. Then reduce readiness initial
delays and failure windows in both StatefulSets so post-start failures are
detected promptly, while retaining the existing probe endpoints and liveness
behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml`:
- Around line 36-53: The liveness and readiness probes currently carry the
slow-startup budget; add a startupProbe to both
jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml lines 36-53
and jupyter/minimal/ubi9-python-3.12/kustomize/base/statefulset.yaml lines
36-52, using each image’s measured startup budget. Then reduce readiness initial
delays and failure windows in both StatefulSets so post-start failures are
detected promptly, while retaining the existing probe endpoints and liveness
behavior.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 93aad417-0f1d-4347-baf4-2e3c0cb89f04

📥 Commits

Reviewing files that changed from the base of the PR and between 46232cd and 7487358.

📒 Files selected for processing (3)

• jupyter/datascience/ubi9-python-3.12/kustomize/base/statefulset.yaml
• jupyter/minimal/ubi9-python-3.12/kustomize/base/statefulset.yaml
• tests/containers/kubernetes_utils.py

[andreapeterson]

Hi! I’d like to work on this issue. I’ll verify the current probe settings and intended startup budgets, keep the change scoped to the minimal and datascience StatefulSets, and update the relevant tests. Could you assign it to me?

[jiridanek]

@andreapeterson yes, why not... there are, essentially, two possibilities to tackle this. They can be taken up one after the another

1. minimalist, improve tests reliability in this repo, by adding a separate startup probe where appropriate
2. add the startup probe also to the dashboard template in https://github.com/opendatahub-io/odh-dashboard so that workbenches started from dashboard also have the startup probe (now they have readiness and liveness only)

https://kubernetes.io/docs/concepts/workloads/pods/probes/

[jiridanek]

And, doing either would be helpful.

[andreapeterson]

thank you! I’ll start with the first option and keep the change scoped to this repo- adding a separate startupProbe where appropriate and updating the related tests. Once that’s working, I can take a look at the dashboard template as a follow-up