Readiness can be green while OpenRag is unusable: /health_check verifies no dependencies but backs all three probes

[andyne13]

Summary

/health_check reports only that the web process is answering. It performs no dependency checks, and the Helm chart wires that same endpoint to startup, readiness and liveness probes. A pod whose Postgres, Milvus, Ray or model endpoints are unreachable is still marked Ready, and Kubernetes routes traffic to it.

Code trace

openrag/api/routers/user/health.py:23-26:

@router.get("/health_check", summary="Health check endpoint for API", dependencies=[])
async def health_check(request: Request) -> str:
    # TODO: Error reporting about llm and vlm.
    return "RAG API is up."

The # TODO is the original author flagging exactly this gap.

infra/charts/openrag-stack/values.yaml:520-534 points all three probes at it:

probes:
  startup:   { httpGet: { path: /health_check, port: openrag }, failureThreshold: 30, periodSeconds: 10 }
  readiness: { httpGet: { path: /health_check, port: openrag }, initialDelaySeconds: 10, periodSeconds: 10, failureThreshold: 3 }
  liveness:  { httpGet: { path: /health_check, port: openrag }, ... }

Impact

• Readiness is meaningless. A rollout completes and the Service starts sending traffic to a pod that cannot serve a single search or chat request. Every request fails at the dependency instead of being routed to a healthy pod.
• Startup gives a false green. failureThreshold: 30 × periodSeconds: 10 allows five minutes for dependencies to come up, but nothing actually verifies they did.
• Rolling updates lose their safety property. Readiness gating is what makes a rolling update safe; a probe that can't fail turns it into an unguarded restart.

Suggested fix

Split the endpoints — they have opposite requirements, and conflating them is what makes this awkward to fix:

• /health_check (liveness) — keep it dependency-free. A liveness probe that fails on a dependency outage causes restart storms precisely when the cluster is already struggling, and restarting the pod cannot fix someone else's Postgres.
• new /ready (readiness + startup) — check the dependencies actually required to serve: Postgres, Milvus, Ray, and the configured model endpoints. Return 503 with a per-dependency breakdown when any is down.

Then repoint readinessProbe and startupProbe at /ready in values.yaml, leaving livenessProbe on /health_check.

Worth keeping the readiness check cheap and cached (a second or two) so probe traffic doesn't become its own load on Postgres and Milvus.

Found by

An independent third-party audit of OpenRag; verified against v2.1.0.