Skip to content

feat(openhands): add budget preflight and reconciliation gate hook jobs - #1231

Draft
hieptl wants to merge 1 commit into
mainfrom
hieptl/ohe-3256-budget-upgrade-gate
Draft

feat(openhands): add budget preflight and reconciliation gate hook jobs#1231
hieptl wants to merge 1 commit into
mainfrom
hieptl/ohe-3256-budget-upgrade-gate

Conversation

@hieptl

@hieptl hieptl commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Description

Adds a budget upgrade preflight and a post-upgrade reconciliation gate to the openhands chart, as two Helm hook Jobs on the enterprise image (OHE-3256):

  • <release>-budget-preflight (pre-upgrade, weight 5, default mode acknowledge): read-only, runs the new image against the not-yet-migrated database and reports each enabled organization's budget state (missing baselines, members missing from LiteLLM, unmapped identities, cap drift, over-cap state, snapshot age, last sync) before enforcement changes.
  • <release>-budget-reconcile-gate (post-upgrade, weight 5, default mode strict): reconciles every enabled organization's LiteLLM caps in-process and verifies them by readback. In strict a blocking finding exits non-zero, fails helm upgrade, and marks the Replicated version failed; acknowledge records the findings and lets the release proceed.

Both Jobs are gated on .Release.IsUpgrade, so they never render on install or under ArgoCD (which renders with helm template), use backoffLimit: 0 with activeDeadlineSeconds under Helm's per-hook wait (300 s / 540 s against the 600 s Replicated timeout), and keep their last run with hook-delete-policy: before-hook-creation so the JSON artifact in the pod log survives until the next upgrade. Each Job prints one org_budget_preflight artifact line.

Also included: budgetPreflight / budgetReconcileGate values blocks, a render-time guard for invalid modes, two support-bundle logs collectors, a Replicated Config group "Upgrade Checks" (budget_preflight_mode, budget_reconcile_gate_mode) mapped into the chart values, helm-unittest suites for both Jobs and the guard, and an upgrade-rollback runbook section covering artifact capture, strict-gate recovery, manual runs, and restoring LiteLLM-side caps after a rollback.

Depends on OpenHands/enterprise#362, which adds run_budget_preflight.py to the image. This PR must not merge before a release containing that script is the chart's default image.tag; with an older image the pre-upgrade Job would fail on the missing module and block upgrades. Kept as a draft until then.

Validation:

  • helm template t charts/openhands --is-upgrade --show-only templates/budget-preflight-job.yaml --show-only templates/budget-reconcile-gate-job.yaml renders both Jobs with the expected hooks and env; without --is-upgrade nothing renders.
  • helm unittest charts/openhands: 150 passed in 31 suites.
  • helm lint charts/openhands: passed.
  • make lint (Replicated): exit 0; only pre-existing warnings on troubleshoot/secrets.yaml.

Helm Chart Checklist

  • I have tested the chart upgrade path from the previous version
  • I have verified backwards compatibility with existing values.yaml configurations
  • I have updated the chart's README.md if there are any breaking changes or new required values

New keys ship with defaults and values.schema.json does not restrict additional top-level keys, so existing values files are unaffected. The README lists no job values, so no README change was needed. A live upgrade on a Replicated test instance has not been run yet.

Additional Notes

  • The strict default on the post-upgrade gate is deliberate: it is what makes "failed reconciliation blocks release health" true out of the box. Customers with a known-dirty organization can select Acknowledge in the "Upgrade Checks" Config group before upgrading.
  • Mutual exclusion between the hook and the 15-minute budget-maintenance CronJob is tracked separately (OHE-3259); both are idempotent.
  • SaaS (ArgoCD) deployments get neither Job; the runbook documents running the preflight by hand there.

Run two Helm hook Jobs on the enterprise image around every upgrade of
the openhands release. budget-preflight (pre-upgrade, default
acknowledge) reads each enabled organization's budget state before any
manifest is applied. budget-reconcile-gate (post-upgrade, default
strict) reconciles every organization's LiteLLM caps and verifies them
by readback; a blocking finding fails helm upgrade, which is what
Replicated and native Helm report as release health.

Both Jobs are gated on .Release.IsUpgrade so a fresh install and ArgoCD
(which renders with helm template) never run them, keep their last run
until the next upgrade so the JSON artifact stays readable, and use
backoffLimit 0 with deadlines under Helm's per-hook wait. Modes are
validated at render time and exposed to Replicated customers through a
new "Upgrade Checks" Config group. The support bundle collects both Job
logs, and the upgrade/rollback runbook gains a section on reading the
artifacts, recovering from a failed strict gate, and restoring
LiteLLM-side caps after a rollback.

Requires an enterprise-server image that ships run_budget_preflight.py.

Refs OHE-3256
@github-actions github-actions Bot added the type: feat A new feature label Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: feat A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant