feat(openhands): add budget preflight and reconciliation gate hook jobs - #1231
Draft
hieptl wants to merge 1 commit into
Draft
feat(openhands): add budget preflight and reconciliation gate hook jobs#1231hieptl wants to merge 1 commit into
hieptl wants to merge 1 commit into
Conversation
Run two Helm hook Jobs on the enterprise image around every upgrade of the openhands release. budget-preflight (pre-upgrade, default acknowledge) reads each enabled organization's budget state before any manifest is applied. budget-reconcile-gate (post-upgrade, default strict) reconciles every organization's LiteLLM caps and verifies them by readback; a blocking finding fails helm upgrade, which is what Replicated and native Helm report as release health. Both Jobs are gated on .Release.IsUpgrade so a fresh install and ArgoCD (which renders with helm template) never run them, keep their last run until the next upgrade so the JSON artifact stays readable, and use backoffLimit 0 with deadlines under Helm's per-hook wait. Modes are validated at render time and exposed to Replicated customers through a new "Upgrade Checks" Config group. The support bundle collects both Job logs, and the upgrade/rollback runbook gains a section on reading the artifacts, recovering from a failed strict gate, and restoring LiteLLM-side caps after a rollback. Requires an enterprise-server image that ships run_budget_preflight.py. Refs OHE-3256
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a budget upgrade preflight and a post-upgrade reconciliation gate to the
openhandschart, as two Helm hook Jobs on the enterprise image (OHE-3256):<release>-budget-preflight(pre-upgrade, weight 5, default modeacknowledge): read-only, runs the new image against the not-yet-migrated database and reports each enabled organization's budget state (missing baselines, members missing from LiteLLM, unmapped identities, cap drift, over-cap state, snapshot age, last sync) before enforcement changes.<release>-budget-reconcile-gate(post-upgrade, weight 5, default modestrict): reconciles every enabled organization's LiteLLM caps in-process and verifies them by readback. Instricta blocking finding exits non-zero, failshelm upgrade, and marks the Replicated version failed;acknowledgerecords the findings and lets the release proceed.Both Jobs are gated on
.Release.IsUpgrade, so they never render on install or under ArgoCD (which renders withhelm template), usebackoffLimit: 0withactiveDeadlineSecondsunder Helm's per-hook wait (300 s / 540 s against the 600 s Replicated timeout), and keep their last run withhook-delete-policy: before-hook-creationso the JSON artifact in the pod log survives until the next upgrade. Each Job prints oneorg_budget_preflightartifact line.Also included:
budgetPreflight/budgetReconcileGatevalues blocks, a render-time guard for invalid modes, two support-bundlelogscollectors, a Replicated Config group "Upgrade Checks" (budget_preflight_mode,budget_reconcile_gate_mode) mapped into the chart values, helm-unittest suites for both Jobs and the guard, and an upgrade-rollback runbook section covering artifact capture, strict-gate recovery, manual runs, and restoring LiteLLM-side caps after a rollback.Depends on OpenHands/enterprise#362, which adds
run_budget_preflight.pyto the image. This PR must not merge before a release containing that script is the chart's defaultimage.tag; with an older image the pre-upgrade Job would fail on the missing module and block upgrades. Kept as a draft until then.Validation:
helm template t charts/openhands --is-upgrade --show-only templates/budget-preflight-job.yaml --show-only templates/budget-reconcile-gate-job.yamlrenders both Jobs with the expected hooks and env; without--is-upgradenothing renders.helm unittest charts/openhands: 150 passed in 31 suites.helm lint charts/openhands: passed.make lint(Replicated): exit 0; only pre-existing warnings ontroubleshoot/secrets.yaml.Helm Chart Checklist
New keys ship with defaults and
values.schema.jsondoes not restrict additional top-level keys, so existing values files are unaffected. The README lists no job values, so no README change was needed. A live upgrade on a Replicated test instance has not been run yet.Additional Notes
budget-maintenanceCronJob is tracked separately (OHE-3259); both are idempotent.