diff --git a/.github/AGENTS.md b/.github/AGENTS.md index 3aa83489ab6..f2dd821ac30 100644 --- a/.github/AGENTS.md +++ b/.github/AGENTS.md @@ -11,11 +11,7 @@ These rules apply to GitHub Actions workflows and custom actions under `.github/ - Deployment group names are a cross-workflow API. Keep them aligned with `.github/scripts/check-deployment-concurrency.py`; use `cancel-in-progress: false` so a newer run cannot interrupt a remote mutation - or a staged validation/traffic promotion. `desktop_emergency_promote_beta.yml` - is the narrow exception: GitHub's single replaceable pending slot cannot - preserve a break-glass request, so each attempt has an isolated lock and the - server-side exact pointer CAS is the cross-lane arbiter; retain its focused - regression contract. + or a staged validation/traffic promotion. - `deploy-backend-stack-` intentionally covers the four backend Cloud Run services, traffic repair, backend-listen, LLM gateway, and backend-secrets. Those paths share mutable releases, while unrelated services diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 5dc9c5fa8b4..44b4930aa03 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,7 +1,3 @@ -# Release-ring deployment code is production authority. Branch protection must -# require this owner plus the release-ring guard check before merging changes. +# Broad production authority remains protected after the retired release-ring deletion. /.github/workflows/** @Git-on-my-level -/.github/scripts/check_release_rings.py @Git-on-my-level -/backend/scripts/release_rings.py @Git-on-my-level -/backend/scripts/render_release_ring_config.py @Git-on-my-level /backend/charts/backend-secrets/** @Git-on-my-level diff --git a/.github/checks-manifest.yaml b/.github/checks-manifest.yaml index dadb6a0c3ad..354541370b3 100644 --- a/.github/checks-manifest.yaml +++ b/.github/checks-manifest.yaml @@ -14,16 +14,6 @@ checks: triggers: ["all"] lanes: ["local", "ci"] reason: "manifest validity and workflow drift guard" - - id: release-ring-guards - command: ["python3", ".github/scripts/check_release_rings.py"] - triggers: [".github/workflows/release-record.yml", ".github/workflows/deploy-release-ring.yml", ".github/scripts/check_release_rings.py", ".github/scripts/check-deployment-concurrency.py", "backend/scripts/release_rings.py", "backend/scripts/render_release_ring_config.py", "backend/deploy/release_rings.yaml"] - lanes: ["local", "ci"] - reason: "#10049 keeps release records immutable, deploy inputs record-only, and recovery fail-closed" - - id: release-ring-guard-tests - command: ["python3", ".github/scripts/test_check_release_rings.py"] - triggers: [".github/scripts/check_release_rings.py", ".github/scripts/test_check_release_rings.py", ".github/workflows/release-record.yml", ".github/workflows/deploy-release-ring.yml", ".github/checks-manifest.yaml"] - lanes: ["local", "ci"] - reason: "release-ring structural guard must reject missing immutability, restore, and OIDC controls" - id: product-file-line-count-ratchet command: ["python3", ".github/scripts/check_product_file_line_count_ratchet.py", "--changed-files", "{changed_files}", "--base", "{base}"] triggers: ["backend/**/*.py", "desktop/macos/**/*.swift", "desktop/macos/**/*.rs", ".github/scripts/check_product_file_line_count_ratchet.py", ".github/scripts/product_file_line_count_ratchet_baseline/**/*.json"] @@ -44,11 +34,6 @@ checks: triggers: [".github/schemas/desktop-release-evidence-v1.schema.json", ".github/scripts/desktop_release_doctor.py", ".github/scripts/desktop_release_doctor_report.py", ".github/scripts/test_desktop_release_doctor.py", ".github/workflows/desktop_release_doctor.yml"] lanes: ["local", "ci"] reason: "advisory desktop release evidence and drift report" - - id: desktop-qualification-dispatch - command: ["python3", ".github/scripts/test_desktop_qualification_dispatch.py"] - triggers: [".github/scripts/desktop_qualification_dispatch.py", ".github/scripts/test_desktop_qualification_dispatch.py", ".github/workflows/desktop_qualify_beta.yml", "codemagic.yaml", ".github/checks-manifest.yaml"] - lanes: ["local", "ci"] - reason: "#10080 timeout retries must expose state and permit only one trusted qualification execution per candidate dispatch key" - id: desktop-changelog-io-tests command: ["python3", ".github/scripts/test_desktop_changelog.py"] triggers: [".github/scripts/desktop-changelog.py", ".github/scripts/test_desktop_changelog.py", ".github/checks-manifest.yaml"] @@ -355,16 +340,47 @@ checks: triggers: [".github/workflows/gcp_backend.yml", ".github/workflows/gcp_backend_auto_dev.yml", ".github/scripts/verify_backend_release_admission.py", ".github/scripts/verify_auto_backend_release_admission.py", ".github/scripts/check_backend_deploy_source_admission.py", ".github/scripts/test_check_backend_deploy_source_admission.py", ".github/checks-manifest.yaml"] lanes: ["local", "ci"] reason: "#9991 mutation-tests workflow-run event, result, branch, SHA, repository, and manual proof drift" + - id: gcp-backend-production-boundary + command: ["python3", ".github/scripts/check-gcp-backend-production-boundary.py"] + triggers: [".github/workflows/gcp_backend.yml", "backend/scripts/probe-transcription-candidate-from-cloud-run.sh", "backend/tests/unit/test_probe_transcription_candidate_from_cloud_run.py", ".github/scripts/check-gcp-backend-production-boundary.py", ".github/scripts/test_check_gcp_backend_production_boundary.py", ".github/checks-manifest.yaml"] + lanes: ["local", "ci"] + reason: "#10163 locks the supported production Cloud Run-only boundary and canonical tagged-candidate IAM audience without introducing another deployment control plane" + - id: gcp-backend-production-boundary-fixtures + command: ["python3", ".github/scripts/test_check_gcp_backend_production_boundary.py"] + triggers: [".github/workflows/gcp_backend.yml", ".github/scripts/check-gcp-backend-production-boundary.py", ".github/scripts/test_check_gcp_backend_production_boundary.py", ".github/checks-manifest.yaml"] + lanes: ["local", "ci"] + reason: "#10163 mutation-tests prod/all rejection, canonical IAM audience, and tagged request-target drift" + - id: direct-backend-production-admission + command: ["python3", ".github/scripts/check-direct-backend-production-admission.py"] + triggers: [".github/workflows/gcp_backend_agent_proxy.yml", ".github/workflows/gcp_backend_listen_helm.yml", ".github/workflows/gcp_backend_pusher.yml", ".github/workflows/gcp_llm_gateway.yml", ".github/scripts/check-direct-backend-production-admission.py", ".github/scripts/test_check_direct_backend_production_admission.py", ".github/checks-manifest.yaml"] + lanes: ["local", "ci"] + reason: "#10163 prevents direct production writers from deploying caller refs or mislabeling checked-out source" + - id: direct-backend-production-admission-fixtures + command: ["python3", ".github/scripts/test_check_direct_backend_production_admission.py"] + triggers: [".github/workflows/gcp_backend_agent_proxy.yml", ".github/workflows/gcp_backend_listen_helm.yml", ".github/workflows/gcp_backend_pusher.yml", ".github/workflows/gcp_llm_gateway.yml", ".github/scripts/check-direct-backend-production-admission.py", ".github/scripts/test_check_direct_backend_production_admission.py", ".github/checks-manifest.yaml"] + lanes: ["local", "ci"] + reason: "#10163 mutation-tests branch bypass and GITHUB_SHA image mislabeling" + - id: mobile-production-routing + command: ["python3", ".github/scripts/test_check_mobile_production_routing.py"] + triggers: ["codemagic.yaml", ".github/scripts/check-mobile-production-routing.py", ".github/scripts/test_check_mobile_production_routing.py", ".github/checks-manifest.yaml"] + lanes: ["local", "ci"] + reason: "#10163 mutation-tests production-family mobile API routing" + - id: stable-pointer-precondition-cli + command: ["python3", ".github/scripts/check_stable_pointer_precondition.py", "--help"] + triggers: [".github/workflows/desktop_promote_prod.yml", ".github/scripts/check_stable_pointer_precondition.py", ".github/scripts/verify_stable_appcast.py", ".github/scripts/test_stable_promotion_verifiers.py", ".github/checks-manifest.yaml"] + lanes: ["local", "ci"] + reason: "#10163 keeps Stable retry acknowledgement and appcast verification executable" + - id: stable-pointer-precondition-fixtures + command: ["python3", ".github/scripts/test_stable_promotion_verifiers.py"] + triggers: [".github/workflows/desktop_promote_prod.yml", ".github/scripts/check_stable_pointer_precondition.py", ".github/scripts/verify_stable_appcast.py", ".github/scripts/test_stable_promotion_verifiers.py", ".github/checks-manifest.yaml"] + lanes: ["local", "ci"] + reason: "#10163 mutation-sensitive Stable retry and default-channel feed fixtures" exempt: - path: ".github/scripts/check-desktop-auto-beta-candidate.py" reason: "release pipeline check requires published candidate artifacts and workflow inputs" - path: ".github/scripts/check-desktop-prod-promotion-policy.py" reason: "release-policy full-tree check, not diff-scoped" - - path: ".github/scripts/check-desktop-release-promotion.py" - reason: "release pipeline check with workflow inputs" - - path: ".github/scripts/check-desktop-release-terminology.py" - reason: "release-policy full-tree check, not diff-scoped" - path: ".github/scripts/check-release-process-guards.py" reason: "release-policy full-tree check, not diff-scoped" - path: ".github/scripts/check-deployment-concurrency.py" diff --git a/.github/schemas/desktop-release-manifest-v1.schema.json b/.github/schemas/desktop-release-manifest-v1.schema.json index 7ec7986fd49..2e3dbbe2700 100644 --- a/.github/schemas/desktop-release-manifest-v1.schema.json +++ b/.github/schemas/desktop-release-manifest-v1.schema.json @@ -19,6 +19,11 @@ "dmg_url", "dmg_sha256", "ed_signature", + "beta_zip_url", + "beta_zip_sha256", + "beta_dmg_url", + "beta_dmg_sha256", + "beta_ed_signature", "qualification_evidence_asset", "qualification_evidence_sha256", "qualification_tier", @@ -46,6 +51,11 @@ "dmg_url": { "$ref": "#/$defs/dmgUrl" }, "dmg_sha256": { "$ref": "#/$defs/sha256" }, "ed_signature": { "type": "string", "pattern": ".*\\S.*" }, + "beta_zip_url": { "$ref": "#/$defs/betaZipUrl" }, + "beta_zip_sha256": { "$ref": "#/$defs/sha256" }, + "beta_dmg_url": { "$ref": "#/$defs/betaDmgUrl" }, + "beta_dmg_sha256": { "$ref": "#/$defs/sha256" }, + "beta_ed_signature": { "type": "string", "pattern": ".*\\S.*" }, "qualification_evidence_asset": { "type": "string", "pattern": "^qualification-evidence-[^/]+\\.json$" @@ -157,6 +167,14 @@ "dmgUrl": { "type": "string", "pattern": "^https://github\\.com/BasedHardware/omi/releases/download/v[0-9]+\\.[0-9]+(?:\\.[0-9]+)?(?:%2[Bb]|\\+)[1-9][0-9]*-macos/omi\\.dmg$" + }, + "betaZipUrl": { + "type": "string", + "pattern": "^https://github\\.com/BasedHardware/omi/releases/download/v[0-9]+\\.[0-9]+(?:\\.[0-9]+)?(?:%2[Bb]|\\+)[1-9][0-9]*-macos/Omi\\.Beta\\.zip$" + }, + "betaDmgUrl": { + "type": "string", + "pattern": "^https://github\\.com/BasedHardware/omi/releases/download/v[0-9]+\\.[0-9]+(?:\\.[0-9]+)?(?:%2[Bb]|\\+)[1-9][0-9]*-macos/omi-beta\\.dmg$" } } } diff --git a/.github/scripts/check-deployment-concurrency.py b/.github/scripts/check-deployment-concurrency.py index 14cc865d453..07d7ce4d777 100644 --- a/.github/scripts/check-deployment-concurrency.py +++ b/.github/scripts/check-deployment-concurrency.py @@ -37,8 +37,6 @@ class LockContract: # new deploy writer cannot silently bypass the audited lock graph. LOCK_CONTRACTS = { "desktop_backend_auto_dev.yml": LockContract("desktop-backend-auto-dev"), - "desktop_promote_prod.yml": LockContract("desktop-backend-promote-prod"), - "deploy-release-ring.yml": LockContract("deploy-release-ring-${{ inputs.ring }}"), "gcp_admin.yml": LockContract( "deploy-cloud-run-omi-admin-dashboard-${{ github.ref == 'refs/heads/development' && 'development' || github.ref == 'refs/heads/main' && 'prod' || format('nondeploy-{0}', github.run_id) }}" ), @@ -91,9 +89,7 @@ class LockContract: # deployable public values. It never runs Helm, kubectl apply, or a Cloud Run # mutation; classifying credential acquisition alone as a writer would hide # that distinction and force a meaningless deploy lock. -READ_ONLY_WORKFLOW_EXEMPTIONS = { - "release-record.yml": 'kubectl -n "${ring}-omi-backend" get configmap "${ring}-omi-backend-config" -o json', -} +READ_ONLY_WORKFLOW_EXEMPTIONS = {} # Firestore index creation is a schema migration, not ordinary deploy work. # Keep a single auditable writer so backend readiness can stay read-only. @@ -248,8 +244,9 @@ def validate_serving_release_vector(name: str, text: str) -> list[str]: errors.append(f"{name}: release-vector verification must use the canonical verifier") if "--environment" not in verifier_text: errors.append(f"{name}: release-vector verification must bind an environment") - if name == "gcp_backend.yml" and "github.event.inputs.deploy_targets == 'all'" not in verifier_text: - errors.append(f"{name}: all-tier release-vector verification must not run for cloud-run-only deploys") + if name == "gcp_backend.yml": + if "github.event.inputs.deploy_targets" not in verifier_text or "--cloud-run-only" not in verifier_text: + errors.append(f"{name}: cloud-run-only promotion must use the Cloud Run-only release-vector contract") return errors diff --git a/.github/scripts/check-desktop-auto-beta-candidate.py b/.github/scripts/check-desktop-auto-beta-candidate.py index 96a833bd8d3..da84fdf4ae9 100644 --- a/.github/scripts/check-desktop-auto-beta-candidate.py +++ b/.github/scripts/check-desktop-auto-beta-candidate.py @@ -131,9 +131,6 @@ def validate(args: argparse.Namespace) -> dict: metadata = parse_metadata(release.get("body") or "") if metadata.get("channel") != "candidate" or metadata.get("isLive", "").lower() not in {"false", "0", "no"}: fail("automatic beta qualification requires channel: candidate and isLive: false") - if metadata.get("stableCandidate", "").lower() in {"true", "1", "yes"}: - fail("automatic beta qualification refuses a release already nominated for stable") - expected_version = match.group("version") expected_build = match.group("build") checks = _validate_smoke_contract( diff --git a/.github/scripts/check-desktop-prod-promotion-policy.py b/.github/scripts/check-desktop-prod-promotion-policy.py old mode 100755 new mode 100644 index 68db36d11bb..7dce9b13503 --- a/.github/scripts/check-desktop-prod-promotion-policy.py +++ b/.github/scripts/check-desktop-prod-promotion-policy.py @@ -1,185 +1,74 @@ #!/usr/bin/env python3 -"""Guard the desktop backend prod promotion workflow. - -Prod desktop backend deploys must be a manual stable-promotion action. This -check is deliberately text-based and narrow: it fails on the risky regressions -we have already seen, without requiring PyYAML in CI. -""" +"""Guard the manual qualified-artifact Stable pointer promotion.""" from pathlib import Path WORKFLOW = Path(".github/workflows/desktop_promote_prod.yml") - -def fail(message: str) -> None: - raise SystemExit(f"FAIL: {message}") - - -def require(needle: str, text: str, message: str) -> None: - if needle not in text: - fail(message) - - -def require_order(text: str, first: str, second: str, message: str) -> None: - first_index = text.find(first) - second_index = text.find(second) - if first_index == -1 or second_index == -1 or first_index >= second_index: - fail(message) - - -def workflow_triggers(text: str) -> list[str]: - lines = text.splitlines() - try: - start = next(i for i, line in enumerate(lines) if line == "on:") - except StopIteration: - fail("workflow is missing top-level on: block") - - triggers: list[str] = [] - for line in lines[start + 1 :]: - if not line.strip() or line.startswith("#"): - continue - if line and not line.startswith(" "): - break - if line.startswith(" ") and not line.startswith(" "): - triggers.append(line.strip().split(":", 1)[0]) - return triggers +REQUIRED = ( + "on:\n workflow_dispatch:", + "confirm:", + "promote-stable", + "operation:", + "expected_current_release_id:", + "expected_generation:", + "qualification_run_id:", + "environment: prod", + "Validate trusted qualification run for initial promotion", + "Fetch exact retained qualified manifest", + '"https://api.omi.me/v2/desktop/releases/$RELEASE_TAG"', + "manifest_sha256", + "Verify current beta and stable pointer compare-and-swap inputs", + '"$BASE/macos-beta"', + "check_stable_pointer_precondition.py", + "desktop_update_channels/macos-stable", + "desktop_release_manifests/$RELEASE_TAG", + "Publish immutable stable repair installer", + "Advance explicit stable pointer", + "Bridge stable for legacy desktop clients", + "Publish latest stable repair route", + "Verify exact pointer, hashes, and stable feed", + "https://api.omi.me/v2/desktop/channels/promote", + "Authorization: Bearer $ACCESS_TOKEN", + "appcast.xml?identity=stable", + "verify_stable_appcast.py", + ".event", + ".path", + "--if-generation-match=0", + 'operation\": os.environ[\"OPERATION\"]', +) + +ORDERED_STEPS = ( + "Fetch exact retained qualified manifest", + "Verify current beta and stable pointer compare-and-swap inputs", + "Publish immutable stable repair installer", + "Advance explicit stable pointer", + "Bridge stable for legacy desktop clients", + "Publish latest stable repair route", + "Verify exact pointer, hashes, and stable feed", +) + + +def validate(text: str) -> list[str]: + errors = [f"missing Stable pointer-promotion guard: {fragment}" for fragment in REQUIRED if fragment not in text] + for forbidden in ("break_glass", "Deploy Desktop Backend", "gcloud run deploy", "desktop-backend-prod-deployed"): + if forbidden in text: + errors.append(f"stable pointer promotion must not contain backend deployment or bypass path: {forbidden}") + if "\n push:" in text or "\n schedule:" in text or "\n release:" in text: + errors.append("stable pointer promotion must remain manual-only") + order = [text.find(fragment) for fragment in ORDERED_STEPS] + if -1 in order or order != sorted(order): + errors.append("stable promotion must fetch and verify retained identity before pointer mutation, then bridge and verify") + return errors def main() -> int: - text = WORKFLOW.read_text() - triggers = workflow_triggers(text) - - require("on:\n workflow_dispatch:", text, "prod promotion must be workflow_dispatch only") - if triggers != ["workflow_dispatch"]: - fail(f"prod promotion must allow only workflow_dispatch, got: {', '.join(triggers) or ''}") - require("release_tag:", text, "manual promotion must require an explicit release tag") - require("confirm:", text, "manual promotion must require an explicit confirmation input") - require("promote-stable", text, "manual promotion confirmation phrase must remain explicit") - if "\n force:" in text or "inputs.force" in text: - fail("prod promotion must stay roll-forward only; do not expose a force rollback input") - - forbidden_triggers = [ - "\n release:", - "\n schedule:", - "\n push:", - "\n pull_request:", - "\n pull_request_target:", - ] - for trigger in forbidden_triggers: - if trigger in text: - fail(f"desktop backend prod promotion must not use automatic trigger {trigger.strip()}") - - require("check-desktop-release-promotion.py", text, "workflow must run pre-release sanity checks") - require("--break-glass", text, "workflow must expose an audited emergency bypass") - require("--break-glass-confirm", text, "workflow must require typed break-glass confirmation") - require("--break-glass-reason", text, "workflow must require a break-glass audit rationale") - require("I-ACCEPT-STABLE-PROMOTION-RISK", text, "workflow must require the stable-promotion risk phrase") - require("--target-sha", text, "workflow must validate qualification and nomination against the tag SHA") - require("Audited break glass by", text, "workflow must record the actor and break-glass reason") - require( - "PROMOTION_CHECK_ARGS=(", - text, - "workflow must keep promotion-check arguments populated when break glass is disabled under set -u", - ) - if "BREAK_GLASS_ARGS=()" in text: - fail("normal stable promotion must not expand an empty optional array under set -u") - require( - 'git grep -q "OMI_DESKTOP_RELEASE_TAG" "$TARGET_SHA"', - text, - "workflow must reject tags that cannot consume release identity env vars", - ) - require( - 'git grep -q "release_tag" "$TARGET_SHA"', text, "workflow must reject tags that cannot report release identity" - ) - require( - "Preflight Omi Bot token configuration", - text, - "workflow must verify GitHub App token secrets before prod mutations", - ) - require( - "Preflight Omi Bot repository access", - text, - "workflow must verify GitHub App repository access before prod mutations", - ) - require( - "Preflight Firestore bridge release", - text, - "workflow must verify the Firestore bridge release before prod deploy", - ) - require( - "EXPECTED_VERSION: ${{ steps.plan.outputs.release_version }}", - text, - "workflow must compare Firestore bridge version to the release tag", - ) - require( - "EXPECTED_BUILD_NUMBER: ${{ steps.plan.outputs.release_build_number }}", - text, - "workflow must compare Firestore bridge build number to the release tag", - ) - require( - "Firestore bridge release version mismatch", text, "workflow must fail on mismatched Firestore bridge release" - ) - require( - "Firestore bridge release build_number mismatch", - text, - "workflow must fail on mismatched Firestore bridge build number", - ) - require("Deploy Desktop Backend to Production", text, "guard should cover the prod deploy workflow") - require( - "Verify prod backend release identity", - text, - "prod deploy must verify the backend release identity before release metadata changes", - ) - require("Promote Firestore release stable", text, "workflow must promote the Rust appcast Firestore release") - require( - "Generate Omi Bot token for release mutations", - text, - "workflow must mint the GitHub App token immediately before late release/tag mutations", - ) - require( - "mark-desktop-release-stable.py", text, "workflow must mark the release stable only after backend verification" - ) - require( - "Clear desktop update cache", - text, - "workflow should clear Python desktop update cache after stable metadata changes", - ) - require("Advance prod-tracking tag", text, "workflow must move the prod tracking tag after promotion succeeds") - require( - "This promotion workflow is roll-forward only", - text, - "workflow must reject older releases instead of force-rolling back", - ) - require("grep -qE '^v.+-macos$'", text, "prod deploys must be limited to macOS desktop release tags") - require("OMI_DESKTOP_RELEASE_TAG=", text, "prod deploy must stamp release tag into Cloud Run") - require("OMI_DESKTOP_RELEASE_SHA=", text, "prod deploy must stamp release sha into Cloud Run") - require("OMI_DESKTOP_RELEASE_CHANNEL=stable", text, "prod deploy must stamp stable channel into Cloud Run") - require( - "RELEASE_SECRET=RELEASE_SECRET:latest", text, "prod deploy must expose release secret for Firestore promotion" - ) - require_order( - text, - "Preflight Omi Bot repository access", - "Deploy Desktop Backend to Production", - "GitHub App token access must be verified before any prod deploy mutation", - ) - require_order( - text, - "Promote Firestore release stable", - "Generate Omi Bot token for release mutations", - "GitHub App token should be generated after the long deploy path to avoid expiration before tag advancement", - ) - require_order( - text, - "Generate Omi Bot token for release mutations", - "Advance prod-tracking tag", - "GitHub App token must be available for the final prod-tracking tag mutation", - ) - - if "gh release list" in text: - fail("prod promotion must not scan old releases; deploy only the event/manual target") - - print("desktop prod promotion policy OK") + errors = validate(WORKFLOW.read_text(encoding="utf-8")) + if errors: + for error in errors: + print(f"FAIL: {error}") + return 1 + print("desktop Stable pointer-promotion policy OK") return 0 diff --git a/.github/scripts/check-desktop-release-promotion.py b/.github/scripts/check-desktop-release-promotion.py deleted file mode 100755 index c1dfc6b8220..00000000000 --- a/.github/scripts/check-desktop-release-promotion.py +++ /dev/null @@ -1,127 +0,0 @@ -#!/usr/bin/env python3 -"""Validate that a nominated macOS stable candidate is safe to promote.""" - -from __future__ import annotations - -import argparse -import json -import re -import sys -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).resolve().parent)) -from desktop_release_metadata import ( # noqa: E402 - desktop_qualification_from_metadata, - fail, - parse_metadata, - require_desktop_qualification, - require_stable_candidate, - stable_candidate_from_metadata, -) - -REQUIRED_ASSETS = {"Omi.zip"} -DMG_ASSETS = {"omi.dmg", "Omi.dmg"} -TAG_RE = re.compile(r"^v(?P\d+\.\d+(?:\.\d+)?)\+(?P\d+)-macos$") -BREAK_GLASS_CONFIRM = "I-ACCEPT-STABLE-PROMOTION-RISK" - - -def write_github_output(path: str | None, values: dict[str, str]) -> None: - if not path: - return - with Path(path).open("a") as output: - for key, value in values.items(): - print(f"{key}={value}", file=output) - - -def main() -> int: - parser = argparse.ArgumentParser() - parser.add_argument("--release-json", required=True) - parser.add_argument("--release-tag", required=True) - parser.add_argument("--target-sha", required=True) - parser.add_argument("--github-output") - parser.add_argument( - "--break-glass", - action="store_true", - help="DANGER: bypass qualification and stable-candidate nomination gates", - ) - parser.add_argument( - "--break-glass-confirm", - default="", - help=f"Must be {BREAK_GLASS_CONFIRM} when --break-glass is set", - ) - parser.add_argument("--break-glass-reason", default="", help="Required audit rationale for --break-glass") - args = parser.parse_args() - - release = json.loads(Path(args.release_json).read_text(encoding="utf-8")) - tag_name = release.get("tagName") - if tag_name != args.release_tag: - fail(f"release tag mismatch: expected {args.release_tag}, got {tag_name}") - tag_match = TAG_RE.match(tag_name or "") - if not tag_match: - fail(f"{tag_name!r} is not a v*-macos release tag") - if release.get("isDraft"): - fail(f"{tag_name} is still a draft release") - if release.get("isPrerelease"): - fail(f"{tag_name} is marked as a GitHub prerelease") - - metadata = parse_metadata(release.get("body") or "") - channel = metadata.get("channel") - if channel not in {"beta", "stable"}: - fail(f"{tag_name} must be channel: beta or channel: stable before promotion, got {channel!r}") - if not metadata.get("edSignature"): - fail(f"{tag_name} is missing edSignature metadata") - if metadata.get("isLive", "").strip().lower() not in {"true", "1", "yes"}: - fail(f"{tag_name} must have isLive: true before stable promotion") - - asset_names = {asset.get("name") for asset in release.get("assets", []) if asset.get("name")} - missing_assets = sorted(REQUIRED_ASSETS - asset_names) - if missing_assets: - fail(f"{tag_name} is missing required release asset(s): {', '.join(missing_assets)}") - if not (asset_names & DMG_ASSETS): - fail(f"{tag_name} is missing a DMG release asset") - - break_glass_reason = args.break_glass_reason.strip() - break_glass = ( - args.break_glass - and args.break_glass_confirm == BREAK_GLASS_CONFIRM - and bool(break_glass_reason) - and "\n" not in break_glass_reason - and "\r" not in break_glass_reason - ) - if args.break_glass and not break_glass: - fail( - f"--break-glass requires --break-glass-confirm {BREAK_GLASS_CONFIRM} " - "and a non-empty --break-glass-reason" - ) - - qualification = desktop_qualification_from_metadata(metadata) - candidate = stable_candidate_from_metadata(metadata) - if not break_glass: - require_desktop_qualification(qualification, target_sha=args.target_sha, asset_names=asset_names) - require_stable_candidate( - candidate, - release_tag=args.release_tag, - target_sha=args.target_sha, - qualification_evidence=qualification.evidence, - ) - - write_github_output( - args.github_output, - { - "release_channel": channel, - "release_version": tag_match.group("version"), - "release_build_number": tag_match.group("build"), - "firestore_doc_id": f"v{tag_match.group('version')}+{tag_match.group('build')}", - "qualification_sha": qualification.sha, - "qualification_source": qualification.source, - "stable_candidate_by": candidate.nominated_by, - "break_glass_used": "true" if break_glass else "false", - }, - ) - mode = "break-glass" if break_glass else "nominated stable candidate" - print(f"desktop stable promotion sanity OK: {tag_name} ({mode})") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/.github/scripts/check-desktop-release-terminology.py b/.github/scripts/check-desktop-release-terminology.py deleted file mode 100755 index d9e6925c36c..00000000000 --- a/.github/scripts/check-desktop-release-terminology.py +++ /dev/null @@ -1,64 +0,0 @@ -#!/usr/bin/env python3 -"""Guard canonical desktop release terminology and nomination boundaries.""" - -from __future__ import annotations - -import re -from pathlib import Path - -ROOT = Path(__file__).resolve().parents[2] -OPERATOR_SURFACES = ( - "AGENTS.md", - "codemagic.yaml", - "desktop/macos/AGENTS.md", - "desktop/macos/docs/agent-prod-promotion-runbook.md", - "docs/doc/developer/desktop-updates.mdx", - ".github/workflows/desktop_promote_beta.yml", - ".github/workflows/desktop_promote_prod.yml", - ".github/workflows/desktop_nominate_stable_candidate.yml", - "web/admin/app/api/omi/releases/route.ts", - "web/admin/app/(protected)/dashboard/releases/page.tsx", -) - - -def main() -> int: - failures: list[str] = [] - legacy_term = re.compile(r"\bbless(?:ed|ing)?\b", re.IGNORECASE) - for relative in OPERATOR_SURFACES: - path = ROOT / relative - for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), start=1): - if legacy_term.search(line) and "legacy" not in line.lower(): - failures.append(f"{relative}:{line_number}: desktop beta qualification uses legacy terminology") - - canonical_script = ROOT / "desktop/macos/scripts/qualify-desktop-beta.sh" - legacy_script = ROOT / "desktop/macos/scripts/bless-release.sh" - if not canonical_script.is_file(): - failures.append("desktop beta qualification script is missing") - if legacy_script.exists(): - failures.append("legacy desktop beta qualification script still exists") - - nomination = (ROOT / ".github/workflows/desktop_nominate_stable_candidate.yml").read_text(encoding="utf-8") - for forbidden in ("/v2/desktop/channels/promote", "mark-desktop-release-stable.py", "desktop_promote_prod.yml -f"): - if forbidden in nomination: - failures.append(f"stable-candidate nomination must not perform promotion action {forbidden!r}") - for required in ( - "workflow_dispatch:", - "desktop_update_channels/macos-beta", - "desktop_release_manifests/${RELEASE_TAG}", - "--beta-source-sha", - "nominate-desktop-stable-candidate.py", - "Stable visibility was not changed", - ): - if required not in nomination: - failures.append(f"stable-candidate workflow is missing guard fragment {required!r}") - - if failures: - for failure in failures: - print(f"FAIL: {failure}") - return 1 - print("desktop release terminology and nomination boundaries OK") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/.github/scripts/check-direct-backend-production-admission.py b/.github/scripts/check-direct-backend-production-admission.py new file mode 100644 index 00000000000..3963c90ecbc --- /dev/null +++ b/.github/scripts/check-direct-backend-production-admission.py @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 +"""Inventory direct prod writers and require checked-out main source identity.""" + +from __future__ import annotations + +from pathlib import Path +import re + +WORKFLOWS = ( + Path(".github/workflows/gcp_backend_agent_proxy.yml"), + Path(".github/workflows/gcp_backend_listen_helm.yml"), + Path(".github/workflows/gcp_backend_pusher.yml"), + Path(".github/workflows/gcp_llm_gateway.yml"), +) +CHECKOUT = "ref: ${{ github.event.inputs.environment == 'prod' && 'main' || github.event.inputs.branch }}" +ORIGIN_MAIN_FETCH = "git fetch --no-tags origin +refs/heads/main:refs/remotes/origin/main" +ANCESTRY_GUARD = 'git merge-base --is-ancestor "$CHECKED_OUT_SHA" origin/main' +HEAD_IDENTITY = "CHECKED_OUT_SHA=$(git rev-parse HEAD)" +IMAGE_IDENTITY = 'IMAGE_TAG=$(git rev-parse --short=7 "$CHECKED_OUT_SHA")' +DIAGNOSTIC = "ERROR: checked-out HEAD $CHECKED_OUT_SHA is not an ancestor of fresh origin/main" +PERSISTED_IMAGE_IDENTITY = 'echo "IMAGE_TAG=$IMAGE_TAG" >> "$GITHUB_ENV"' + + +def validate(root: Path) -> list[str]: + errors: list[str] = [] + for relative in WORKFLOWS: + path = root / relative + text = path.read_text(encoding="utf-8") if path.exists() else "" + if CHECKOUT not in text: + errors.append(f"{relative} must reject caller refs for production checkout") + if ORIGIN_MAIN_FETCH not in text or ANCESTRY_GUARD not in text: + errors.append(f"{relative} must prove production source is on fresh origin/main") + if HEAD_IDENTITY not in text or IMAGE_IDENTITY not in text: + errors.append(f"{relative} must derive image identity from checked-out HEAD") + if DIAGNOSTIC not in text: + errors.append(f"{relative} must diagnose a rejected checked-out source identity") + if "${GITHUB_SHA::7}" in text: + errors.append(f"{relative} must not label built source with GITHUB_SHA") + if text.count("uses: actions/checkout@v7") != 1: + errors.append(f"{relative} must not check out a second source after production admission") + if text.count(HEAD_IDENTITY) != 1: + errors.append(f"{relative} must establish checked-out source identity exactly once") + persistent_image_writes = [ + line.strip() + for line in re.findall(r"(?m)^[^\n]*\bIMAGE_TAG=[^\n]*>>\s*[\"']?\$GITHUB_ENV[\"']?[^\n]*$", text) + ] + image_authorities = [] + for line in text.splitlines(): + assignment = re.match(r"^\s*(?:run:\s+)?IMAGE_TAG=(.*)$", line) + if assignment and not re.match(r"[\"']?\$\{?IMAGE_TAG\}?\b", assignment.group(1)): + image_authorities.append(line.strip()) + if ( + text.count(IMAGE_IDENTITY) != 1 + or image_authorities != [IMAGE_IDENTITY] + or persistent_image_writes != [PERSISTED_IMAGE_IDENTITY] + ): + errors.append(f"{relative} must establish its immutable image tag exactly once from checked-out HEAD") + return errors + + +if __name__ == "__main__": + raise SystemExit(1 if validate(Path(".")) else 0) diff --git a/.github/scripts/check-gcp-backend-production-boundary.py b/.github/scripts/check-gcp-backend-production-boundary.py new file mode 100644 index 00000000000..b2ce8165890 --- /dev/null +++ b/.github/scripts/check-gcp-backend-production-boundary.py @@ -0,0 +1,34 @@ +#!/usr/bin/env python3 +"""Keep production Cloud Run deployment inside its supported boundary.""" + +from __future__ import annotations + +from pathlib import Path + +WORKFLOW = Path(".github/workflows/gcp_backend.yml") +EARLY_PROD_ALL_REJECTION = 'if [[ "$DEPLOY_ENVIRONMENT" == "prod" && "$DEPLOY_TARGETS" == "all" ]]; then' +CANONICAL_AUDIENCE = '''identity_audience="$(gcloud run services describe "${{ env.SERVICE }}" \\ + --project="${{ vars.GCP_PROJECT_ID }}" --region="${{ env.REGION }}" --format='value(status.url)')"''' +AUDIENCE_DIFFERENCE_GUARD = '''[[ "$identity_audience" != "$candidate_url" ]] || { + echo 'ERROR: canonical Cloud Run IAM audience must differ from tagged candidate URL' >&2; exit 1; + }''' +CANDIDATE_REQUEST_TARGET = '--candidate-url "${{ steps.transcription-candidate.outputs.url }}"' + + +def validate(root: Path) -> list[str]: + path = root / WORKFLOW + text = path.read_text(encoding="utf-8") if path.exists() else "" + errors: list[str] = [] + if text.count(EARLY_PROD_ALL_REJECTION) != 1: + errors.append("gcp_backend.yml must reject exactly environment=prod, deploy_targets=all before side effects") + if CANONICAL_AUDIENCE not in text or "BACKEND_CLOUD_RUN_IAM_AUDIENCE" in text: + errors.append("gcp_backend.yml must derive the Cloud Run IAM audience only from backend status.url") + if AUDIENCE_DIFFERENCE_GUARD not in text: + errors.append("gcp_backend.yml must reject a canonical IAM audience equal to the tagged candidate URL") + if text.count(CANDIDATE_REQUEST_TARGET) != 1: + errors.append("gcp_backend.yml must retain the tagged candidate URL as the production probe request target") + return errors + + +if __name__ == "__main__": + raise SystemExit(1 if validate(Path(".")) else 0) diff --git a/.github/scripts/check-mobile-production-routing.py b/.github/scripts/check-mobile-production-routing.py new file mode 100644 index 00000000000..477fcfc6060 --- /dev/null +++ b/.github/scripts/check-mobile-production-routing.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +"""Fail closed when a production-family mobile package is not pinned to prod.""" + +from __future__ import annotations + +import re +from pathlib import Path + +WORKFLOWS = ( + "ios-internal-auto", + "android-internal-auto", + "ios-prod-testflight", + "android-prod-internal", + "ios-prod-patch", + "android-prod-patch", + "macos-prod-appstore", +) +DESKTOP_WORKFLOW = "omi-desktop-swift-release" +PIN = "https://api.omi.me/" +DESKTOP_PIN = "https://api.omi.me" + + +def _workflow_block(text: str, workflow: str) -> str | None: + match = re.search(rf"(?ms)^ {re.escape(workflow)}:\n(.*?)(?=^ [A-Za-z0-9_-]+:\n|\Z)", text) + return match.group(1) if match else None + + +def validate(root: Path) -> list[str]: + text = (root / "codemagic.yaml").read_text(encoding="utf-8") + errors: list[str] = [] + for workflow in WORKFLOWS: + block = _workflow_block(text, workflow) + assignments = re.findall(r"(?m)^\s*echo API_BASE_URL=([^\s]+) >> \.env\s*$", block or "") + if assignments != [PIN]: + errors.append( + f"{workflow} must contain exactly one immutable API_BASE_URL=https://api.omi.me/ assignment" + ) + desktop_block = _workflow_block(text, DESKTOP_WORKFLOW) + desktop_assignments = re.findall(r"(?m)^\s*OMI_PYTHON_API_URL:\s*[\"']?([^\"'\s]+)[\"']?\s*$", desktop_block or "") + if desktop_assignments != [DESKTOP_PIN]: + errors.append( + f"{DESKTOP_WORKFLOW} must contain exactly one immutable OMI_PYTHON_API_URL=https://api.omi.me assignment" + ) + return errors + + +if __name__ == "__main__": + raise SystemExit(1 if validate(Path(".")) else 0) diff --git a/.github/scripts/check-release-process-guards.py b/.github/scripts/check-release-process-guards.py index 28c45e05c32..8b581bee27d 100644 --- a/.github/scripts/check-release-process-guards.py +++ b/.github/scripts/check-release-process-guards.py @@ -11,7 +11,6 @@ import tempfile from pathlib import Path - ROOT = Path(__file__).resolve().parents[2] @@ -20,6 +19,7 @@ def main() -> int: errors.extend(check_desktop_codemagic_release()) errors.extend(check_desktop_preview_publishing()) errors.extend(check_desktop_qualification_runner()) + errors.extend(check_no_unprovisioned_beta_backend_hosts()) errors.extend(check_mobile_codemagic_release_triggers()) errors.extend(check_docs_workflow_scripts()) errors.extend(check_python_cli_release_version_source()) @@ -75,7 +75,6 @@ def check_desktop_codemagic_release() -> list[str]: "desktop/macos/dmg-assets/dmgbuild_settings.py", "scripts/scan-public-artifact-secrets.py", ".github/scripts/desktop-changelog.py", - ".github/scripts/desktop_qualification_dispatch.py", ] for required_file in required_files: if not (ROOT / required_file).exists(): @@ -104,10 +103,9 @@ def check_desktop_codemagic_release() -> list[str]: if "desktop_qualify_beta.yml" not in desktop_workflow_body: errors.append("desktop release must dispatch the trusted macOS qualification workflow") for required_fragment in ( - "dispatch_key=\"codemagic:${CM_TAG}\"", "for attempt in 1 2 3", "preserving immutable evidence", - "authoritative atomic", + "duplicate dispatches", ): if required_fragment not in desktop_workflow_body: errors.append(f"desktop qualification handoff is missing reliable dispatch fragment: {required_fragment}") @@ -303,12 +301,9 @@ def check_desktop_qualification_runner() -> list[str]: "--no-promote", "desktop_promote_beta.yml", "actions/create-github-app-token@v3", - "dispatch_key:", - "retry_nonce:", - "desktop_qualification_dispatch.py claim", - "desktop_qualification_dispatch.py complete", "desktop-beta-qualification-${{ inputs.release_tag }}", - "authoritative atomic serialiser", + "cancel-in-progress: false", + "safe without a second release-body claim state machine", ): if required_fragment not in text: errors.append(f"desktop qualification runner is missing required guard fragment: {required_fragment}") @@ -321,7 +316,43 @@ def check_desktop_qualification_runner() -> list[str]: "callback canary", ): if required_fragment not in candidate_gate_text: - errors.append(f"desktop beta candidate gate is missing UserNotifications callback evidence guard: {required_fragment}") + errors.append( + f"desktop beta candidate gate is missing UserNotifications callback evidence guard: {required_fragment}" + ) + return errors + + +def check_no_unprovisioned_beta_backend_hosts() -> list[str]: + """Production-family clients must share the established production backend. + + This keeps the #10090 beta-host routing regression from returning while + deliberately excluding docs and Git history, where the retired names are + useful migration evidence rather than shipped routing. + """ + hosts = ("api-beta.omi.me", "pusher-beta.omi.me", "agent-beta.omi.me") + paths = [ROOT / "app", ROOT / "desktop/macos", ROOT / "codemagic.yaml", ROOT / ".github/workflows"] + non_shipped_parts = {".build", ".dart_tool", "Pods", "test", "tests", "Tests", "test_driver"} + errors: list[str] = [] + for path in paths: + files = ( + [path] + if path.is_file() + else [ + item + for item in path.rglob("*") + if item.is_file() and not non_shipped_parts.intersection(item.relative_to(path).parts) + ] + ) + for file in files: + try: + text = file.read_text(encoding="utf-8") + except (FileNotFoundError, UnicodeDecodeError): + continue + for host in hosts: + if host in text: + errors.append( + f"shipped release source references unprovisioned beta backend host {host}: {file.relative_to(ROOT)}" + ) return errors @@ -337,7 +368,11 @@ def check_mobile_codemagic_release_triggers() -> list[str]: errors.append(f"codemagic.yaml is missing {workflow_id}") continue body = match.group("body") - if re.search(r"\n triggering:\n(?:(?!\n [A-Za-z_]).)*\n events:\n(?:(?!\n [A-Za-z_]).)*\n - push\b", body, flags=re.DOTALL): + if re.search( + r"\n triggering:\n(?:(?!\n [A-Za-z_]).)*\n events:\n(?:(?!\n [A-Za-z_]).)*\n - push\b", + body, + flags=re.DOTALL, + ): errors.append(f"{workflow_id} must not directly trigger on push; GitHub paths filtering dispatches it") workflow = ROOT / ".github/workflows/mobile_internal_auto.yml" diff --git a/.github/scripts/check_backend_deploy_source_admission.py b/.github/scripts/check_backend_deploy_source_admission.py index 82e7939957e..9966a7a568b 100644 --- a/.github/scripts/check_backend_deploy_source_admission.py +++ b/.github/scripts/check_backend_deploy_source_admission.py @@ -39,6 +39,7 @@ "github.event.inputs.mode == 'repair-traffic-only'", ) ) +MANUAL_DEPLOY_NEEDS = "needs: [validate-production-boundary, firestore_readiness]" def require_fragment(errors: list[str], text: str, fragment: str, message: str) -> None: @@ -326,6 +327,12 @@ def validate_manual_workflow(text: str) -> list[str]: return errors if folded_job_condition(firestore_job) != MANUAL_DEPLOY_CONDITION: errors.append("manual source admission must use exactly the main-ref deploy condition") + require_fragment( + errors, + firestore_job, + "needs: validate-production-boundary", + "manual source admission must wait for production-boundary validation", + ) for fragment, message in ( ("actions: 'read'", "manual source admission must read the Release Eligibility workflow result"), ( @@ -390,8 +397,8 @@ def validate_manual_workflow(text: str) -> list[str]: require_fragment( errors, deploy_job, - "needs: firestore_readiness", - "manual deployment must depend on source admission", + MANUAL_DEPLOY_NEEDS, + "manual deployment must depend on production-boundary validation and source admission", ) if folded_job_condition(deploy_job) != MANUAL_DEPLOY_CONDITION: errors.append("manual deployment must use exactly the main-ref deploy condition") diff --git a/.github/scripts/check_release_rings.py b/.github/scripts/check_release_rings.py deleted file mode 100644 index 4ffc7e31dc5..00000000000 --- a/.github/scripts/check_release_rings.py +++ /dev/null @@ -1,115 +0,0 @@ -#!/usr/bin/env python3 -"""Keep stateless release-ring workflow invariants structural and fail-closed.""" - -from __future__ import annotations - -import sys -from pathlib import Path - -ROOT = Path(__file__).resolve().parents[2] - - -def require(text: str, path: Path, fragments: tuple[str, ...]) -> list[str]: - return [ - f"{path}: missing required release-ring guard {fragment!r}" for fragment in fragments if fragment not in text - ] - - -def check() -> list[str]: - record_path = ROOT / ".github/workflows/release-record.yml" - deploy_path = ROOT / ".github/workflows/deploy-release-ring.yml" - errors: list[str] = [] - for path in (record_path, deploy_path): - if not path.exists(): - errors.append(f"{path}: release-ring workflow is missing") - continue - if errors: - return errors - - record = record_path.read_text(encoding="utf-8") - deploy = deploy_path.read_text(encoding="utf-8") - errors.extend( - require( - record, - record_path, - ( - "workflow_run:", - 'workflows: ["Release Eligibility"]', - "github.event.workflow_run.head_sha", - "github.event.workflow_run.head_branch == 'main'", - "docker/build-push-action@v7", - "backend/scripts/release_rings.py create-record", - "backend/scripts/release_rings.py resolve-secrets", - "backend/scripts/release_rings.py materialize-runtime", - "backend/charts/backend-secrets/prod_omi_backend_secrets_values.yaml", - "get-gke-credentials@v3", - "gcloud storage cp --if-generation-match=0", - "gh workflow run deploy-release-ring.yml --ref main", - "workload_identity_provider:", - ), - ) - ) - errors.extend( - require( - deploy, - deploy_path, - ( - "group: deploy-release-ring-${{ inputs.ring }}", - "environment: ${{ inputs.ring }}", - "github.ref == 'refs/heads/main'", - "RELEASE_ID: ${{ inputs.release_id }}", - "PROD_CONFIRM: ${{ inputs.confirm }}", - 'test "$PROD_CONFIRM" = "deploy-prod"', - "ref: ${{ github.sha }}", - "backend/scripts/release_rings.py validate-record", - "Require a non-held beta-soaked record before prod promotion", - "Checkout immutable release source inputs", - "ref: ${{ steps.record.outputs.git_sha }}", - "path: artifacts/release-ring/source", - "render_backend_runtime_env.py", - "Preflight all recorded GKE configuration before mutation", - "--dry-run=server", - "DRY_RUN=true", - "--update-secrets", - "backend/scripts/deploy-backend-secrets.sh", - "Verify beta endpoint prerequisites before mutation", - "cloud_run_traffic_snapshot.py capture", - "release_ring_gke_snapshot.py capture", - "--no-traffic", - "cloud_run_traffic_snapshot.py restore", - "--remove-tag \"ring-${GITHUB_RUN_ID}\"", - "--delete-missing", - "release_ring_gke_snapshot.py restore", - "wait_external_secret_refresh.py", - "active-before.json", - "steps.snapshot.outputs.active_generation", - "state=partial_mutation", - "--hold", - "--if-generation-match=\"$generation\"", - "workload_identity_provider:", - "PAGER_WEBHOOK", - "smoke_cloud_run_health.py", - "transcription-release-candidate-probe", - "Gate beta candidate on authenticated known-audio transcription", - ), - ) - ) - for path, text in ((record_path, record), (deploy_path, deploy)): - if "credentials_json:" in text: - errors.append(f"{path}: release-ring workflows must use ring-bound OIDC, not JSON credentials") - if "release_sha:" in deploy or "branch:" in deploy or "image:" in deploy: - errors.append(f"{deploy_path}: deploy accepts free-form source or image input") - for expression in ("${{ inputs.release_id }}", "${{ inputs.confirm }}"): - if deploy.count(expression) != 1: - errors.append(f"{deploy_path}: dispatch input {expression} must enter shell only through the job env") - topology_path = ROOT / "backend/deploy/release_rings.yaml" - if not topology_path.exists() or "backend: backend-beta" not in topology_path.read_text(encoding="utf-8"): - errors.append("backend/deploy/release_rings.yaml: beta must use distinct Cloud Run service identities") - return errors - - -if __name__ == "__main__": - problems = check() - for problem in problems: - print(f"ERROR: {problem}", file=sys.stderr) - raise SystemExit(bool(problems)) diff --git a/.github/scripts/check_stable_pointer_precondition.py b/.github/scripts/check_stable_pointer_precondition.py new file mode 100644 index 00000000000..907b63f96f1 --- /dev/null +++ b/.github/scripts/check_stable_pointer_precondition.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +"""Fail-closed Stable pointer preflight, including one acknowledged lost response.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + + +def _fields(path: Path) -> dict: + data = json.loads(path.read_text(encoding="utf-8")) + return data.get("fields", {}) if isinstance(data, dict) else {} + + +def _text(fields: dict, name: str) -> str: + return fields.get(name, {}).get("stringValue", "") + + +def _generation(fields: dict) -> int: + return int(fields.get("generation", {}).get("integerValue", "0")) + + +def verify(*, beta: dict, stable: dict, release_id: str, expected_release_id: str, expected_generation: int, operation: str) -> None: + current_id = _text(stable, "release_id") + current_generation = _generation(stable) + if current_id == release_id: + if current_generation != expected_generation + 1: + raise ValueError("acknowledged Stable retry has unrelated generation drift") + return + if operation == "promote" and _text(beta, "release_id") != release_id: + raise ValueError("Stable promotion requires the exact current qualified beta release ID") + if current_id != expected_release_id or current_generation != expected_generation: + raise ValueError( + f"Stable pointer CAS mismatch: expected {expected_release_id!r}/{expected_generation}, " + f"current {current_id!r}/{current_generation}" + ) + if operation == "repoint" and not current_id: + raise ValueError("repoint requires an existing stable pointer") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--beta", type=Path, required=True) + parser.add_argument("--stable", type=Path, required=True) + parser.add_argument("--release-id", required=True) + parser.add_argument("--expected-release-id", default="") + parser.add_argument("--expected-generation", type=int, required=True) + parser.add_argument("--operation", choices=("promote", "repoint"), required=True) + args = parser.parse_args() + verify(beta=_fields(args.beta), stable=_fields(args.stable), release_id=args.release_id, expected_release_id=args.expected_release_id, expected_generation=args.expected_generation, operation=args.operation) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.github/scripts/desktop_qualification_dispatch.py b/.github/scripts/desktop_qualification_dispatch.py deleted file mode 100644 index e18d98ed096..00000000000 --- a/.github/scripts/desktop_qualification_dispatch.py +++ /dev/null @@ -1,199 +0,0 @@ -#!/usr/bin/env python3 -"""Maintain the observable, deduplicated desktop-qualification handoff state. - -This state is deliberately operational metadata, not qualification evidence and -not promotion authority. The trusted qualification workflow is the only place -that can claim a dispatch key and run the expensive qualification work. -""" - -from __future__ import annotations - -import argparse -import json -import re -import sys -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).resolve().parent)) -from desktop_release_metadata import fail, parse_metadata, update_metadata # noqa: E402 - -KEY_RE = re.compile(r"^[A-Za-z0-9._:+-]{1,160}$") -TIMESTAMP_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$") -STATES = frozenset({"pending", "queued", "running", "failed", "qualified", "dispatch_failed"}) - - -def _string(value: str, label: str, *, pattern: re.Pattern[str] | None = None, limit: int = 0) -> str: - result = value.strip() - if not result or "\n" in result or "\r" in result: - fail(f"{label} must be a non-empty single-line value") - if limit and len(result) > limit: - fail(f"{label} must be at most {limit} characters") - if pattern is not None and not pattern.fullmatch(result): - fail(f"{label} has an invalid format") - return result - - -def _attempt(value: int) -> str: - if value < 0 or value > 99: - fail("dispatch attempt must be between 0 and 99") - return str(value) - - -def _values( - *, - state: str, - key: str, - attempt: int, - updated_at: str, - diagnostic: str, - run_id: str = "", - run_url: str = "", -) -> dict[str, str]: - if state not in STATES: - fail(f"unknown qualification dispatch state: {state}") - values = { - "qualificationDispatchState": state, - "qualificationDispatchKey": _string(key, "dispatch key", pattern=KEY_RE), - "qualificationDispatchAttempt": _attempt(attempt), - "qualificationDispatchUpdatedAt": _string(updated_at, "updated_at", pattern=TIMESTAMP_RE), - "qualificationDispatchDiagnostic": _string(diagnostic, "diagnostic", limit=240), - } - if run_id: - values["qualificationDispatchRunId"] = _string(run_id, "run ID", pattern=re.compile(r"^\d+$")) - if run_url: - values["qualificationDispatchRunUrl"] = _string(run_url, "run URL", limit=240) - return values - - -def _metadata(body: str) -> dict[str, str]: - return parse_metadata(body) - - -def _current_attempt(metadata: dict[str, str]) -> int: - value = metadata.get("qualificationDispatchAttempt", "0") - if not value.isdigit(): - fail("existing qualification dispatch attempt is invalid") - return int(value) - - -def claim( - body: str, - *, - key: str, - updated_at: str, - run_id: str, - run_url: str, - allow_retry: bool, -) -> tuple[str, bool, bool, str]: - """Claim one qualification run inside the authoritative workflow serialiser.""" - metadata = _metadata(body) - current_state = metadata.get("qualificationDispatchState", "") - current_key = metadata.get("qualificationDispatchKey", "") - if current_state and current_state not in STATES: - fail(f"unknown existing qualification dispatch state: {current_state}") - if bool(current_state) != bool(current_key): - fail("existing qualification dispatch claim is incomplete") - reclaimed_orphaned_claim = False - if current_state == "running": - # Every caller is inside the workflow's non-cancelling, per-release - # concurrency group. Reaching this claim means the previous runner has - # stopped, so its lingering running record is orphaned and may be - # reclaimed immediately without introducing a second active runner. - reclaimed_orphaned_claim = True - elif current_key == key and current_state == "qualified": - return body, False, True, "dispatch key already qualified" - elif current_key == key and current_state == "failed": - return body, False, False, "dispatch key already failed" - elif current_key == key and current_state in {"pending", "queued", "dispatch_failed"}: - # Older candidates may retain this observational state. The runner's - # concurrency gate is authoritative, so it can claim them once. - pass - elif current_state in {"pending", "queued"}: - return body, False, False, f"candidate already {current_state} under another dispatch key" - elif current_state == "qualified": - return body, False, True, "candidate already has completed qualification" - elif current_state in {"failed", "dispatch_failed"} and not allow_retry: - return body, False, False, "candidate has a terminal dispatch result; supply an explicit retry nonce" - attempt = _current_attempt(metadata) + 1 - values = _values( - state="running", - key=key, - attempt=attempt, - updated_at=updated_at, - diagnostic=( - "trusted qualification runner reclaimed an orphaned dispatch claim" - if reclaimed_orphaned_claim - else "trusted qualification runner claimed this dispatch key" - ), - run_id=run_id, - run_url=run_url, - ) - return update_metadata(body, values), True, False, "reclaimed orphaned claim" if reclaimed_orphaned_claim else "claimed" - - -def complete(body: str, *, key: str, updated_at: str, passed: bool) -> str: - """Write the terminal runner state without editing factual qualification keys.""" - metadata = _metadata(body) - if ( - metadata.get("qualificationDispatchKey", "") != key - or metadata.get("qualificationDispatchState", "") != "running" - ): - fail("only the active trusted dispatch key may complete qualification") - return update_metadata( - body, - _values( - state="qualified" if passed else "failed", - key=key, - attempt=_current_attempt(metadata), - updated_at=updated_at, - diagnostic=( - "trusted qualification completed; factual evidence is recorded separately" - if passed - else "trusted qualification failed; candidate remains non-live" - ), - run_id=metadata.get("qualificationDispatchRunId", ""), - run_url=metadata.get("qualificationDispatchRunUrl", ""), - ), - ) - - -def _write(path: Path, body: str, payload: dict[str, object]) -> None: - path.write_text(body, encoding="utf-8") - print(json.dumps(payload, sort_keys=True)) - - -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__) - subparsers = parser.add_subparsers(dest="command", required=True) - for command in ("claim", "complete"): - sub = subparsers.add_parser(command) - sub.add_argument("--input", required=True, type=Path) - sub.add_argument("--output", required=True, type=Path) - sub.add_argument("--key", required=True) - sub.add_argument("--updated-at", required=True) - claim_parser = subparsers.choices["claim"] - claim_parser.add_argument("--run-id", required=True) - claim_parser.add_argument("--run-url", required=True) - claim_parser.add_argument("--allow-retry", action="store_true") - complete_parser = subparsers.choices["complete"] - complete_parser.add_argument("--passed", required=True, choices=("true", "false")) - args = parser.parse_args() - body = args.input.read_text(encoding="utf-8") - if args.command == "claim": - result, should_run, should_promote, reason = claim( - body, - key=args.key, - updated_at=args.updated_at, - run_id=args.run_id, - run_url=args.run_url, - allow_retry=args.allow_retry, - ) - _write(args.output, result, {"reason": reason, "should_promote": should_promote, "should_run": should_run}) - else: - result = complete(body, key=args.key, updated_at=args.updated_at, passed=args.passed == "true") - _write(args.output, result, {"changed": True}) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/.github/scripts/desktop_qualification_evidence.py b/.github/scripts/desktop_qualification_evidence.py new file mode 100644 index 00000000000..b099509578c --- /dev/null +++ b/.github/scripts/desktop_qualification_evidence.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +"""Build and verify immutable trusted desktop qualification evidence. + +The evidence is uploaded as a GitHub Actions artifact by the trusted +qualification run. Its run ID is the authority boundary: promotion verifies +that run came from this workflow on main and then compares freshly downloaded +release bytes with this document. GitHub release bodies/assets are never the +qualification authority. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +from pathlib import Path +import re +from typing import Any + +ARTIFACTS = ("Omi.zip", "Omi.dmg", "omi.dmg", "Omi.Beta.zip", "omi-beta.dmg") +ZIP_SIGNATURES = {"Omi.zip": "edSignature", "Omi.Beta.zip": "betaEdSignature"} +SHA256_RE = re.compile(r"^[0-9a-f]{64}$") + + +def _fail(message: str) -> None: + raise ValueError(f"qualification evidence {message}") + + +def _metadata(body: str) -> dict[str, str]: + match = re.search(r"KEY_VALUE_START\s*(.*?)\s*KEY_VALUE_END", body, re.DOTALL) + if not match: + return {} + return { + key.strip(): value.strip() + for key, value in (line.split(":", 1) for line in match.group(1).splitlines() if ":" in line) + } + + +def _asset(release: dict[str, Any], name: str) -> dict[str, Any]: + matches = [item for item in release.get("assets", []) if item.get("name") == name] + if len(matches) != 1: + _fail(f"requires exactly one {name} asset") + return matches[0] + + +def _url(asset: dict[str, Any]) -> str: + value = asset.get("url") or asset.get("browser_download_url") + if not isinstance(value, str) or not value.startswith("https://"): + _fail("contains an invalid asset URL") + return value + + +def file_sha256(path: Path) -> str: + return hashlib.file_digest(path.open("rb"), "sha256").hexdigest() + + +def build_evidence( + release: dict[str, Any], release_tag: str, source_sha: str, files: dict[str, Path] +) -> dict[str, Any]: + if release.get("tagName") != release_tag: + _fail("release ID does not match requested tag") + if not re.fullmatch(r"[0-9a-f]{40}", source_sha): + _fail("source SHA is not an exact 40-character SHA") + gate = files.pop("__candidate_gate__") + candidate_gate = json.loads(gate.read_text(encoding="utf-8")) + if candidate_gate.get("passed") is not True or candidate_gate.get("release_tag") != release_tag: + _fail("was not created after the passing candidate gate") + metadata = _metadata(str(release.get("body") or "")) + artifacts: dict[str, dict[str, str]] = {} + stable_dmg = [name for name in ("Omi.dmg", "omi.dmg") if name in files] + if len(stable_dmg) != 1: + _fail("requires exactly one stable DMG file") + required = {"Omi.zip", stable_dmg[0], "Omi.Beta.zip", "omi-beta.dmg"} + if set(files) != required: + _fail("does not contain all four qualified artifacts") + for name, path in files.items(): + if not path.is_file(): + _fail(f"is missing downloaded {name}") + asset = _asset(release, name) + digest = file_sha256(path) + published = str(asset.get("digest") or "").removeprefix("sha256:") + if published and published != digest: + _fail(f"{name} differs from its published candidate digest") + item = {"url": _url(asset), "sha256": digest} + signature_key = ZIP_SIGNATURES.get(name) + if signature_key: + signature = metadata.get(signature_key, "") + if not signature: + _fail(f"is missing {signature_key}") + item["signature"] = signature + artifacts[name] = item + return { + "schema_version": 1, + "release_id": release_tag, + "source_sha": source_sha, + "source_qualification": { + "passed": True, + "tier": "T2", + "subject": "source-built named-bundle", + "fault_evidence": "trusted qualification runner", + }, + "signed_artifact_verification": { + "passed": True, + "subject": "exact signed ZIP/DMG bytes", + "checks": ["sha256", "Sparkle signature", "notarization", "signed smoke"], + }, + "artifacts": artifacts, + } + + +def verify_evidence( + evidence: dict[str, Any], release: dict[str, Any], release_tag: str, source_sha: str, digests: dict[str, str] +) -> None: + if ( + evidence.get("schema_version") != 1 + or evidence.get("release_id") != release_tag + or evidence.get("source_sha") != source_sha + ): + _fail("release ID or source SHA does not match the trusted run") + source_qualification = evidence.get("source_qualification") + signed_artifacts = evidence.get("signed_artifact_verification") + if not isinstance(source_qualification, dict) or source_qualification.get("passed") is not True or source_qualification.get("tier") != "T2": + _fail("does not prove source-built named-bundle T2 qualification") + if not isinstance(signed_artifacts, dict) or signed_artifacts.get("passed") is not True: + _fail("does not prove exact signed artifact verification") + if signed_artifacts.get("subject") != "exact signed ZIP/DMG bytes": + _fail("must not claim signed production bytes ran T2") + artifacts = evidence.get("artifacts") + if not isinstance(artifacts, dict): + _fail("does not contain artifacts") + metadata = _metadata(str(release.get("body") or "")) + expected_names = set(digests) + if set(artifacts) != expected_names: + _fail("artifact set differs from the downloaded release") + for name, actual_sha in digests.items(): + item = artifacts.get(name) + if not isinstance(item, dict) or item.get("sha256") != actual_sha or not SHA256_RE.fullmatch(actual_sha): + _fail(f"{name} hash differs from trusted evidence") + if item.get("url") != _url(_asset(release, name)): + _fail(f"{name} URL differs from trusted evidence") + signature_key = ZIP_SIGNATURES.get(name) + if signature_key and item.get("signature") != metadata.get(signature_key): + _fail(f"{name} signature differs from trusted evidence") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("command", choices=("build", "verify")) + parser.add_argument("--release-json", required=True) + parser.add_argument("--release-tag", required=True) + parser.add_argument("--source-sha", required=True) + parser.add_argument("--evidence", required=True) + parser.add_argument("--candidate-gate") + parser.add_argument("--asset", action="append", default=[]) + args = parser.parse_args() + release = json.loads(Path(args.release_json).read_text(encoding="utf-8")) + files: dict[str, Path] = {} + for raw in args.asset: + name, sep, path = raw.partition("=") + if not sep or name not in ARTIFACTS: + raise SystemExit("--asset must be NAME=PATH for a required release artifact") + files[name] = Path(path) + if args.command == "build": + if not args.candidate_gate: + raise SystemExit("build requires --candidate-gate") + files["__candidate_gate__"] = Path(args.candidate_gate) + result = build_evidence(release, args.release_tag, args.source_sha, files) + Path(args.evidence).write_text(json.dumps(result, sort_keys=True, indent=2) + "\n", encoding="utf-8") + else: + evidence = json.loads(Path(args.evidence).read_text(encoding="utf-8")) + verify_evidence( + evidence, + release, + args.release_tag, + args.source_sha, + {name: file_sha256(path) for name, path in files.items()}, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.github/scripts/desktop_release_doctor.py b/.github/scripts/desktop_release_doctor.py index 0ead501b6be..c64e16fa527 100644 --- a/.github/scripts/desktop_release_doctor.py +++ b/.github/scripts/desktop_release_doctor.py @@ -25,7 +25,6 @@ format_summary, ) - REPOSITORY = "BasedHardware/omi" PRIVATE_KEY_VALUE_BLOCK_RE = re.compile(r"", re.DOTALL) STALE_STABLE_PROSE_RE = re.compile(r"stable\s+(?:remains\s+)?blocked", re.IGNORECASE) @@ -69,10 +68,6 @@ def _metadata_from_release_body(body: object) -> tuple[dict[str, str], bool]: "qualifiedBetaSha", "qualifiedBetaTier", "qualifiedBetaEvidence", - "stableCandidate", - "stableCandidateTag", - "stableCandidateSha", - "stableCandidateQualificationEvidence", ) public_metadata = {key: metadata[key] for key in safe_keys if key in metadata} human_prose = PRIVATE_KEY_VALUE_BLOCK_RE.sub("", body) @@ -115,13 +110,17 @@ def _http_json(url: str, *, token: str | None = None) -> object: if token: headers["Authorization"] = f"Bearer {token}" request = urllib.request.Request(url, headers=headers) - with urllib.request.urlopen(request, timeout=30) as response: # nosec B310: URLs are fixed release-control endpoints. + with urllib.request.urlopen( + request, timeout=30 + ) as response: # nosec B310: URLs are fixed release-control endpoints. return json.loads(response.read().decode("utf-8")) def _http_text(url: str) -> str: request = urllib.request.Request(url, headers={"Accept": "application/xml,application/json,text/plain"}) - with urllib.request.urlopen(request, timeout=30) as response: # nosec B310: URLs are fixed release-control endpoints. + with urllib.request.urlopen( + request, timeout=30 + ) as response: # nosec B310: URLs are fixed release-control endpoints. return response.read().decode("utf-8") @@ -179,7 +178,9 @@ def _project_release_summary(release: object) -> dict[str, object]: return _unavailable("GitHub release response was not an object") metadata, stale_human_prose = _metadata_from_release_body(release.get("body")) assets = release.get("assets", []) - asset_names = [asset.get("name") for asset in assets if isinstance(asset, dict) and isinstance(asset.get("name"), str)] + asset_names = [ + asset.get("name") for asset in assets if isinstance(asset, dict) and isinstance(asset.get("name"), str) + ] return { "tag_name": _optional_string(release.get("tagName")), "is_draft": release.get("isDraft") is True, @@ -193,7 +194,11 @@ def _project_release_summary(release: object) -> dict[str, object]: def _safe_static_json(bucket: str, name: str) -> dict[str, object]: object_path = f"{bucket.rstrip('/')}/{name}" result = subprocess.run( - ("gcloud", "storage", "cat", object_path), check=False, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE + ("gcloud", "storage", "cat", object_path), + check=False, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, ) if result.returncode != 0: return _unavailable(f"static object unavailable: {name}") @@ -277,7 +282,16 @@ def collect_snapshot( "desktop_release_manifests", release_tag, access_token, - allowed_fields=("release_id", "platform", "version", "build_number", "source_sha", "zip_sha256", "dmg_sha256", "qualification"), + allowed_fields=( + "release_id", + "platform", + "version", + "build_number", + "source_sha", + "zip_sha256", + "dmg_sha256", + "qualification", + ), ), "pointers": { "beta": _safe_firestore_document( @@ -285,14 +299,30 @@ def collect_snapshot( "desktop_update_channels", "macos-beta", access_token, - allowed_fields=("platform", "channel", "release_id", "version", "build_number", "generation", "updated_at"), + allowed_fields=( + "platform", + "channel", + "release_id", + "version", + "build_number", + "generation", + "updated_at", + ), ), "stable": _safe_firestore_document( project_id, "desktop_update_channels", "macos-stable", access_token, - allowed_fields=("platform", "channel", "release_id", "version", "build_number", "generation", "updated_at"), + allowed_fields=( + "platform", + "channel", + "release_id", + "version", + "build_number", + "generation", + "updated_at", + ), ), }, "legacy_release": _safe_firestore_document( @@ -311,9 +341,15 @@ def collect_snapshot( "stable": _safe_static_json(bucket, "stable/latest.json"), }, "backend": _safe_health(service_url), - "tracking": {"desktop_backend_prod_deployed_sha": tracking_sha} if tracking_sha else _unavailable("tracking tag is absent"), + "tracking": ( + {"desktop_backend_prod_deployed_sha": tracking_sha} + if tracking_sha + else _unavailable("tracking tag is absent") + ), "codemagic": _unavailable("Codemagic API is not yet a release-control dependency"), - "metrics": {name: _unavailable("metric collection is advisory work not yet wired") for name in METRIC_CONTRACTS}, + "metrics": { + name: _unavailable("metric collection is advisory work not yet wired") for name in METRIC_CONTRACTS + }, } diff --git a/.github/scripts/desktop_release_manifest.py b/.github/scripts/desktop_release_manifest.py index cd921ba8372..089c8d6fedf 100644 --- a/.github/scripts/desktop_release_manifest.py +++ b/.github/scripts/desktop_release_manifest.py @@ -20,7 +20,6 @@ from typing import Any, NoReturn from urllib.parse import unquote, urlparse - SCHEMA_VERSION = 1 PLATFORM = "macos" BACKEND_MODES = frozenset({"app_only", "backend_required"}) @@ -39,6 +38,11 @@ "dmg_url", "dmg_sha256", "ed_signature", + "beta_zip_url", + "beta_zip_sha256", + "beta_dmg_url", + "beta_dmg_sha256", + "beta_ed_signature", "qualification_evidence_asset", "qualification_evidence_sha256", "qualification_tier", @@ -223,6 +227,11 @@ def validate_manifest(value: object) -> dict[str, Any]: _require_release_asset_url(manifest, "dmg_url", release_id=release_id, asset_name="omi.dmg") _require_sha256(manifest, "dmg_sha256") _require_string(manifest, "ed_signature") + _require_release_asset_url(manifest, "beta_zip_url", release_id=release_id, asset_name="Omi.Beta.zip") + _require_sha256(manifest, "beta_zip_sha256") + _require_release_asset_url(manifest, "beta_dmg_url", release_id=release_id, asset_name="omi-beta.dmg") + _require_sha256(manifest, "beta_dmg_sha256") + _require_string(manifest, "beta_ed_signature") evidence_asset = _require_string(manifest, "qualification_evidence_asset") if not EVIDENCE_ASSET_RE.fullmatch(evidence_asset): diff --git a/.github/scripts/desktop_release_metadata.py b/.github/scripts/desktop_release_metadata.py index 037d9d866d5..723011dfcf9 100644 --- a/.github/scripts/desktop_release_metadata.py +++ b/.github/scripts/desktop_release_metadata.py @@ -19,20 +19,6 @@ class DesktopQualification: source: str -@dataclass(frozen=True) -class StableCandidate: - nominated: bool - tag: str - sha: str - nominated_at: str - nominated_by: str - rationale: str - qualification_evidence: str - soak_review: str - telemetry_review: str - release_notes_review: str - - def fail(message: str) -> NoReturn: raise SystemExit(f"FAIL: {message}") @@ -108,52 +94,6 @@ def require_desktop_qualification( fail(f"release is missing qualification evidence asset {qualification.evidence!r}") -def stable_candidate_from_metadata(metadata: dict[str, str]) -> StableCandidate: - return StableCandidate( - nominated=metadata.get("stableCandidate", "").strip().lower() in TRUE_VALUES, - tag=metadata.get("stableCandidateTag", "").strip(), - sha=metadata.get("stableCandidateSha", "").strip(), - nominated_at=metadata.get("stableCandidateAt", "").strip(), - nominated_by=metadata.get("stableCandidateBy", "").strip(), - rationale=metadata.get("stableCandidateRationale", "").strip(), - qualification_evidence=metadata.get("stableCandidateQualificationEvidence", "").strip(), - soak_review=metadata.get("stableCandidateSoakReview", "").strip(), - telemetry_review=metadata.get("stableCandidateTelemetryReview", "").strip(), - release_notes_review=metadata.get("stableCandidateReleaseNotesReview", "").strip(), - ) - - -def require_stable_candidate( - candidate: StableCandidate, - *, - release_tag: str, - target_sha: str, - qualification_evidence: str, -) -> None: - if not candidate.nominated: - fail("release must be explicitly nominated as a stable candidate") - required = { - "stableCandidateTag": candidate.tag, - "stableCandidateSha": candidate.sha, - "stableCandidateAt": candidate.nominated_at, - "stableCandidateBy": candidate.nominated_by, - "stableCandidateRationale": candidate.rationale, - "stableCandidateQualificationEvidence": candidate.qualification_evidence, - "stableCandidateSoakReview": candidate.soak_review, - "stableCandidateTelemetryReview": candidate.telemetry_review, - "stableCandidateReleaseNotesReview": candidate.release_notes_review, - } - missing = [key for key, value in required.items() if not value] - if missing: - fail(f"stable-candidate nomination is missing: {', '.join(missing)}") - if candidate.tag != release_tag: - fail("stable-candidate tag does not match the requested release") - if candidate.sha != target_sha: - fail("stable-candidate SHA does not match the release tag commit") - if candidate.qualification_evidence != qualification_evidence: - fail("stable-candidate nomination does not reference the current qualification evidence") - - def update_metadata(body: str, values: dict[str, str]) -> str: """Replace or append keys inside the release metadata block.""" if any("\n" in value or "\r" in value for value in values.values()): diff --git a/.github/scripts/desktop_repair_installer.py b/.github/scripts/desktop_repair_installer.py index 189155f620e..f172ee4d331 100644 --- a/.github/scripts/desktop_repair_installer.py +++ b/.github/scripts/desktop_repair_installer.py @@ -54,9 +54,8 @@ def build_repair_bundle(manifest: dict[str, Any], bucket: str) -> dict[str, Any] release_path = quote(release_id, safe=".-_~+") public_base = f"https://storage.googleapis.com/{quote(bucket_name, safe='.-_~')}" - artifact_object = f"stable/{release_id}/Omi.dmg" repair_object = f"stable/{release_id}/repair.json" - installer_url = f"{public_base}/stable/{release_path}/Omi.dmg" + installer_url = _required_string(manifest, "dmg_url") repair_manifest_url = f"{public_base}/stable/{release_path}/repair.json" repair = { @@ -95,7 +94,6 @@ def build_repair_bundle(manifest: dict[str, Any], bucket: str) -> dict[str, Any] """ return { - "artifact_object": artifact_object, "repair_object": repair_object, "repair": repair, "latest": latest, diff --git a/.github/scripts/fixtures/desktop_release_manifest/v1/app-only.json b/.github/scripts/fixtures/desktop_release_manifest/v1/app-only.json index f20a407b28e..94c80493a5d 100644 --- a/.github/scripts/fixtures/desktop_release_manifest/v1/app-only.json +++ b/.github/scripts/fixtures/desktop_release_manifest/v1/app-only.json @@ -10,6 +10,11 @@ "dmg_url": "https://github.com/BasedHardware/omi/releases/download/v0.12.71%2B12071-macos/omi.dmg", "dmg_sha256": "sha256:3333333333333333333333333333333333333333333333333333333333333333", "ed_signature": "sparkle-ed25519-signature", + "beta_zip_url": "https://github.com/BasedHardware/omi/releases/download/v0.12.71%2B12071-macos/Omi.Beta.zip", + "beta_zip_sha256": "sha256:5555555555555555555555555555555555555555555555555555555555555555", + "beta_dmg_url": "https://github.com/BasedHardware/omi/releases/download/v0.12.71%2B12071-macos/omi-beta.dmg", + "beta_dmg_sha256": "sha256:6666666666666666666666666666666666666666666666666666666666666666", + "beta_ed_signature": "beta-sparkle-ed25519-signature", "qualification_evidence_asset": "qualification-evidence-0.12.71+12071-20260712T010203Z.json", "qualification_evidence_sha256": "sha256:4444444444444444444444444444444444444444444444444444444444444444", "qualification_tier": "T2", diff --git a/.github/scripts/fixtures/desktop_release_manifest/v1/backend-required.json b/.github/scripts/fixtures/desktop_release_manifest/v1/backend-required.json index c8e9a059a1f..e942d055b1d 100644 --- a/.github/scripts/fixtures/desktop_release_manifest/v1/backend-required.json +++ b/.github/scripts/fixtures/desktop_release_manifest/v1/backend-required.json @@ -10,6 +10,11 @@ "dmg_url": "https://github.com/BasedHardware/omi/releases/download/v0.12.72%2B12072-macos/omi.dmg", "dmg_sha256": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", "ed_signature": "sparkle-ed25519-signature", + "beta_zip_url": "https://github.com/BasedHardware/omi/releases/download/v0.12.72%2B12072-macos/Omi.Beta.zip", + "beta_zip_sha256": "sha256:5555555555555555555555555555555555555555555555555555555555555555", + "beta_dmg_url": "https://github.com/BasedHardware/omi/releases/download/v0.12.72%2B12072-macos/omi-beta.dmg", + "beta_dmg_sha256": "sha256:6666666666666666666666666666666666666666666666666666666666666666", + "beta_ed_signature": "beta-sparkle-ed25519-signature", "qualification_evidence_asset": "qualification-evidence-0.12.72+12072-20260712T020304Z.json", "qualification_evidence_sha256": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", "qualification_tier": "T2", diff --git a/.github/scripts/nominate-desktop-stable-candidate.py b/.github/scripts/nominate-desktop-stable-candidate.py deleted file mode 100755 index 38d5239f0f7..00000000000 --- a/.github/scripts/nominate-desktop-stable-candidate.py +++ /dev/null @@ -1,116 +0,0 @@ -#!/usr/bin/env python3 -"""Validate and record a desktop stable-candidate nomination.""" - -from __future__ import annotations - -import argparse -import json -import sys -from datetime import datetime, timezone -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).resolve().parent)) -from desktop_release_metadata import ( # noqa: E402 - desktop_qualification_from_metadata, - fail, - parse_metadata, - require_desktop_qualification, - update_metadata, -) - - -def _single_line(name: str, value: str) -> str: - value = value.strip() - if not value: - fail(f"{name} is required") - if "\n" in value or "\r" in value: - fail(f"{name} must be a single-line value") - return value - - -def nominate( - release: dict, - *, - release_tag: str, - target_sha: str, - beta_release_id: str, - beta_source_sha: str, - nominator: str, - rationale: str, - soak_review: str, - telemetry_review: str, - release_notes_review: str, - nominated_at: str, -) -> str: - if release.get("tagName") != release_tag: - fail(f"release tag mismatch: expected {release_tag}, got {release.get('tagName')}") - if release.get("isDraft") or release.get("isPrerelease"): - fail("stable candidates must be published GitHub releases") - if beta_release_id != release_tag: - fail(f"beta pointer references {beta_release_id!r}, not requested release {release_tag!r}") - if beta_source_sha != target_sha: - fail(f"beta manifest source SHA {beta_source_sha!r} does not match tag commit {target_sha!r}") - - metadata = parse_metadata(release.get("body") or "") - if metadata.get("channel") != "beta": - fail(f"stable candidates must currently be channel: beta, got {metadata.get('channel')!r}") - if metadata.get("isLive", "").strip().lower() not in {"true", "1", "yes"}: - fail("stable candidates must be live on the beta channel") - - asset_names = {asset.get("name") for asset in release.get("assets", []) if asset.get("name")} - qualification = desktop_qualification_from_metadata(metadata) - require_desktop_qualification(qualification, target_sha=target_sha, asset_names=asset_names) - - values = { - "stableCandidate": "true", - "stableCandidateTag": release_tag, - "stableCandidateSha": target_sha, - "stableCandidateAt": _single_line("nominated_at", nominated_at), - "stableCandidateBy": _single_line("nominator", nominator), - "stableCandidateRationale": _single_line("rationale", rationale), - "stableCandidateQualificationEvidence": qualification.evidence, - "stableCandidateSoakReview": _single_line("soak_review", soak_review), - "stableCandidateTelemetryReview": _single_line("telemetry_review", telemetry_review), - "stableCandidateReleaseNotesReview": _single_line("release_notes_review", release_notes_review), - } - return update_metadata(release.get("body") or "", values) - - -def main() -> int: - parser = argparse.ArgumentParser() - parser.add_argument("--release-json", required=True) - parser.add_argument("--release-tag", required=True) - parser.add_argument("--target-sha", required=True) - parser.add_argument("--beta-release-id", required=True) - parser.add_argument("--beta-source-sha", required=True) - parser.add_argument("--nominator", required=True) - parser.add_argument("--rationale", required=True) - parser.add_argument("--soak-review", required=True) - parser.add_argument("--telemetry-review", required=True) - parser.add_argument("--release-notes-review", required=True) - parser.add_argument("--nominated-at") - parser.add_argument("--output", required=True) - args = parser.parse_args() - - release = json.loads(Path(args.release_json).read_text(encoding="utf-8")) - nominated_at = args.nominated_at or datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") - body = nominate( - release, - release_tag=args.release_tag, - target_sha=args.target_sha, - beta_release_id=args.beta_release_id, - beta_source_sha=args.beta_source_sha, - nominator=args.nominator, - rationale=args.rationale, - soak_review=args.soak_review, - telemetry_review=args.telemetry_review, - release_notes_review=args.release_notes_review, - nominated_at=nominated_at, - ) - Path(args.output).write_text(body, encoding="utf-8") - print(f"stable candidate nominated: {args.release_tag}") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/.github/scripts/prepare-desktop-beta-promotion.py b/.github/scripts/prepare-desktop-beta-promotion.py index 2648352c5e8..82308c08618 100755 --- a/.github/scripts/prepare-desktop-beta-promotion.py +++ b/.github/scripts/prepare-desktop-beta-promotion.py @@ -11,11 +11,10 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) from desktop_release_metadata import ( # noqa: E402 - desktop_qualification_from_metadata, fail, parse_metadata, - require_desktop_qualification, ) +from desktop_qualification_evidence import verify_evidence # noqa: E402 TAG_RE = re.compile(r"^v(?P\d+\.\d+(?:\.\d+)?)\+(?P\d+)-macos$") @@ -34,8 +33,10 @@ def prepare_manifest( zip_sha256: str, dmg_sha256: str, *, + beta_zip_sha256: str, + beta_dmg_sha256: str, + qualification_evidence: dict, allow_stable_channel: bool = False, - allow_unqualified: bool = False, ) -> dict: if release.get("tagName") != release_tag: fail(f"release tag mismatch: expected {release_tag}, got {release.get('tagName')}") @@ -61,31 +62,43 @@ def prepare_manifest( accepted = "candidate, beta, or stable" if allow_stable_channel else "candidate or beta" fail(f"release channel must be {accepted}, got {channel!r}") - asset_names = {asset.get("name") for asset in release.get("assets", []) if asset.get("name")} - qualification = desktop_qualification_from_metadata(metadata) - if not allow_unqualified: - require_desktop_qualification(qualification, target_sha=target_sha, asset_names=asset_names) - zip_asset = _asset(release, {"Omi.zip"}) dmg_asset = _asset(release, {"Omi.dmg", "omi.dmg"}) + beta_zip_asset = _asset(release, {"Omi.Beta.zip"}) + beta_dmg_asset = _asset(release, {"omi-beta.dmg"}) signature = metadata.get("edSignature", "").strip() - if not signature: - fail("release is missing edSignature") + beta_signature = metadata.get("betaEdSignature", "").strip() + if not signature or not beta_signature: + fail("release is missing stable or beta Sparkle signature") + try: + verify_evidence( + qualification_evidence, + release, + release_tag, + target_sha, + { + "Omi.zip": zip_sha256, + dmg_asset["name"]: dmg_sha256, + "Omi.Beta.zip": beta_zip_sha256, + "omi-beta.dmg": beta_dmg_sha256, + }, + ) + except ValueError as exc: + raise ValueError(str(exc)) from exc changelog = [item.strip() for item in metadata.get("changelog", "").split("|") if item.strip()] version = match.group("version") build = int(match.group("build")) - if allow_unqualified: - qualification_manifest = {"passed": False, "tier": "emergency"} - else: - qualification_manifest = {"passed": True, "tier": "T2", "evidence_asset": qualification.evidence} - if not allow_unqualified and qualification.source == "legacy": - # Preserve the immutable manifest shape used by releases registered - # before canonical qualification metadata existed. This keeps exact - # beta-promotion retries idempotent. - qualification_manifest["blessed_at"] = qualification.qualified_at - elif not allow_unqualified: - qualification_manifest["qualified_at"] = qualification.qualified_at + qualification_manifest = { + "passed": True, + "tier": "T2", + "source": "trusted_github_actions_artifact", + "evidence_asset": metadata.get("qualifiedBetaEvidence") + or f"desktop-qualification-evidence-{release_tag}", + "source_subject": "source-built named-bundle", + "signed_artifact_subject": "exact signed ZIP/DMG bytes", + "signed_artifact_checks": ["sha256", "Sparkle signature", "notarization", "signed smoke"], + } return { "release_id": release_tag, @@ -94,13 +107,18 @@ def prepare_manifest( "build_number": build, "zip_url": zip_asset.get("url"), "dmg_url": dmg_asset.get("url"), + "beta_zip_url": beta_zip_asset.get("url"), + "beta_dmg_url": beta_dmg_asset.get("url"), "ed_signature": signature, + "beta_ed_signature": beta_signature, "published_at": release.get("publishedAt"), "changelog": changelog, "mandatory": metadata.get("mandatory", "false").lower() in {"true", "1", "yes"}, "source_sha": target_sha, "zip_sha256": zip_sha256, "dmg_sha256": dmg_sha256, + "beta_zip_sha256": beta_zip_sha256, + "beta_dmg_sha256": beta_dmg_sha256, "qualification": qualification_manifest, } @@ -112,20 +130,25 @@ def main() -> int: parser.add_argument("--target-sha", required=True) parser.add_argument("--zip-sha256", required=True) parser.add_argument("--dmg-sha256", required=True) + parser.add_argument("--beta-zip-sha256", required=True) + parser.add_argument("--beta-dmg-sha256", required=True) + parser.add_argument("--qualification-evidence", required=True) parser.add_argument("--allow-stable-channel", action="store_true") - parser.add_argument("--allow-unqualified", action="store_true") parser.add_argument("--output", required=True) args = parser.parse_args() release = json.loads(Path(args.release_json).read_text()) + evidence = json.loads(Path(args.qualification_evidence).read_text()) manifest = prepare_manifest( release, args.release_tag, args.target_sha, args.zip_sha256, args.dmg_sha256, + beta_zip_sha256=args.beta_zip_sha256, + beta_dmg_sha256=args.beta_dmg_sha256, + qualification_evidence=evidence, allow_stable_channel=args.allow_stable_channel, - allow_unqualified=args.allow_unqualified, ) Path(args.output).write_text(json.dumps(manifest, indent=2) + "\n") print(f"qualified beta manifest prepared: {args.release_tag}") diff --git a/.github/scripts/product_file_line_count_ratchet_baseline/desktop-swift-root.json b/.github/scripts/product_file_line_count_ratchet_baseline/desktop-swift-root.json index eb4abe08a2c..97012a4849f 100644 --- a/.github/scripts/product_file_line_count_ratchet_baseline/desktop-swift-root.json +++ b/.github/scripts/product_file_line_count_ratchet_baseline/desktop-swift-root.json @@ -2,7 +2,7 @@ "files": { "desktop/macos/Desktop/Sources/AuthService.swift": 3294, "desktop/macos/Desktop/Sources/CloudConnectorFormAutomation.swift": 1678, - "desktop/macos/Desktop/Sources/DesktopAutomationBridge.swift": 4240, + "desktop/macos/Desktop/Sources/DesktopAutomationBridge.swift": 4239, "desktop/macos/Desktop/Sources/MemoryExportService.swift": 1578, "desktop/macos/Desktop/Sources/OmiApp.swift": 1638 }, diff --git a/.github/scripts/test_check_backend_deploy_source_admission.py b/.github/scripts/test_check_backend_deploy_source_admission.py index 2650561a526..b98dbdee1bc 100644 --- a/.github/scripts/test_check_backend_deploy_source_admission.py +++ b/.github/scripts/test_check_backend_deploy_source_admission.py @@ -373,11 +373,36 @@ def test_manual_workflow_rejects_fail_open_ref_or_mode_conditions(self) -> None: self.mutate( root, CHECKER.MANUAL_WORKFLOW_PATH, - " deploy:\n needs: firestore_readiness\n if: >-\n github.ref == 'refs/heads/main' &&\n github.event.inputs.mode == 'deploy'\n", - " deploy:\n needs: firestore_readiness\n if: >-\n github.ref == 'refs/heads/main' &&\n github.event.inputs.mode == 'deploy' || true\n", + " deploy:\n needs: [validate-production-boundary, firestore_readiness]\n if: >-\n github.ref == 'refs/heads/main' &&\n github.event.inputs.mode == 'deploy'\n", + " deploy:\n needs: [validate-production-boundary, firestore_readiness]\n if: >-\n github.ref == 'refs/heads/main' &&\n github.event.inputs.mode == 'deploy' || true\n", ) self.assertIn("manual deployment must use exactly the main-ref deploy condition", CHECKER.validate(root)) + def test_manual_workflow_rejects_boundary_dependency_bypasses(self) -> None: + root = self.fixture_root() + self.mutate( + root, + CHECKER.MANUAL_WORKFLOW_PATH, + " needs: validate-production-boundary\n", + "", + ) + self.assertIn( + "manual source admission must wait for production-boundary validation", + CHECKER.validate(root), + ) + + root = self.fixture_root() + self.mutate( + root, + CHECKER.MANUAL_WORKFLOW_PATH, + "needs: [validate-production-boundary, firestore_readiness]", + "needs: firestore_readiness", + ) + self.assertIn( + "manual deployment must depend on production-boundary validation and source admission", + CHECKER.validate(root), + ) + def test_auto_workflow_rejects_github_sha_or_incomplete_source_binding(self) -> None: root = self.fixture_root() self.mutate( diff --git a/.github/scripts/test_check_desktop_release_promotion.py b/.github/scripts/test_check_desktop_release_promotion.py deleted file mode 100644 index 85eb626d6f4..00000000000 --- a/.github/scripts/test_check_desktop_release_promotion.py +++ /dev/null @@ -1,143 +0,0 @@ -#!/usr/bin/env python3 -"""Tests for the desktop stable-promotion gate.""" - -from __future__ import annotations - -import json -import subprocess -import sys -from pathlib import Path - -REPO_ROOT = Path(__file__).resolve().parents[2] -SCRIPT = REPO_ROOT / ".github/scripts/check-desktop-release-promotion.py" -TAG = "v11.0+11000-macos" -SHA = "a" * 40 -EVIDENCE = "qualification-evidence.json" - - -def run_check(release: dict, *extra: str) -> subprocess.CompletedProcess[str]: - release_path = Path("/tmp/desktop-release-test.json") - release_path.write_text(json.dumps(release), encoding="utf-8") - return subprocess.run( - [ - sys.executable, - str(SCRIPT), - "--release-json", - str(release_path), - "--release-tag", - TAG, - "--target-sha", - SHA, - *extra, - ], - text=True, - capture_output=True, - ) - - -def base_release(metadata: dict[str, str]) -> dict: - lines = ["KEY_VALUE_START", *[f"{key}: {value}" for key, value in metadata.items()], "KEY_VALUE_END"] - return { - "tagName": TAG, - "isDraft": False, - "isPrerelease": False, - "body": "\n".join(lines), - "assets": [{"name": "Omi.zip"}, {"name": "omi.dmg"}, {"name": EVIDENCE}], - } - - -def common_metadata() -> dict[str, str]: - return {"channel": "beta", "isLive": "true", "edSignature": "sig"} - - -def nomination_metadata() -> dict[str, str]: - return { - "stableCandidate": "true", - "stableCandidateTag": TAG, - "stableCandidateSha": SHA, - "stableCandidateAt": "2026-07-10T12:00:00Z", - "stableCandidateBy": "release-operator", - "stableCandidateRationale": "beta soak and launch criteria passed", - "stableCandidateQualificationEvidence": EVIDENCE, - "stableCandidateSoakReview": "24h beta soak reviewed", - "stableCandidateTelemetryReview": "crash and update telemetry reviewed", - "stableCandidateReleaseNotesReview": "stable rollup reviewed", - } - - -def test_canonical_qualification_and_nomination_pass() -> None: - release = base_release( - { - **common_metadata(), - "qualifiedBeta": "true", - "qualifiedBetaAt": "2026-07-10T10:00:00Z", - "qualifiedBetaSha": SHA, - "qualifiedBetaTier": "2", - "qualifiedBetaEvidence": EVIDENCE, - **nomination_metadata(), - } - ) - result = run_check(release) - assert result.returncode == 0, result.stderr - - -def test_legacy_qualification_remains_valid() -> None: - release = base_release( - { - **common_metadata(), - "blessed": "true", - "blessedAt": "2026-07-10T10:00:00Z", - "blessedSha": SHA, - "blessedTier": "2", - "blessedEvidence": EVIDENCE, - **nomination_metadata(), - } - ) - result = run_check(release) - assert result.returncode == 0, result.stderr - - -def test_qualified_beta_without_nomination_fails() -> None: - release = base_release( - { - **common_metadata(), - "qualifiedBeta": "true", - "qualifiedBetaAt": "2026-07-10T10:00:00Z", - "qualifiedBetaSha": SHA, - "qualifiedBetaTier": "2", - "qualifiedBetaEvidence": EVIDENCE, - } - ) - result = run_check(release) - assert result.returncode != 0 - assert "nominated" in result.stderr, result.stderr - - -def test_break_glass_requires_confirmation_and_reason() -> None: - release = base_release(common_metadata()) - rejected = run_check(release, "--break-glass", "--break-glass-confirm", "nope") - assert rejected.returncode != 0 - missing_reason = run_check( - release, - "--break-glass", - "--break-glass-confirm", - "I-ACCEPT-STABLE-PROMOTION-RISK", - ) - assert missing_reason.returncode != 0 - accepted = run_check( - release, - "--break-glass", - "--break-glass-confirm", - "I-ACCEPT-STABLE-PROMOTION-RISK", - "--break-glass-reason", - "urgent security release", - ) - assert accepted.returncode == 0, accepted.stderr - - -if __name__ == "__main__": - test_canonical_qualification_and_nomination_pass() - test_legacy_qualification_remains_valid() - test_qualified_beta_without_nomination_fails() - test_break_glass_requires_confirmation_and_reason() - print("check-desktop-release-promotion tests OK") diff --git a/.github/scripts/test_check_direct_backend_production_admission.py b/.github/scripts/test_check_direct_backend_production_admission.py new file mode 100644 index 00000000000..2d1b34578a0 --- /dev/null +++ b/.github/scripts/test_check_direct_backend_production_admission.py @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +"""Static inventory for direct production backend workflow source admission.""" + +from __future__ import annotations + +import importlib.util +from pathlib import Path +import shutil +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parents[2] +MODULE_PATH = ROOT / ".github/scripts/check-direct-backend-production-admission.py" +SPEC = importlib.util.spec_from_file_location("check_direct_backend_production_admission", MODULE_PATH) +assert SPEC and SPEC.loader +CHECKER = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(CHECKER) + + +class DirectBackendProductionAdmissionTests(unittest.TestCase): + def test_inventory_is_admitted_and_uses_checked_out_source_identity(self) -> None: + self.assertEqual(CHECKER.validate(ROOT), []) + + def test_each_direct_writer_rejects_each_source_admission_mutation_in_a_fresh_fixture(self) -> None: + mutations = { + "caller_ref": (CHECKER.CHECKOUT, "ref: ${{ github.event.inputs.branch }}"), + "fresh_origin": (CHECKER.ORIGIN_MAIN_FETCH, "git fetch --no-tags origin main"), + "ancestry": (CHECKER.ANCESTRY_GUARD, "git merge-base --is-ancestor HEAD origin/main"), + "head_identity": (CHECKER.HEAD_IDENTITY, "CHECKED_OUT_SHA=${GITHUB_SHA}"), + "image_identity": (CHECKER.IMAGE_IDENTITY, "IMAGE_TAG=${GITHUB_SHA::7}"), + "diagnostic": (CHECKER.DIAGNOSTIC, "ERROR: source admission failed"), + "second_checkout": ("\n", "\n - uses: actions/checkout@v7\n"), + "late_image_override": ("\n", "\n - name: late image override\n run: IMAGE_TAG=latest\n"), + "late_persistent_image_override": ( + "\n", + "\n - name: late persistent image override\n" + " run: echo \"IMAGE_TAG=latest\" >> \"$GITHUB_ENV\"\n", + ), + } + for relative in CHECKER.WORKFLOWS: + for name, (expected, replacement) in mutations.items(): + with self.subTest(workflow=relative, mutation=name), tempfile.TemporaryDirectory() as directory: + root = Path(directory) + for fixture_relative in CHECKER.WORKFLOWS: + source = ROOT / fixture_relative + target = root / fixture_relative + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(source, target) + target = root / relative + text = target.read_text(encoding="utf-8") + self.assertIn(expected, text) + target.write_text(text.replace(expected, replacement, 1), encoding="utf-8") + self.assertTrue(CHECKER.validate(root)) + + def test_rejects_multiple_image_tag_authorities_and_late_checkouts(self) -> None: + mutations = ( + " - name: second image authority\n run: IMAGE_TAG=deadbee\n", + " - uses: actions/checkout@v7\n with:\n ref: main\n", + ) + for relative in CHECKER.WORKFLOWS: + for mutation in mutations: + with self.subTest(workflow=relative, mutation=mutation), tempfile.TemporaryDirectory() as directory: + root = Path(directory) + for fixture_relative in CHECKER.WORKFLOWS: + source = ROOT / fixture_relative + target = root / fixture_relative + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(source, target) + target = root / relative + target.write_text(target.read_text(encoding="utf-8") + "\n" + mutation, encoding="utf-8") + self.assertTrue(CHECKER.validate(root)) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/scripts/test_check_gcp_backend_production_boundary.py b/.github/scripts/test_check_gcp_backend_production_boundary.py new file mode 100644 index 00000000000..f056bcc2568 --- /dev/null +++ b/.github/scripts/test_check_gcp_backend_production_boundary.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +"""Mutation-sensitive contract for the production Cloud Run-only boundary.""" + +from __future__ import annotations + +import importlib.util +from pathlib import Path +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parents[2] +MODULE_PATH = ROOT / ".github/scripts/check-gcp-backend-production-boundary.py" +SPEC = importlib.util.spec_from_file_location("check_gcp_backend_production_boundary", MODULE_PATH) +assert SPEC and SPEC.loader +CHECKER = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(CHECKER) + + +class GcpBackendProductionBoundaryTests(unittest.TestCase): + def test_current_workflow_preserves_the_boundary(self) -> None: + self.assertEqual(CHECKER.validate(ROOT), []) + + def test_rejects_exact_production_boundary_and_canonical_audience_mutations(self) -> None: + original = (ROOT / ".github/workflows/gcp_backend.yml").read_text(encoding="utf-8") + mutations = { + "allows_prod_all": ( + 'if [[ "$DEPLOY_ENVIRONMENT" == "prod" && "$DEPLOY_TARGETS" == "all" ]]; then', + 'if [[ "$DEPLOY_ENVIRONMENT" == "prod" && "$DEPLOY_TARGETS" == "cloud-run-only" ]]; then', + ), + "candidate_as_audience": ( + 'identity_audience="$(gcloud run services describe "${{ env.SERVICE }}"', + 'identity_audience="$candidate_url"\n # gcloud run services describe "${{ env.SERVICE }}"', + ), + "environment_override": ( + 'identity_audience="$(gcloud run services describe "${{ env.SERVICE }}"', + 'identity_audience="${{ vars.BACKEND_CLOUD_RUN_IAM_AUDIENCE }}"\n' + ' # gcloud run services describe "${{ env.SERVICE }}"', + ), + "missing_audience_distinction": ( + '[[ "$identity_audience" != "$candidate_url" ]] || {\n' + " echo 'ERROR: canonical Cloud Run IAM audience must differ from tagged candidate URL' >&2; exit 1;\n" + ' }', + 'true', + ), + "candidate_target_replaced": ( + '--candidate-url "${{ steps.transcription-candidate.outputs.url }}"', + '--candidate-url "${{ steps.transcription-candidate.outputs.identity_audience }}"', + ), + } + for name, (expected, replacement) in mutations.items(): + with self.subTest(mutation=name), tempfile.TemporaryDirectory() as directory: + root = Path(directory) + workflow = root / ".github/workflows/gcp_backend.yml" + workflow.parent.mkdir(parents=True) + self.assertIn(expected, original) + workflow.write_text(original.replace(expected, replacement, 1), encoding="utf-8") + self.assertTrue(CHECKER.validate(root)) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/scripts/test_check_mobile_production_routing.py b/.github/scripts/test_check_mobile_production_routing.py new file mode 100644 index 00000000000..eed750b4f4f --- /dev/null +++ b/.github/scripts/test_check_mobile_production_routing.py @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 +"""Regression contract for production-family Codemagic backend routing.""" + +from __future__ import annotations + +import importlib.util +from pathlib import Path +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parents[2] +MODULE_PATH = ROOT / ".github/scripts/check-mobile-production-routing.py" +SPEC = importlib.util.spec_from_file_location("check_mobile_production_routing", MODULE_PATH) +assert SPEC and SPEC.loader +CHECKER = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(CHECKER) + + +class MobileProductionRoutingContractTests(unittest.TestCase): + def test_current_config_is_pinned(self) -> None: + self.assertEqual(CHECKER.validate(ROOT), []) + + def test_rejects_production_family_mutation_to_development_or_arbitrary_url(self) -> None: + original = (ROOT / "codemagic.yaml").read_text(encoding="utf-8") + for bad in ("https://api.omi.dev/", "https://staging.example.test/"): + with self.subTest(bad=bad), tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "codemagic.yaml").write_text( + original.replace('API_BASE_URL=https://api.omi.me/', f'API_BASE_URL={bad}', 1), encoding="utf-8" + ) + self.assertTrue(CHECKER.validate(root)) + + def test_each_production_workflow_rejects_missing_wrong_or_conflicting_assignment(self) -> None: + original = (ROOT / "codemagic.yaml").read_text(encoding="utf-8") + for workflow in CHECKER.WORKFLOWS: + block = CHECKER._workflow_block(original, workflow) + self.assertIsNotNone(block) + assert block is not None + for mutation in ("missing", "wrong", "conflicting"): + with self.subTest(workflow=workflow, mutation=mutation), tempfile.TemporaryDirectory() as directory: + root = Path(directory) + if mutation == "missing": + changed = block.replace("echo API_BASE_URL=https://api.omi.me/ >> .env\n", "", 1) + elif mutation == "wrong": + changed = block.replace( + "echo API_BASE_URL=https://api.omi.me/", + "echo API_BASE_URL=https://api.omi.dev/", + 1, + ) + else: + changed = block.replace( + "echo API_BASE_URL=https://api.omi.me/ >> .env", + "echo API_BASE_URL=https://api.omi.me/ >> .env\n echo API_BASE_URL=https://staging.example.test/ >> .env", + 1, + ) + (root / "codemagic.yaml").write_text(original.replace(block, changed, 1), encoding="utf-8") + self.assertTrue(CHECKER.validate(root)) + + def test_desktop_release_rejects_staging_or_duplicate_late_python_api_assignment(self) -> None: + original = (ROOT / "codemagic.yaml").read_text(encoding="utf-8") + block = CHECKER._workflow_block(original, CHECKER.DESKTOP_WORKFLOW) + self.assertIsNotNone(block) + assert block is not None + for mutation in ("staging", "duplicate_late"): + with self.subTest(mutation=mutation), tempfile.TemporaryDirectory() as directory: + root = Path(directory) + if mutation == "staging": + changed = block.replace( + 'OMI_PYTHON_API_URL: "https://api.omi.me"', + 'OMI_PYTHON_API_URL: "https://staging.example.test"', + 1, + ) + else: + changed = block.replace( + 'OMI_PYTHON_API_URL: "https://api.omi.me"', + 'OMI_PYTHON_API_URL: "https://api.omi.me"\n' + ' OMI_PYTHON_API_URL: "https://staging.example.test"', + 1, + ) + (root / "codemagic.yaml").write_text(original.replace(block, changed, 1), encoding="utf-8") + self.assertTrue(CHECKER.validate(root)) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/scripts/test_check_release_process_guards.py b/.github/scripts/test_check_release_process_guards.py new file mode 100644 index 00000000000..56126855da6 --- /dev/null +++ b/.github/scripts/test_check_release_process_guards.py @@ -0,0 +1,35 @@ +from __future__ import annotations + +import importlib.util +from pathlib import Path + + +SCRIPT = Path(__file__).with_name("check-release-process-guards.py") +SPEC = importlib.util.spec_from_file_location("check_release_process_guards", SCRIPT) +assert SPEC is not None and SPEC.loader is not None +GUARDS = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(GUARDS) + + +def test_beta_backend_guard_ignores_fixtures_but_rejects_shipped_source(tmp_path, monkeypatch): + shipped = tmp_path / "app/lib/env.dart" + fixture = tmp_path / "app/test/env_test.dart" + desktop_fixture = tmp_path / "desktop/macos/Tests/RoutingTests.swift" + workflows = tmp_path / ".github/workflows" + for path in (shipped, fixture, desktop_fixture): + path.parent.mkdir(parents=True, exist_ok=True) + workflows.mkdir(parents=True) + (tmp_path / "codemagic.yaml").write_text("", encoding="utf-8") + + fixture.write_text("const retired = 'https://api-beta.omi.me';", encoding="utf-8") + desktop_fixture.write_text('let retired = "pusher-beta.omi.me"', encoding="utf-8") + shipped.write_text("const api = 'https://api.omi.me';", encoding="utf-8") + monkeypatch.setattr(GUARDS, "ROOT", tmp_path) + + assert GUARDS.check_no_unprovisioned_beta_backend_hosts() == [] + + shipped.write_text("const api = 'https://agent-beta.omi.me';", encoding="utf-8") + assert GUARDS.check_no_unprovisioned_beta_backend_hosts() == [ + "shipped release source references unprovisioned beta backend host " + "agent-beta.omi.me: app/lib/env.dart" + ] diff --git a/.github/scripts/test_check_release_rings.py b/.github/scripts/test_check_release_rings.py deleted file mode 100644 index 2c8a5880cff..00000000000 --- a/.github/scripts/test_check_release_rings.py +++ /dev/null @@ -1,50 +0,0 @@ -from __future__ import annotations - -import importlib.util -import shutil -import sys -from pathlib import Path - -MODULE_PATH = Path(__file__).with_name("check_release_rings.py") -SPEC = importlib.util.spec_from_file_location("check_release_rings", MODULE_PATH) -assert SPEC and SPEC.loader -checker = importlib.util.module_from_spec(SPEC) -sys.modules[SPEC.name] = checker -SPEC.loader.exec_module(checker) - - -def test_checked_in_release_ring_workflows_are_guarded() -> None: - assert checker.check() == [] - - -def test_missing_guard_is_reported(tmp_path: Path, monkeypatch) -> None: - workflow_dir = tmp_path / ".github" / "workflows" - workflow_dir.mkdir(parents=True) - scripts_dir = tmp_path / ".github" / "scripts" - scripts_dir.mkdir(parents=True) - (workflow_dir / "release-record.yml").write_text("workflow_run:\n", encoding="utf-8") - (workflow_dir / "deploy-release-ring.yml").write_text("workflow_dispatch:\n", encoding="utf-8") - monkeypatch.setattr(checker, "ROOT", tmp_path) - - assert any("missing required release-ring guard" in error for error in checker.check()) - - -def test_dispatch_release_id_cannot_be_interpolated_into_shell(tmp_path: Path, monkeypatch) -> None: - for relative in ( - Path(".github/workflows/release-record.yml"), - Path(".github/workflows/deploy-release-ring.yml"), - Path("backend/deploy/release_rings.yaml"), - ): - destination = tmp_path / relative - destination.parent.mkdir(parents=True, exist_ok=True) - shutil.copy2(checker.ROOT / relative, destination) - deploy_path = tmp_path / ".github/workflows/deploy-release-ring.yml" - deploy_path.write_text( - deploy_path.read_text(encoding="utf-8").replace( - 'records/${RELEASE_ID}.json', 'records/${{ inputs.release_id }}.json' - ), - encoding="utf-8", - ) - monkeypatch.setattr(checker, "ROOT", tmp_path) - - assert any("must enter shell only through the job env" in error for error in checker.check()) diff --git a/.github/scripts/test_desktop_qualification_dispatch.py b/.github/scripts/test_desktop_qualification_dispatch.py deleted file mode 100644 index 4d02f8de5bb..00000000000 --- a/.github/scripts/test_desktop_qualification_dispatch.py +++ /dev/null @@ -1,233 +0,0 @@ -"""Behavioral tests for the desktop qualification dispatch ownership protocol.""" - -from __future__ import annotations - -import importlib.util -from pathlib import Path - -SCRIPT = Path(__file__).with_name("desktop_qualification_dispatch.py") -SPEC = importlib.util.spec_from_file_location("desktop_qualification_dispatch", SCRIPT) -assert SPEC is not None and SPEC.loader is not None -dispatch = importlib.util.module_from_spec(SPEC) -SPEC.loader.exec_module(dispatch) - -KEY = "codemagic:v0.12.64+12064-macos" -NOW = "2026-07-20T10:00:00Z" - - -def candidate_body(extra: str = "") -> str: - return "\n".join( - [ - "", - ] - ) - - -def test_serialized_successor_reclaims_an_orphaned_running_claim_immediately() -> None: - running, should_run, should_promote, reason = dispatch.claim( - candidate_body(), - key=KEY, - updated_at=NOW, - run_id="42", - run_url="https://github.com/BasedHardware/omi/actions/runs/42", - allow_retry=False, - ) - assert should_run and not should_promote and reason == "claimed" - - reclaimed, should_run, should_promote, reason = dispatch.claim( - running, - key=KEY, - updated_at="2026-07-20T10:00:01Z", - run_id="43", - run_url="https://github.com/BasedHardware/omi/actions/runs/43", - allow_retry=False, - ) - assert should_run - assert not should_promote - assert reason == "reclaimed orphaned claim" - assert "qualificationDispatchAttempt: 2" in reclaimed - assert "qualificationDispatchRunId: 43" in reclaimed - - -def test_terminal_failure_requires_an_explicit_new_retry_nonce() -> None: - running, should_run, should_promote, _ = dispatch.claim( - candidate_body(), - key=KEY, - updated_at=NOW, - run_id="42", - run_url="https://github.com/BasedHardware/omi/actions/runs/42", - allow_retry=False, - ) - assert should_run and not should_promote - failed = dispatch.complete(running, key=KEY, updated_at=NOW, passed=False) - - unchanged, should_run, should_promote, reason = dispatch.claim( - failed, - key="manual:v0.12.64+12064-macos:retry-1", - updated_at=NOW, - run_id="43", - run_url="https://github.com/BasedHardware/omi/actions/runs/43", - allow_retry=False, - ) - assert unchanged == failed - assert not should_run - assert not should_promote - assert "explicit retry nonce" in reason - - retried, should_run, should_promote, _ = dispatch.claim( - failed, - key="manual:v0.12.64+12064-macos:retry-1", - updated_at=NOW, - run_id="43", - run_url="https://github.com/BasedHardware/omi/actions/runs/43", - allow_retry=True, - ) - assert should_run and not should_promote - assert "qualificationDispatchKey: manual:v0.12.64+12064-macos:retry-1" in retried - - -def test_legacy_dispatch_failure_still_allows_its_same_key_to_claim_once() -> None: - dispatch_failed = candidate_body( - "qualificationDispatchState: dispatch_failed\n" - f"qualificationDispatchKey: {KEY}\n" - "qualificationDispatchAttempt: 3\n" - "qualificationDispatchUpdatedAt: 2026-07-20T09:00:00Z\n" - "qualificationDispatchDiagnostic: dispatch confirmation timed out" - ) - running, should_run, should_promote, reason = dispatch.claim( - dispatch_failed, - key=KEY, - updated_at=NOW, - run_id="44", - run_url="https://github.com/BasedHardware/omi/actions/runs/44", - allow_retry=False, - ) - assert should_run and not should_promote and reason == "claimed" - assert "qualificationDispatchState: running" in running - - -def test_qualified_claim_retries_request_promotion_without_rerunning() -> None: - running, should_run, should_promote, _ = dispatch.claim( - candidate_body(), - key=KEY, - updated_at=NOW, - run_id="42", - run_url="https://github.com/BasedHardware/omi/actions/runs/42", - allow_retry=False, - ) - assert should_run and not should_promote - qualified = dispatch.complete(running, key=KEY, updated_at=NOW, passed=True) - - unchanged, should_run, should_promote, reason = dispatch.claim( - qualified, - key=KEY, - updated_at="2026-07-20T10:05:00Z", - run_id="43", - run_url="https://github.com/BasedHardware/omi/actions/runs/43", - allow_retry=False, - ) - - assert unchanged == qualified - assert not should_run - assert should_promote - assert reason == "dispatch key already qualified" - - -def test_unknown_existing_dispatch_state_is_rejected() -> None: - body = candidate_body("qualificationDispatchState: abandoned\nqualificationDispatchKey: " + KEY) - try: - dispatch.claim( - body, - key=KEY, - updated_at=NOW, - run_id="42", - run_url="https://github.com/BasedHardware/omi/actions/runs/42", - allow_retry=False, - ) - except SystemExit as exc: - assert "unknown existing" in str(exc) - else: - raise AssertionError("unknown existing dispatch state should fail closed") - - -def test_orphaned_claim_from_another_dispatch_key_does_not_block_recovery() -> None: - running, should_run, should_promote, _ = dispatch.claim( - candidate_body(), - key="manual:v0.12.64+12064-macos:retry-1", - updated_at=NOW, - run_id="42", - run_url="https://github.com/BasedHardware/omi/actions/runs/42", - allow_retry=True, - ) - assert should_run and not should_promote - - recovered, should_run, should_promote, reason = dispatch.claim( - running, - key=KEY, - updated_at="2026-07-20T10:00:01Z", - run_id="43", - run_url="https://github.com/BasedHardware/omi/actions/runs/43", - allow_retry=False, - ) - assert should_run - assert not should_promote - assert reason == "reclaimed orphaned claim" - assert f"qualificationDispatchKey: {KEY}" in recovered - - -def test_dispatch_status_cannot_modify_factual_qualification_evidence() -> None: - body = candidate_body() - running, should_run, should_promote, _ = dispatch.claim( - body, - key=KEY, - updated_at=NOW, - run_id="42", - run_url="https://github.com/BasedHardware/omi/actions/runs/42", - allow_retry=False, - ) - assert should_run and not should_promote - qualified = dispatch.complete(running, key=KEY, updated_at=NOW, passed=True) - - assert "qualifiedBetaEvidence: qualification-evidence-failed.json" in qualified - assert "qualificationDispatchState: qualified" in qualified - - -def test_orphaned_claim_recovery_does_not_depend_on_the_predecessor_timestamp() -> None: - body = candidate_body( - "qualificationDispatchState: running\n" - f"qualificationDispatchKey: {KEY}\n" - "qualificationDispatchAttempt: 1\n" - "qualificationDispatchUpdatedAt: not-a-timestamp\n" - "qualificationDispatchDiagnostic: runner claimed this key" - ) - reclaimed, should_run, should_promote, reason = dispatch.claim( - body, - key=KEY, - updated_at="2026-07-20T10:00:01Z", - run_id="43", - run_url="https://github.com/BasedHardware/omi/actions/runs/43", - allow_retry=False, - ) - assert should_run - assert not should_promote - assert reason == "reclaimed orphaned claim" - assert "qualificationDispatchUpdatedAt: 2026-07-20T10:00:01Z" in reclaimed - - -if __name__ == "__main__": - test_serialized_successor_reclaims_an_orphaned_running_claim_immediately() - test_terminal_failure_requires_an_explicit_new_retry_nonce() - test_legacy_dispatch_failure_still_allows_its_same_key_to_claim_once() - test_qualified_claim_retries_request_promotion_without_rerunning() - test_unknown_existing_dispatch_state_is_rejected() - test_orphaned_claim_from_another_dispatch_key_does_not_block_recovery() - test_dispatch_status_cannot_modify_factual_qualification_evidence() - test_orphaned_claim_recovery_does_not_depend_on_the_predecessor_timestamp() - print("desktop qualification dispatch tests OK") diff --git a/.github/scripts/test_desktop_release_manifest.py b/.github/scripts/test_desktop_release_manifest.py index 9724bcc759e..3c6781168f5 100644 --- a/.github/scripts/test_desktop_release_manifest.py +++ b/.github/scripts/test_desktop_release_manifest.py @@ -12,7 +12,6 @@ import tempfile import unittest - SCRIPT_DIR = Path(__file__).resolve().parent MODULE_PATH = SCRIPT_DIR / "desktop_release_manifest.py" FIXTURES = SCRIPT_DIR / "fixtures" / "desktop_release_manifest" / "v1" @@ -123,15 +122,9 @@ def test_rejects_unicode_digits_whitespace_signature_and_non_z_time(self) -> Non def test_artifact_urls_are_bound_to_repository_release_and_asset(self) -> None: invalid_urls = { "arbitrary host": "https://example.com/Omi.zip", - "other release": ( - "https://github.com/BasedHardware/omi/releases/download/v0.12.70%2B12070-macos/Omi.zip" - ), - "other repository": ( - "https://github.com/attacker/omi/releases/download/v0.12.71%2B12071-macos/Omi.zip" - ), - "wrong asset": ( - "https://github.com/BasedHardware/omi/releases/download/v0.12.71%2B12071-macos/other.zip" - ), + "other release": ("https://github.com/BasedHardware/omi/releases/download/v0.12.70%2B12070-macos/Omi.zip"), + "other repository": ("https://github.com/attacker/omi/releases/download/v0.12.71%2B12071-macos/Omi.zip"), + "wrong asset": ("https://github.com/BasedHardware/omi/releases/download/v0.12.71%2B12071-macos/other.zip"), } for label, value in invalid_urls.items(): with self.subTest(label=label): @@ -151,7 +144,7 @@ def test_canonical_digest_is_stable_across_key_order_and_whitespace(self) -> Non ) self.assertEqual( manifest_contract.manifest_digest(manifest), - "sha256:bec1b723483d46f415ff57d86b7fc59c3fc8e9faf484043834665453aafc10e7", + "sha256:9d2e2de26473a24adf498b3a814a532c7ee56d6ae18746d2444d9d9ca9c20beb", ) def test_valid_identity_and_digest_mutations_fail_detached_verification(self) -> None: @@ -167,6 +160,8 @@ def test_valid_identity_and_digest_mutations_fail_detached_verification(self) -> "build_number": 12073, "zip_url": "https://github.com/BasedHardware/omi/releases/download/v0.12.73%2B12073-macos/Omi.zip", "dmg_url": "https://github.com/BasedHardware/omi/releases/download/v0.12.73%2B12073-macos/omi.dmg", + "beta_zip_url": "https://github.com/BasedHardware/omi/releases/download/v0.12.73%2B12073-macos/Omi.Beta.zip", + "beta_dmg_url": "https://github.com/BasedHardware/omi/releases/download/v0.12.73%2B12073-macos/omi-beta.dmg", } ) release_identity["compatibility_contract"].update( diff --git a/.github/scripts/test_desktop_release_manifest_schema.py b/.github/scripts/test_desktop_release_manifest_schema.py new file mode 100644 index 00000000000..757db37324e --- /dev/null +++ b/.github/scripts/test_desktop_release_manifest_schema.py @@ -0,0 +1,28 @@ +from __future__ import annotations + +import json +from pathlib import Path + +from jsonschema import Draft202012Validator + + +SCHEMA_PATH = Path(__file__).resolve().parents[1] / "schemas/desktop-release-manifest-v1.schema.json" + + +def test_beta_artifact_schema_accepts_beta_names_and_keeps_stable_names_distinct(): + schema = json.loads(SCHEMA_PATH.read_text(encoding="utf-8")) + definitions = schema["$defs"] + stable_zip = "https://github.com/BasedHardware/omi/releases/download/v0.12.64+12064-macos/Omi.zip" + stable_dmg = "https://github.com/BasedHardware/omi/releases/download/v0.12.64+12064-macos/omi.dmg" + beta_zip = "https://github.com/BasedHardware/omi/releases/download/v0.12.64+12064-macos/Omi.Beta.zip" + beta_dmg = "https://github.com/BasedHardware/omi/releases/download/v0.12.64+12064-macos/omi-beta.dmg" + + Draft202012Validator(definitions["zipUrl"]).validate(stable_zip) + Draft202012Validator(definitions["dmgUrl"]).validate(stable_dmg) + Draft202012Validator(definitions["betaZipUrl"]).validate(beta_zip) + Draft202012Validator(definitions["betaDmgUrl"]).validate(beta_dmg) + + assert list(Draft202012Validator(definitions["zipUrl"]).iter_errors(beta_zip)) + assert list(Draft202012Validator(definitions["dmgUrl"]).iter_errors(beta_dmg)) + assert schema["properties"]["beta_zip_url"]["$ref"] == "#/$defs/betaZipUrl" + assert schema["properties"]["beta_dmg_url"]["$ref"] == "#/$defs/betaDmgUrl" diff --git a/.github/scripts/test_stable_promotion_verifiers.py b/.github/scripts/test_stable_promotion_verifiers.py new file mode 100644 index 00000000000..5c5d0303b82 --- /dev/null +++ b/.github/scripts/test_stable_promotion_verifiers.py @@ -0,0 +1,69 @@ +from __future__ import annotations + +import importlib.util +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] + + +def _load(name: str): + path = ROOT / ".github/scripts" / name + spec = importlib.util.spec_from_file_location(name.replace("-", "_"), path) + assert spec and spec.loader + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +APPCAST = _load("verify_stable_appcast.py") +POINTER = _load("check_stable_pointer_precondition.py") + + +def _fields(release_id: str, generation: int) -> dict: + return {"release_id": {"stringValue": release_id}, "generation": {"integerValue": str(generation)}} + + +class StablePromotionVerifierTests(unittest.TestCase): + def test_lost_response_retry_accepts_only_the_expected_next_generation(self): + POINTER.verify( + beta=_fields("target", 4), + stable=_fields("target", 8), + release_id="target", + expected_release_id="previous", + expected_generation=7, + operation="promote", + ) + with self.assertRaisesRegex(ValueError, "unrelated generation drift"): + POINTER.verify( + beta=_fields("target", 4), + stable=_fields("target", 9), + release_id="target", + expected_release_id="previous", + expected_generation=7, + operation="promote", + ) + + def test_stable_appcast_ignores_beta_item_but_rejects_two_default_items(self): + manifest = { + "build_number": 9, + "version": "1.0", + "zip_url": "https://example.test/Omi.zip", + "ed_signature": "sig", + } + xml = ''' +91.0 +91.0beta''' + with tempfile.TemporaryDirectory() as directory: + feed = Path(directory) / "feed.xml" + feed.write_text(xml, encoding="utf-8") + APPCAST.verify(manifest, feed) + feed.write_text(xml.replace("beta", ""), encoding="utf-8") + with self.assertRaisesRegex(ValueError, "default/non-beta"): + APPCAST.verify(manifest, feed) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/scripts/verify_stable_appcast.py b/.github/scripts/verify_stable_appcast.py new file mode 100644 index 00000000000..5d49df2fbba --- /dev/null +++ b/.github/scripts/verify_stable_appcast.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 +"""Verify the default Stable appcast item for one retained manifest.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from xml.etree import ElementTree as ET + +SPARKLE = "http://www.andymatuschak.org/xml-namespaces/sparkle" + + +def verify(manifest: dict, feed: Path) -> None: + root = ET.parse(feed).getroot() + matches = [] + for item in root.findall(".//item"): + enclosure = item.find("enclosure") + if enclosure is None or item.findtext(f"{{{SPARKLE}}}channel") == "beta": + continue + if ( + item.findtext(f"{{{SPARKLE}}}version") == str(manifest["build_number"]) + and item.findtext(f"{{{SPARKLE}}}shortVersionString") == manifest["version"] + and enclosure.get("url") == manifest["zip_url"] + and enclosure.get(f"{{{SPARKLE}}}edSignature") == manifest["ed_signature"] + ): + matches.append(item) + if len(matches) != 1: + raise ValueError("stable appcast must contain exactly one default/non-beta immutable requested item") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--manifest", type=Path, required=True) + parser.add_argument("--feed", type=Path, required=True) + args = parser.parse_args() + verify(json.loads(args.manifest.read_text(encoding="utf-8")), args.feed) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.github/workflows/deploy-release-ring.yml b/.github/workflows/deploy-release-ring.yml deleted file mode 100644 index e05324b0a69..00000000000 --- a/.github/workflows/deploy-release-ring.yml +++ /dev/null @@ -1,518 +0,0 @@ -name: Deploy release ring - -on: - workflow_dispatch: - inputs: - ring: - description: Serving ring to deploy - required: true - type: choice - options: [beta, prod] - release_id: - description: Immutable GCS release-record identifier - required: true - type: string - confirm: - description: Required only for prod; type deploy-prod - required: false - default: '' - type: string - -# GitHub Actions is the lock. The GCS active-pointer generation is a backstop -# against a manual/break-glass writer, not a second lock service. -concurrency: - group: deploy-release-ring-${{ inputs.ring }} - cancel-in-progress: false - -jobs: - deploy: - if: github.ref == 'refs/heads/main' - environment: ${{ inputs.ring }} - permissions: - contents: read - id-token: write - issues: write - runs-on: ubuntu-latest - env: - REGION: us-central1 - RELEASE_BUCKET: ${{ vars.RELEASE_RECORDS_BUCKET }} - RELEASE_RUNTIME_PROJECT: ${{ vars.RELEASE_RUNTIME_PROJECT_ID }} - RELEASE_GKE_CLUSTER: ${{ vars.RELEASE_GKE_CLUSTER }} - RELEASE_RING: ${{ inputs.ring }} - RELEASE_ID: ${{ inputs.release_id }} - PROD_CONFIRM: ${{ inputs.confirm }} - steps: - - name: Refuse free-form or ungated prod deploys - run: | - set -euo pipefail - test -n "$RELEASE_BUCKET" - test -n "$RELEASE_RUNTIME_PROJECT" - test -n "$RELEASE_GKE_CLUSTER" - [[ "$RELEASE_RING" == "beta" || "$RELEASE_RING" == "prod" ]] - [[ "$RELEASE_ID" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{2,127}$ ]] - if [[ "$RELEASE_RING" == "prod" ]]; then - test "$PROD_CONFIRM" = "deploy-prod" - fi - - - name: Checkout immutable reviewed deployment controller - uses: actions/checkout@v7 - with: - ref: ${{ github.sha }} - - - name: Authenticate with ring-bound OIDC identity - uses: google-github-actions/auth@v3 - with: - workload_identity_provider: ${{ vars.RELEASE_RINGS_WIF_PROVIDER }} - service_account: ${{ vars.RELEASE_RING_DEPLOYER_SERVICE_ACCOUNT }} - - - name: Set up GCP and Helm - uses: google-github-actions/setup-gcloud@v3 - - - uses: google-github-actions/get-gke-credentials@v3 - with: - cluster_name: ${{ vars.RELEASE_GKE_CLUSTER }} - location: ${{ env.REGION }} - project_id: ${{ env.RELEASE_RUNTIME_PROJECT }} - - - uses: azure/setup-helm@v5 - - - name: Resolve and validate only the requested immutable record - id: record - run: | - set -euo pipefail - mkdir -p artifacts/release-ring/config - gcloud storage cp "gs://${RELEASE_BUCKET}/records/${RELEASE_ID}.json" artifacts/release-ring/record.json - python3 backend/scripts/release_rings.py validate-record --record artifacts/release-ring/record.json - test "$(jq -r .release_id artifacts/release-ring/record.json)" = "$RELEASE_ID" - while IFS=$'\t' read -r name reference; do - object="${reference%%#sha256:*}" - destination="artifacts/release-ring/config/${name//\//-}.yaml" - gcloud storage cp "$object" "$destination" - expected="${reference##*#sha256:}" - actual="$(sha256sum "$destination" | awk '{print $1}')" - test "$actual" = "$expected" - done < <(jq -r '.rendered_config | to_entries[] | "\(.key)\t\(.value)"' artifacts/release-ring/record.json) - backend_image="$(jq -r '.images.backend' artifacts/release-ring/record.json)" - printf 'backend_image=%s\n' "$backend_image" >> "$GITHUB_OUTPUT" - printf 'git_sha=%s\n' "$(jq -r .git_sha artifacts/release-ring/record.json)" >> "$GITHUB_OUTPUT" - - - name: Require a non-held beta-soaked record before prod promotion - if: inputs.ring == 'prod' - env: - MIN_SOAK_SECONDS: ${{ vars.RELEASE_BETA_MIN_SOAK_SECONDS || '43200' }} - run: | - set -euo pipefail - test "$MIN_SOAK_SECONDS" -ge 1 - gcloud storage cp "gs://${RELEASE_BUCKET}/rings/beta/active.json" artifacts/release-ring/beta-active.json - if jq -e --arg id "$RELEASE_ID" '.held_release_ids | index($id)' artifacts/release-ring/beta-active.json; then - echo "Refusing held beta release ${RELEASE_ID}." >&2 - exit 1 - fi - verified="$(gcloud storage ls --recursive "gs://${RELEASE_BUCKET}/receipts/beta/${RELEASE_ID}/" | grep '/verified.json$' | sort | tail -1)" - test -n "$verified" - gcloud storage cp "$verified" artifacts/release-ring/beta-verified.json - test "$(jq -r .state artifacts/release-ring/beta-verified.json)" = verified - verified_at="$(jq -r .created_at artifacts/release-ring/beta-verified.json)" - test $(( $(date -u +%s) - $(date -u -d "$verified_at" +%s) )) -ge "$MIN_SOAK_SECONDS" - - - name: Checkout immutable release source inputs - uses: actions/checkout@v7 - with: - ref: ${{ steps.record.outputs.git_sha }} - path: artifacts/release-ring/source - - - name: Verify immutable release source checkout - run: | - set -euo pipefail - test "$(git -C artifacts/release-ring/source rev-parse HEAD)" = "${{ steps.record.outputs.git_sha }}" - - - name: Render recorded Cloud Run runtime configuration - id: runtime - run: | - python3 backend/scripts/render_backend_runtime_env.py \ - --env "$RELEASE_RING" \ - --manifest "artifacts/release-ring/config/${RELEASE_RING}-backend.yaml" \ - | tee artifacts/release-ring/runtime-outputs.txt >> "$GITHUB_OUTPUT" - - - name: Preflight all recorded GKE configuration before mutation - run: | - set -euo pipefail - namespace="$(jq -r --arg ring "$RELEASE_RING" '.topology[$ring].namespace' artifacts/release-ring/record.json)" - source_root="artifacts/release-ring/source" - kubectl -n "$namespace" apply --dry-run=server \ - -f "artifacts/release-ring/config/${RELEASE_RING}-backend-config.yaml" >/dev/null - ENVIRONMENT="$RELEASE_RING" \ - NAMESPACE="$namespace" \ - RELEASE_NAME="${RELEASE_RING}-omi-backend-secrets" \ - VALUES_FILE="artifacts/release-ring/config/${RELEASE_RING}-backend-secrets.yaml" \ - CHART_DIR="${source_root}/backend/charts/backend-secrets" \ - GCP_PROJECT_ID="$RELEASE_RUNTIME_PROJECT" \ - GKE_CLUSTER="$RELEASE_GKE_CLUSTER" \ - REGION="$REGION" \ - BACKEND_SECRETS_GSA="${{ vars.BACKEND_SECRETS_GSA }}" \ - DRY_RUN=true \ - backend/scripts/deploy-backend-secrets.sh - preflight_component() { - component="$1" - release="$2" - image="$(jq -r --arg component "$component" '.images[$component]' artifacts/release-ring/record.json)" - repository="${image%@sha256:*}" - digest="sha256:${image##*@sha256:}" - helm template "$release" "${source_root}/backend/charts/${component}" \ - --namespace "$namespace" \ - -f "artifacts/release-ring/config/${RELEASE_RING}-${component}.yaml" \ - --set-string image.repository="$repository" \ - --set-string image.digest="$digest" \ - | kubectl -n "$namespace" apply --dry-run=server -f - >/dev/null - } - preflight_component backend-listen "${RELEASE_RING}-omi-backend-listen" - preflight_component pusher "${RELEASE_RING}-omi-pusher" - preflight_component llm-gateway "${RELEASE_RING}-omi-llm-gateway" - preflight_component agent-proxy "${RELEASE_RING}-omi-agent-proxy" - - - name: Verify beta endpoint prerequisites before mutation - if: inputs.ring == 'beta' - run: | - set -euo pipefail - namespace="$(jq -r '.topology.beta.namespace' artifacts/release-ring/record.json)" - gcloud compute addresses describe beta-pusher-ilb-ip-address \ - --project "$RELEASE_RUNTIME_PROJECT" --region "$REGION" >/dev/null - gcloud compute addresses describe beta-agent-proxy-ip-address \ - --project "$RELEASE_RUNTIME_PROJECT" --global >/dev/null - kubectl -n "$namespace" get managedcertificate beta-agent-proxy-cert >/dev/null - getent ahosts api-beta.omi.me >/dev/null - getent ahosts pusher-beta.omi.me >/dev/null - getent ahosts agent-beta.omi.me >/dev/null - - - name: Snapshot serving state before mutation - id: snapshot - run: | - set -euo pipefail - namespace="$(jq -r --arg ring "$RELEASE_RING" '.topology[$ring].namespace' artifacts/release-ring/record.json)" - services=() - for logical in backend backend-sync backend-sync-backfill backend-integration; do - services+=(--service "$(jq -r --arg ring "$RELEASE_RING" --arg logical "$logical" '.topology[$ring].cloud_run_services[$logical]' artifacts/release-ring/record.json)") - done - mkdir -p artifacts/release-ring - python3 backend/scripts/cloud_run_traffic_snapshot.py capture \ - --project "$RELEASE_RUNTIME_PROJECT" \ - --region "$REGION" \ - "${services[@]}" \ - --allow-missing \ - --output artifacts/release-ring/cloud-run-snapshot.json - python3 backend/scripts/release_ring_gke_snapshot.py capture \ - --namespace "$namespace" \ - --config-map "${RELEASE_RING}-omi-backend-config" \ - --release "backend-secrets=${RELEASE_RING}-omi-backend-secrets" \ - --release "backend-listen=${RELEASE_RING}-omi-backend-listen" \ - --release "pusher=${RELEASE_RING}-omi-pusher" \ - --release "llm-gateway=${RELEASE_RING}-omi-llm-gateway" \ - --release "agent-proxy=${RELEASE_RING}-omi-agent-proxy" \ - --output artifacts/release-ring/gke-snapshot.json - active_object="gs://${RELEASE_BUCKET}/rings/${RELEASE_RING}/active.json" - active_generation=0 - if gcloud storage cp "$active_object" artifacts/release-ring/active-before.json \ - 2>artifacts/release-ring/active-before-error.txt; then - active_generation="$(gcloud storage objects describe "$active_object" --format='value(generation)')" - rm artifacts/release-ring/active-before-error.txt - active_pointer="$(jq -c . artifacts/release-ring/active-before.json)" - elif grep -Eiq 'not.?found|404' artifacts/release-ring/active-before-error.txt; then - active_pointer=null - else - sed 's/^/active pointer read failed: /' artifacts/release-ring/active-before-error.txt >&2 - exit 1 - fi - jq -n \ - --arg namespace "$namespace" \ - --argjson cloud_run "$(jq -c . artifacts/release-ring/cloud-run-snapshot.json)" \ - --argjson gke "$(jq -c . artifacts/release-ring/gke-snapshot.json)" \ - --argjson active_pointer "$active_pointer" \ - --argjson active_generation "$active_generation" \ - '{schema_version: 1, namespace: $namespace, cloud_run: $cloud_run, gke: $gke, active_pointer: $active_pointer, active_generation: $active_generation}' \ - > artifacts/release-ring/snapshot.json - snapshot_sha="$(sha256sum artifacts/release-ring/snapshot.json | awk '{print $1}')" - snapshot_object="gs://${RELEASE_BUCKET}/receipts/${RELEASE_RING}/${RELEASE_ID}/${GITHUB_RUN_ID}/snapshot.json" - gcloud storage cp --if-generation-match=0 artifacts/release-ring/snapshot.json "$snapshot_object" - printf 'reference=%s#sha256:%s\n' "$snapshot_object" "$snapshot_sha" >> "$GITHUB_OUTPUT" - printf 'active_generation=%s\n' "$active_generation" >> "$GITHUB_OUTPUT" - - - name: Write pre-mutation receipt - run: | - python3 backend/scripts/release_rings.py receipt \ - --ring "$RELEASE_RING" \ - --release-id "$RELEASE_ID" \ - --run-id "$GITHUB_RUN_ID" \ - --state started \ - --snapshot-reference "${{ steps.snapshot.outputs.reference }}" \ - --component cloud-run=pending \ - --component gke=pending \ - --output artifacts/release-ring/receipt-started.json - gcloud storage cp --if-generation-match=0 artifacts/release-ring/receipt-started.json "gs://${RELEASE_BUCKET}/receipts/${RELEASE_RING}/${RELEASE_ID}/${GITHUB_RUN_ID}/started.json" - - - name: Create no-traffic Cloud Run revisions from recorded digest - id: cloud_run_candidate - run: | - set -euo pipefail - cloud_run_flags="${{ steps.runtime.outputs.cloud_run_flags }}" - recorded_output() { - python3 - "$1" artifacts/release-ring/runtime-outputs.txt <<'PY' - from pathlib import Path - import sys - - name, source = sys.argv[1:] - delimiter = f"__BACKEND_RUNTIME_ENV_{name}__" - lines = Path(source).read_text(encoding="utf-8").splitlines() - start = lines.index(f"{name}<<{delimiter}") + 1 - end = lines.index(delimiter, start) - print("\n".join(lines[start:end])) - PY - } - for logical in backend backend-sync backend-sync-backfill backend-integration; do - service="$(jq -r --arg ring "$RELEASE_RING" --arg logical "$logical" '.topology[$ring].cloud_run_services[$logical]' artifacts/release-ring/record.json)" - case "$logical" in - backend) - env_vars="$(recorded_output backend_env_vars)" - secrets="$(recorded_output backend_secrets)" - ;; - backend-sync) - env_vars="$(recorded_output backend_sync_env_vars)" - secrets="$(recorded_output backend_sync_secrets)" - ;; - backend-sync-backfill) - env_vars="$(recorded_output backend_sync_backfill_env_vars)" - secrets="$(recorded_output backend_sync_backfill_secrets)" - ;; - backend-integration) - env_vars="$(recorded_output backend_integration_env_vars)" - secrets="$(recorded_output backend_integration_secrets)" - ;; - esac - env_vars="$(printf '%s' "$env_vars" | tr '\n' ',')" - secrets="$(printf '%s' "$secrets" | tr '\n' ',')" - gcloud run deploy "$service" \ - --project "$RELEASE_RUNTIME_PROJECT" \ - --region "$REGION" \ - --image "${{ steps.record.outputs.backend_image }}" \ - --set-env-vars "$env_vars" \ - --update-secrets "$secrets" \ - --tag "ring-${GITHUB_RUN_ID}" \ - --no-traffic \ - $cloud_run_flags \ - --quiet - gcloud run revisions describe "$(gcloud run services describe "$service" --project "$RELEASE_RUNTIME_PROJECT" --region "$REGION" --format='value(status.latestReadyRevisionName)')" \ - --project "$RELEASE_RUNTIME_PROJECT" --region "$REGION" --format='value(status.conditions[0].status)' | grep -qx True - done - - - name: Smoke the no-traffic authenticated backend candidate - id: backend_candidate - run: | - set -euo pipefail - service="$(jq -r --arg ring "$RELEASE_RING" '.topology[$ring].cloud_run_services.backend' artifacts/release-ring/record.json)" - revision="$(gcloud run services describe "$service" --project "$RELEASE_RUNTIME_PROJECT" --region "$REGION" --format='value(status.latestReadyRevisionName)')" - url="$(python3 backend/scripts/resolve_cloud_run_tagged_url.py --project "$RELEASE_RUNTIME_PROJECT" --region "$REGION" --service "$service" --revision "$revision" --tag "ring-${GITHUB_RUN_ID}")" - CLOUD_RUN_IDENTITY_TOKEN="$(gcloud auth print-identity-token --audiences="$url")" - export CLOUD_RUN_IDENTITY_TOKEN - python3 backend/scripts/smoke_cloud_run_health.py --base-url "$url" --cloud-run-identity-token-env CLOUD_RUN_IDENTITY_TOKEN - printf 'url=%s\n' "$url" >> "$GITHUB_OUTPUT" - - # Production deliberately has no public Cloud Run revision URL. Its - # promotion gate is the successful beta full-route probe plus its soak; - # beta is the public, no-traffic candidate where this end-to-end probe is - # safe and meaningful. - - name: Gate beta candidate on authenticated known-audio transcription - if: inputs.ring == 'beta' - uses: ./.github/actions/transcription-release-candidate-probe - with: - candidate_api_url: ${{ steps.backend_candidate.outputs.url }} - project_id: ${{ env.RELEASE_RUNTIME_PROJECT }} - firebase_auth_project_id: based-hardware - evidence_path: ${{ runner.temp }}/beta-transcription-capability-probe.json - - - name: Preserve beta full-route probe evidence - if: always() && inputs.ring == 'beta' - run: | - mkdir -p artifacts/release-ring - if [[ -f "$RUNNER_TEMP/beta-transcription-capability-probe.json" ]]; then - cp "$RUNNER_TEMP/beta-transcription-capability-probe.json" artifacts/release-ring/beta-transcription-capability-probe.json - fi - - - name: Apply recorded Helm values and recorded digests - id: gke_candidate - run: | - set -euo pipefail - namespace="$(jq -r --arg ring "$RELEASE_RING" '.topology[$ring].namespace' artifacts/release-ring/record.json)" - source_root="artifacts/release-ring/source" - kubectl -n "$namespace" apply -f "artifacts/release-ring/config/${RELEASE_RING}-backend-config.yaml" - ENVIRONMENT="$RELEASE_RING" \ - NAMESPACE="$namespace" \ - RELEASE_NAME="${RELEASE_RING}-omi-backend-secrets" \ - VALUES_FILE="artifacts/release-ring/config/${RELEASE_RING}-backend-secrets.yaml" \ - CHART_DIR="${source_root}/backend/charts/backend-secrets" \ - GCP_PROJECT_ID="$RELEASE_RUNTIME_PROJECT" \ - GKE_CLUSTER="$RELEASE_GKE_CLUSTER" \ - REGION="$REGION" \ - BACKEND_SECRETS_GSA="${{ vars.BACKEND_SECRETS_GSA }}" \ - backend/scripts/deploy-backend-secrets.sh - python3 backend/scripts/verify_pusher_config_references.py \ - --environment "$RELEASE_RING" --namespace "$namespace" \ - --values-file "artifacts/release-ring/config/${RELEASE_RING}-pusher.yaml" - deploy_component() { - component="$1" - release="$2" - chart="$3" - image="$(jq -r --arg component "$component" '.images[$component]' artifacts/release-ring/record.json)" - repository="${image%@sha256:*}" - digest="sha256:${image##*@sha256:}" - helm -n "$namespace" upgrade --install "$release" "$chart" \ - -f "artifacts/release-ring/config/${RELEASE_RING}-${component}.yaml" \ - --set-string image.repository="$repository" \ - --set-string image.digest="$digest" - kubectl -n "$namespace" rollout status "deployment/$release" --timeout=1800s - } - deploy_component backend-listen "${RELEASE_RING}-omi-backend-listen" "${source_root}/backend/charts/backend-listen" - deploy_component pusher "${RELEASE_RING}-omi-pusher" "${source_root}/backend/charts/pusher" - deploy_component llm-gateway "${RELEASE_RING}-omi-llm-gateway" "${source_root}/backend/charts/llm-gateway" - deploy_component agent-proxy "${RELEASE_RING}-omi-agent-proxy" "${source_root}/backend/charts/agent-proxy" - - - name: Shift validated Cloud Run revisions to serving traffic - id: promote_traffic - run: | - set -euo pipefail - for logical in backend backend-sync backend-sync-backfill backend-integration; do - service="$(jq -r --arg ring "$RELEASE_RING" --arg logical "$logical" '.topology[$ring].cloud_run_services[$logical]' artifacts/release-ring/record.json)" - revision="$(gcloud run services describe "$service" --project "$RELEASE_RUNTIME_PROJECT" --region "$REGION" --format='value(status.latestReadyRevisionName)')" - gcloud run services update-traffic "$service" --project "$RELEASE_RUNTIME_PROJECT" --region "$REGION" --to-revisions="${revision}=100" --quiet - done - - - name: Verify serving ring composition - id: verify - run: | - set -euo pipefail - expected="${{ steps.record.outputs.backend_image }}" - for logical in backend backend-sync backend-sync-backfill backend-integration; do - service="$(jq -r --arg ring "$RELEASE_RING" --arg logical "$logical" '.topology[$ring].cloud_run_services[$logical]' artifacts/release-ring/record.json)" - actual="$(gcloud run services describe "$service" --project "$RELEASE_RUNTIME_PROJECT" --region "$REGION" --format='value(spec.template.spec.containers[0].image)')" - test "$actual" = "$expected" - done - namespace="$(jq -r --arg ring "$RELEASE_RING" '.topology[$ring].namespace' artifacts/release-ring/record.json)" - for deployment in "${RELEASE_RING}-omi-backend-listen" "${RELEASE_RING}-omi-pusher" "${RELEASE_RING}-omi-llm-gateway" "${RELEASE_RING}-omi-agent-proxy"; do - kubectl -n "$namespace" rollout status "deployment/$deployment" --timeout=300s - done - - - name: Advance active pointer with GCS compare-and-swap - if: success() - run: | - set -euo pipefail - object="gs://${RELEASE_BUCKET}/rings/${RELEASE_RING}/active.json" - if [[ -f artifacts/release-ring/active-before.json ]]; then - existing=(--existing artifacts/release-ring/active-before.json) - else - existing=() - fi - python3 backend/scripts/release_rings.py active-pointer \ - --ring "$RELEASE_RING" --release-id "$RELEASE_ID" \ - "${existing[@]}" --output artifacts/release-ring/active-next.json - gcloud storage cp --if-generation-match="${{ steps.snapshot.outputs.active_generation }}" \ - artifacts/release-ring/active-next.json "$object" - - - name: Write verified receipt - if: success() - run: | - python3 backend/scripts/release_rings.py receipt \ - --ring "$RELEASE_RING" --release-id "$RELEASE_ID" --run-id "$GITHUB_RUN_ID" \ - --state verified --snapshot-reference "${{ steps.snapshot.outputs.reference }}" \ - --component cloud-run=verified --component gke=verified \ - --output artifacts/release-ring/receipt-verified.json - gcloud storage cp --if-generation-match=0 artifacts/release-ring/receipt-verified.json "gs://${RELEASE_BUCKET}/receipts/${RELEASE_RING}/${RELEASE_ID}/${GITHUB_RUN_ID}/verified.json" - - - name: Restore pre-mutation state once after a deterministic failure - id: restore - if: failure() && steps.snapshot.outcome == 'success' - run: | - set +e - python3 backend/scripts/cloud_run_traffic_snapshot.py restore \ - --snapshot artifacts/release-ring/cloud-run-snapshot.json \ - --remove-tag "ring-${GITHUB_RUN_ID}" \ - --delete-missing \ - --evidence-path artifacts/release-ring/cloud-run-restore.json - result=$? - python3 backend/scripts/release_ring_gke_snapshot.py restore \ - --snapshot artifacts/release-ring/gke-snapshot.json \ - --evidence-path artifacts/release-ring/gke-restore.json - result=$((result || $?)) - if jq -e '.helm_releases["backend-secrets"].present == true' artifacts/release-ring/gke-snapshot.json >/dev/null; then - namespace="$(jq -r .namespace artifacts/release-ring/gke-snapshot.json)" - force_sync_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" - kubectl -n "$namespace" annotate externalsecret \ - "${RELEASE_RING}-omi-backend-external-secret" \ - force-sync="$force_sync_at" --overwrite - result=$((result || $?)) - python3 backend/scripts/wait_external_secret_refresh.py \ - --namespace "$namespace" \ - --name "${RELEASE_RING}-omi-backend-external-secret" \ - --min-refresh-time "$force_sync_at" \ - --timeout-seconds 120 - result=$((result || $?)) - fi - test "$result" -eq 0 - - - name: Hold failed record and write recovery receipt - if: failure() && steps.snapshot.outcome == 'success' - env: - PAGER_WEBHOOK: ${{ secrets.RELEASE_PAGER_WEBHOOK }} - run: | - set -euo pipefail - object="gs://${RELEASE_BUCKET}/rings/${RELEASE_RING}/active.json" - generation=0 - if gcloud storage cp "$object" artifacts/release-ring/active-current.json \ - 2>artifacts/release-ring/active-current-error.txt; then - generation="$(gcloud storage objects describe "$object" --format='value(generation)')" - rm artifacts/release-ring/active-current-error.txt - if jq -e --arg id "$RELEASE_ID" '.current_release_id == $id' \ - artifacts/release-ring/active-current.json >/dev/null \ - && [[ -f artifacts/release-ring/active-before.json ]]; then - existing=(--existing artifacts/release-ring/active-before.json) - elif jq -e --arg id "$RELEASE_ID" '.current_release_id == $id' \ - artifacts/release-ring/active-current.json >/dev/null; then - existing=() - else - existing=(--existing artifacts/release-ring/active-current.json) - fi - elif grep -Eiq 'not.?found|404' artifacts/release-ring/active-current-error.txt; then - existing=() - else - sed 's/^/active pointer read failed: /' artifacts/release-ring/active-current-error.txt >&2 - exit 1 - fi - state=partial_mutation - if [[ "${{ steps.restore.outcome }}" == "success" ]]; then state=restored; fi - python3 backend/scripts/release_rings.py active-pointer \ - --ring "$RELEASE_RING" --release-id "$RELEASE_ID" --hold \ - "${existing[@]}" --output artifacts/release-ring/active-held.json - gcloud storage cp --if-generation-match="$generation" artifacts/release-ring/active-held.json "$object" - python3 backend/scripts/release_rings.py receipt \ - --ring "$RELEASE_RING" --release-id "$RELEASE_ID" --run-id "$GITHUB_RUN_ID" \ - --state "$state" --snapshot-reference "${{ steps.snapshot.outputs.reference }}" \ - --component cloud-run="${{ steps.restore.outcome }}" --component gke="${{ steps.restore.outcome }}" \ - --output artifacts/release-ring/receipt-recovery.json - gcloud storage cp --if-generation-match=0 artifacts/release-ring/receipt-recovery.json "gs://${RELEASE_BUCKET}/receipts/${RELEASE_RING}/${RELEASE_ID}/${GITHUB_RUN_ID}/recovery.json" - if [[ "$state" == partial_mutation ]]; then - echo "::error title=Release partial mutation::${RELEASE_RING} release ${RELEASE_ID} could not be restored." - test -n "$PAGER_WEBHOOK" - curl --fail --silent --show-error -X POST -H 'content-type: application/json' \ - --data "{\"ring\":\"${RELEASE_RING}\",\"release_id\":\"${RELEASE_ID}\",\"state\":\"partial_mutation\"}" \ - "$PAGER_WEBHOOK" - fi - exit 1 - - - name: Preserve deployment evidence - if: always() - uses: actions/upload-artifact@v7 - with: - name: release-ring-${{ env.RELEASE_RING }}-${{ env.RELEASE_ID }}-${{ github.run_id }} - path: | - artifacts/release-ring/*.json - artifacts/release-ring/*.txt - artifacts/release-ring/config - if-no-files-found: warn - retention-days: 30 diff --git a/.github/workflows/desktop_emergency_promote_beta.yml b/.github/workflows/desktop_emergency_promote_beta.yml deleted file mode 100644 index c1854cb42b0..00000000000 --- a/.github/workflows/desktop_emergency_promote_beta.yml +++ /dev/null @@ -1,179 +0,0 @@ -name: Emergency Promote macOS Beta Candidate - -# Break-glass path for shipping a signed candidate to beta when the -# qualification gate itself is broken. Beta only: this workflow cannot touch -# stable, and the backend rejects any non-beta target. -# -# The default path (desktop_qualify_beta -> desktop_promote_beta) stays -# fail-closed. This is the escape hatch, deliberately small: the only guards -# are the ones a machine can check -- tag/SHA binding, artifact digests, and a -# pointer compare-and-swap. There is no approval ceremony, because paperwork -# is not a control an automated actor can be stopped by. -# -# Every run opens a `qualification-gate-failure` issue. If those pile up, the -# qualification gate is the bug and this hatch is masking it. - -on: - workflow_dispatch: - inputs: - release_tag: - description: 'Exact signed v*-macos candidate tag' - required: true - type: string - reason: - description: 'Why the qualification gate could not be used' - required: true - type: string - confirm: - description: 'Type emergency-promote-beta' - required: true - type: string - -permissions: - contents: read - issues: write - id-token: write - -concurrency: - group: desktop-beta-pointer - cancel-in-progress: false - -jobs: - promote: - environment: prod - runs-on: ubuntu-latest - steps: - - name: Confirm intent - run: test "${{ inputs.confirm }}" = "emergency-promote-beta" - - - name: Checkout promotion controls - uses: actions/checkout@v7 - with: - ref: main - fetch-depth: 0 - - - name: Generate Omi Bot token - id: app-token - uses: actions/create-github-app-token@v3 - with: - app-id: ${{ secrets.OMI_BOT_APP_ID }} - private-key: ${{ secrets.OMI_BOT_PRIVATE_KEY }} - - - name: Bind tag to signed artifacts - id: candidate - env: - GH_TOKEN: ${{ steps.app-token.outputs.token }} - RELEASE_TAG: ${{ inputs.release_tag }} - REPO: ${{ github.repository }} - run: | - set -euo pipefail - git fetch --force origin "+refs/tags/${RELEASE_TAG}:refs/tags/${RELEASE_TAG}" - TAG_SHA=$(git rev-list -n1 "$RELEASE_TAG") - WORK=/tmp/desktop-emergency-beta - mkdir -p "$WORK/assets" - gh release view "$RELEASE_TAG" --repo "$REPO" \ - --json tagName,body,isDraft,isPrerelease,publishedAt,assets > "$WORK/release.json" - gh release download "$RELEASE_TAG" --repo "$REPO" \ - --pattern Omi.zip --pattern '*.dmg' --dir "$WORK/assets" - DMG_PATH=$(find "$WORK/assets" -maxdepth 1 -name '*.dmg' -type f -print -quit) - test -n "$DMG_PATH" - { - echo "target_sha=$TAG_SHA" - echo "dmg_path=$DMG_PATH" - echo "zip_sha256=$(sha256sum "$WORK/assets/Omi.zip" | cut -d' ' -f1)" - echo "dmg_sha256=$(sha256sum "$DMG_PATH" | cut -d' ' -f1)" - } >> "$GITHUB_OUTPUT" - - - name: Prepare immutable manifest - run: | - set -euo pipefail - python3 .github/scripts/prepare-desktop-beta-promotion.py \ - --release-json /tmp/desktop-emergency-beta/release.json \ - --release-tag "${{ inputs.release_tag }}" \ - --target-sha "${{ steps.candidate.outputs.target_sha }}" \ - --zip-sha256 "${{ steps.candidate.outputs.zip_sha256 }}" \ - --dmg-sha256 "${{ steps.candidate.outputs.dmg_sha256 }}" \ - --allow-unqualified \ - --output /tmp/desktop-emergency-beta/manifest.json - - - name: Google Auth - uses: google-github-actions/auth@v3 - with: - credentials_json: ${{ secrets.GCP_CREDENTIALS }} - - - name: Register manifest and compare-and-swap macOS beta - env: - PROJECT_ID: ${{ vars.GCP_PROJECT_ID }} - RELEASE_TAG: ${{ inputs.release_tag }} - run: | - set -euo pipefail - ADMIN_KEY=$(gcloud secrets versions access latest --secret=ADMIN_KEY --project "$PROJECT_ID") - curl -fsS -X POST -H 'Content-Type: application/json' -H "secret-key: $ADMIN_KEY" \ - --data-binary @/tmp/desktop-emergency-beta/manifest.json \ - https://api.omi.me/v2/desktop/releases > /dev/null - - # Read the live pointer here rather than making an operator paste it - # mid-incident. The generation CAS below still rejects any concurrent - # mutation between this read and the write. - WORK=/tmp/desktop-emergency-beta - curl -fsS -H "secret-key: $ADMIN_KEY" \ - 'https://api.omi.me/v2/desktop/channels/live?platform=macos&channel=beta' \ - > "$WORK/current.json" - python3 - "$WORK/current.json" "$WORK/promote.json" <<'PY' - import json, os, sys - - current = json.load(open(sys.argv[1]))["pointer"] - json.dump( - { - "platform": "macos", - "channel": "beta", - "release_id": os.environ["RELEASE_TAG"], - "expected_current_release_id": current["release_id"], - "expected_generation": int(current["generation"]), - "emergency": True, - }, - open(sys.argv[2], "w"), - ) - PY - - curl -fsS -X POST -H 'Content-Type: application/json' -H "secret-key: $ADMIN_KEY" \ - --data-binary @"$WORK/promote.json" \ - https://api.omi.me/v2/desktop/channels/promote > "$WORK/response.json" - - python3 - "$WORK/current.json" "$WORK/response.json" <<'PY' - import json, os, sys - - expected_generation = int(json.load(open(sys.argv[1]))["pointer"]["generation"]) + 1 - response = json.load(open(sys.argv[2])) - pointer = response.get("pointer") or {} - if ( - response.get("success") is not True - or pointer.get("channel") != "beta" - or pointer.get("release_id") != os.environ["RELEASE_TAG"] - or pointer.get("generation") != expected_generation - ): - raise SystemExit(f"beta pointer compare-and-swap did not succeed: {response}") - print(json.dumps(response, indent=2)) - PY - - - name: Record that the qualification gate was bypassed - if: ${{ always() && job.status != 'cancelled' }} - env: - GH_TOKEN: ${{ steps.app-token.outputs.token }} - REPO: ${{ github.repository }} - run: | - set -euo pipefail - gh issue create --repo "$REPO" \ - --title "Qualification gate bypassed for ${{ inputs.release_tag }}" \ - --label qualification-gate-failure \ - --body "$(printf '%s\n' \ - "Beta was promoted by \`desktop_emergency_promote_beta\`, bypassing qualification." \ - "" \ - "- Tag: \`${{ inputs.release_tag }}\`" \ - "- SHA: \`${{ steps.candidate.outputs.target_sha }}\`" \ - "- Operator: @${{ github.triggering_actor }}" \ - "- Reason: ${{ inputs.reason }}" \ - "- Run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \ - "- Result: ${{ job.status }}" \ - "" \ - "The gate failing is the defect. Fix the gate, then close this.")" diff --git a/.github/workflows/desktop_nominate_stable_candidate.yml b/.github/workflows/desktop_nominate_stable_candidate.yml deleted file mode 100644 index f2cdd7ba954..00000000000 --- a/.github/workflows/desktop_nominate_stable_candidate.yml +++ /dev/null @@ -1,128 +0,0 @@ -name: Nominate Desktop Stable Candidate - -on: - workflow_dispatch: - inputs: - release_tag: - description: 'Qualified beta v*-macos tag to nominate' - required: true - rationale: - description: 'Why this beta is ready for stable consideration' - required: true - soak_review: - description: 'Soak duration and outcome reviewed' - required: true - telemetry_review: - description: 'Crash, update, and health telemetry reviewed' - required: true - release_notes_review: - description: 'Stable release-note rollup reviewed' - required: true - -permissions: - contents: read - -concurrency: - group: desktop-stable-candidate-nomination - cancel-in-progress: false - -jobs: - nominate: - environment: prod - permissions: - contents: write - id-token: write - runs-on: ubuntu-latest - steps: - - name: Checkout nomination controls - uses: actions/checkout@v7 - with: - fetch-depth: 0 - - - name: Generate Omi Bot token - id: app-token - uses: actions/create-github-app-token@v3 - with: - app-id: ${{ secrets.OMI_BOT_APP_ID }} - private-key: ${{ secrets.OMI_BOT_PRIVATE_KEY }} - - - name: Google Auth - uses: google-github-actions/auth@v3 - with: - credentials_json: ${{ secrets.GCP_CREDENTIALS }} - - - name: Validate qualified beta and prepare nomination - env: - GH_TOKEN: ${{ steps.app-token.outputs.token }} - PROJECT_ID: ${{ vars.GCP_PROJECT_ID }} - RELEASE_TAG: ${{ inputs.release_tag }} - REPO: ${{ github.repository }} - NOMINATOR: ${{ github.actor }} - RATIONALE: ${{ inputs.rationale }} - SOAK_REVIEW: ${{ inputs.soak_review }} - TELEMETRY_REVIEW: ${{ inputs.telemetry_review }} - RELEASE_NOTES_REVIEW: ${{ inputs.release_notes_review }} - run: | - set -euo pipefail - if ! printf '%s\n' "$RELEASE_TAG" | grep -qE '^v[0-9]+\.[0-9]+(\.[0-9]+)?\+[0-9]+-macos$'; then - echo "Invalid macOS release tag: $RELEASE_TAG" >&2 - exit 1 - fi - - git fetch --force origin "+refs/tags/${RELEASE_TAG}:refs/tags/${RELEASE_TAG}" - TARGET_SHA=$(git rev-list -n1 "$RELEASE_TAG") - gh release view "$RELEASE_TAG" --repo "$REPO" \ - --json tagName,body,isDraft,isPrerelease,assets \ - > /tmp/desktop-stable-candidate-release.json - - ACCESS_TOKEN=$(gcloud auth print-access-token) - POINTER_URL="https://firestore.googleapis.com/v1/projects/${PROJECT_ID}/databases/(default)/documents/desktop_update_channels/macos-beta" - MANIFEST_URL="https://firestore.googleapis.com/v1/projects/${PROJECT_ID}/databases/(default)/documents/desktop_release_manifests/${RELEASE_TAG}" - curl -fsS -H "Authorization: Bearer ${ACCESS_TOKEN}" "$POINTER_URL" \ - > /tmp/desktop-beta-pointer.json - curl -fsS -H "Authorization: Bearer ${ACCESS_TOKEN}" "$MANIFEST_URL" \ - > /tmp/desktop-beta-manifest.json - BETA_RELEASE_ID=$(python3 - <<'PY' - import json - - pointer = json.load(open("/tmp/desktop-beta-pointer.json", encoding="utf-8")) - print(pointer.get("fields", {}).get("release_id", {}).get("stringValue", "")) - PY - ) - if [ -z "$BETA_RELEASE_ID" ]; then - echo "The macOS beta pointer is missing release_id" >&2 - exit 1 - fi - BETA_SOURCE_SHA=$(python3 - <<'PY' - import json - - manifest = json.load(open("/tmp/desktop-beta-manifest.json", encoding="utf-8")) - print(manifest.get("fields", {}).get("source_sha", {}).get("stringValue", "")) - PY - ) - if [ -z "$BETA_SOURCE_SHA" ]; then - echo "The macOS beta manifest is missing source_sha" >&2 - exit 1 - fi - - python3 .github/scripts/nominate-desktop-stable-candidate.py \ - --release-json /tmp/desktop-stable-candidate-release.json \ - --release-tag "$RELEASE_TAG" \ - --target-sha "$TARGET_SHA" \ - --beta-release-id "$BETA_RELEASE_ID" \ - --beta-source-sha "$BETA_SOURCE_SHA" \ - --nominator "$NOMINATOR" \ - --rationale "$RATIONALE" \ - --soak-review "$SOAK_REVIEW" \ - --telemetry-review "$TELEMETRY_REVIEW" \ - --release-notes-review "$RELEASE_NOTES_REVIEW" \ - --output /tmp/desktop-stable-candidate-body.md - - gh release edit "$RELEASE_TAG" --repo "$REPO" \ - --notes-file /tmp/desktop-stable-candidate-body.md - { - echo "Nominated stable candidate: $RELEASE_TAG ($TARGET_SHA)" - echo "Nominator: $NOMINATOR" - echo "Beta pointer remained unchanged at: $BETA_RELEASE_ID" - echo "Stable visibility was not changed. Use desktop_promote_prod.yml separately." - } >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/desktop_promote_beta.yml b/.github/workflows/desktop_promote_beta.yml index e6f6db5837f..f0f5d91fe6d 100644 --- a/.github/workflows/desktop_promote_beta.yml +++ b/.github/workflows/desktop_promote_beta.yml @@ -11,9 +11,14 @@ on: required: false default: false type: boolean + qualification_run_id: + description: 'Successful trusted desktop_qualify_beta.yml run ID containing immutable evidence' + required: true + type: string permissions: contents: read + actions: read id-token: write concurrency: @@ -28,6 +33,7 @@ jobs: - name: Checkout promotion controls uses: actions/checkout@v7 with: + ref: main fetch-depth: 0 - name: Generate Omi Bot token @@ -59,12 +65,21 @@ jobs: fi echo "Automatic beta request targets newest candidate $RELEASE_TAG." >> "$GITHUB_STEP_SUMMARY" + - name: Download immutable qualification evidence + uses: actions/download-artifact@v7 + with: + name: desktop-qualification-evidence-${{ inputs.release_tag }} + run-id: ${{ inputs.qualification_run_id }} + github-token: ${{ github.token }} + path: /tmp/desktop-qualification-evidence + - name: Download and validate qualified candidate id: candidate env: GH_TOKEN: ${{ steps.app-token.outputs.token }} RELEASE_TAG: ${{ inputs.release_tag }} REPO: ${{ github.repository }} + QUALIFICATION_RUN_ID: ${{ inputs.qualification_run_id }} run: | set -euo pipefail if ! printf '%s\n' "$RELEASE_TAG" | grep -qE '^v[0-9]+\.[0-9]+(\.[0-9]+)?\+[0-9]+-macos$'; then @@ -74,19 +89,31 @@ jobs: git fetch --force origin "+refs/tags/${RELEASE_TAG}:refs/tags/${RELEASE_TAG}" TARGET_SHA=$(git rev-list -n1 "$RELEASE_TAG") - mkdir -p /tmp/desktop-beta-assets + [[ "$QUALIFICATION_RUN_ID" =~ ^[0-9]+$ ]] || { echo 'qualification_run_id must be numeric' >&2; exit 1; } + run=$(gh api "repos/$REPO/actions/runs/$QUALIFICATION_RUN_ID") + test "$(jq -r .conclusion <<<"$run")" = success + test "$(jq -r .repository.full_name <<<"$run")" = "$REPO" + test "$(jq -r .head_repository.full_name <<<"$run")" = "$REPO" + test "$(jq -r .event <<<"$run")" = workflow_dispatch + test "$(jq -r .path <<<"$run")" = .github/workflows/desktop_qualify_beta.yml + test "$(jq -r .head_branch <<<"$run")" = main + test "$(jq -r .name <<<"$run")" = 'Qualify Desktop Beta Candidate' + test "$(jq -r .head_sha <<<"$run")" = "$TARGET_SHA" gh release view "$RELEASE_TAG" --repo "$REPO" \ --json tagName,body,isDraft,isPrerelease,publishedAt,assets \ > /tmp/desktop-beta-release.json + mkdir -p /tmp/desktop-beta-assets gh release download "$RELEASE_TAG" --repo "$REPO" \ - --pattern 'Omi.zip' --pattern '*.dmg' --dir /tmp/desktop-beta-assets + --pattern 'Omi.zip' --pattern 'Omi.dmg' --pattern 'omi.dmg' --pattern 'Omi.Beta.zip' --pattern 'omi-beta.dmg' --dir /tmp/desktop-beta-assets ZIP_PATH=/tmp/desktop-beta-assets/Omi.zip - DMG_PATH=$(find /tmp/desktop-beta-assets -maxdepth 1 -type f -name '*.dmg' -print -quit) + DMG_PATH=$(find /tmp/desktop-beta-assets -maxdepth 1 -type f \( -name Omi.dmg -o -name omi.dmg \) -print -quit) test -f "$ZIP_PATH" test -n "$DMG_PATH" ZIP_SHA=$(sha256sum "$ZIP_PATH" | cut -d' ' -f1) DMG_SHA=$(sha256sum "$DMG_PATH" | cut -d' ' -f1) + BETA_ZIP_SHA=$(sha256sum /tmp/desktop-beta-assets/Omi.Beta.zip | cut -d' ' -f1) + BETA_DMG_SHA=$(sha256sum /tmp/desktop-beta-assets/omi-beta.dmg | cut -d' ' -f1) python3 .github/scripts/prepare-desktop-beta-promotion.py \ --release-json /tmp/desktop-beta-release.json \ @@ -94,6 +121,9 @@ jobs: --target-sha "$TARGET_SHA" \ --zip-sha256 "$ZIP_SHA" \ --dmg-sha256 "$DMG_SHA" \ + --beta-zip-sha256 "$BETA_ZIP_SHA" \ + --beta-dmg-sha256 "$BETA_DMG_SHA" \ + --qualification-evidence /tmp/desktop-qualification-evidence/qualification-evidence.json \ --output /tmp/desktop-release-manifest.json { @@ -208,18 +238,18 @@ jobs: "version": manifest["version"], "build_number": manifest["build_number"], "tag": manifest["release_id"], - "download_url": manifest["dmg_url"], + "download_url": manifest["beta_dmg_url"], "published_at": manifest["published_at"], }, indent=2)) PY python3 - <<'PY' > /tmp/macos-beta-index.html import html import json - url = json.load(open('/tmp/desktop-release-manifest.json'))['dmg_url'] + url = json.load(open('/tmp/desktop-release-manifest.json'))['beta_dmg_url'] escaped = html.escape(url, quote=True) print(f'Download Omi Beta') PY - python3 -c "import json; print(json.load(open('/tmp/desktop-release-manifest.json'))['dmg_url'])" \ + python3 -c "import json; print(json.load(open('/tmp/desktop-release-manifest.json'))['beta_dmg_url'])" \ > /tmp/macos-beta-latest-url.txt for file in index.html redirect.json latest-url.txt; do gcloud storage cp --cache-control='no-store,max-age=0' \ diff --git a/.github/workflows/desktop_promote_prod.yml b/.github/workflows/desktop_promote_prod.yml index 585440cefe5..cf0c02f53ad 100644 --- a/.github/workflows/desktop_promote_prod.yml +++ b/.github/workflows/desktop_promote_prod.yml @@ -1,634 +1,209 @@ -name: Promote Desktop Backend to Prod - -# Prod promotion is intentionally manual and ordered: -# 1. choose an explicitly nominated stable candidate, -# 2. deploy the Rust desktop backend from that exact tag, -# 3. verify /health reports that release identity, -# 4. promote the Firestore bridge release for old Rust appcast clients, -# 5. mark the GitHub Release metadata channel stable, -# 6. move desktop-backend-prod-deployed. -# -# Do not add schedule, push, or release triggers here. Release-edited events -# have missed promotions before, and automatic triggers can expose stable users -# to a backend that was not promoted deliberately. +name: Promote Qualified Desktop Stable +# Stable is a protected distribution pointer, not a backend deployment. It +# promotes the exact signed artifact already qualified on beta; the existing +# backend deployment workflows remain independent. on: workflow_dispatch: inputs: release_tag: - description: 'Nominated stable-candidate v*-macos release tag' + description: 'Qualified current beta v*-macos release tag' required: true + type: string confirm: - description: 'Type promote-stable to deploy prod and mark the release stable' + description: 'Type promote-stable to advance the stable pointer' required: true - break_glass: - description: 'DANGER: bypass qualification and stable-candidate gates' - required: false - default: false - type: boolean - break_glass_confirm: - description: 'Type I-ACCEPT-STABLE-PROMOTION-RISK with break_glass' + type: string + operation: + description: 'promote advances Stable; repoint selects a retained qualified manifest' + required: true + default: promote + type: choice + options: [promote, repoint] + expected_current_release_id: + description: 'Current stable release ID (empty only for the first promotion)' required: false default: '' - break_glass_reason: - description: 'Required audited rationale when break_glass is enabled' + type: string + expected_generation: + description: 'Current stable pointer generation (0 when Stable is unset)' + required: true + type: string + qualification_run_id: + description: 'Successful trusted qualification run required only for an initial promote' required: false default: '' + type: string permissions: contents: read + actions: read + id-token: write concurrency: - group: desktop-backend-promote-prod + group: desktop-stable-promotion cancel-in-progress: false env: - SERVICE: desktop-backend - REGION: us-central1 - TRACK_TAG: desktop-backend-prod-deployed GCS_DESKTOP_UPDATES_BUCKET: ${{ vars.GCS_DESKTOP_UPDATES_BUCKET || 'gs://omi_macos_updates' }} jobs: promote: environment: prod - permissions: - contents: read - id-token: write - runs-on: ubuntu-latest-m + runs-on: ubuntu-latest steps: - - name: Delete huge unnecessary tools folder - run: rm -rf /opt/hostedtoolcache - - name: Checkout promotion controls uses: actions/checkout@v7 with: + ref: main fetch-depth: 0 - - name: Validate release promotion request - id: plan - env: - GH_TOKEN: ${{ github.token }} - REPO: ${{ github.repository }} - RELEASE_TAG: ${{ inputs.release_tag }} - CONFIRM: ${{ inputs.confirm }} - BREAK_GLASS: ${{ inputs.break_glass }} - BREAK_GLASS_CONFIRM: ${{ inputs.break_glass_confirm }} - BREAK_GLASS_REASON: ${{ inputs.break_glass_reason }} - run: | - set -euo pipefail - - if [ "$CONFIRM" != "promote-stable" ]; then - echo "confirm must be exactly: promote-stable" >&2 - exit 1 - fi - - if ! printf '%s\n' "$RELEASE_TAG" | grep -qE '^v.+-macos$'; then - echo "Release tag '$RELEASE_TAG' is not a macOS desktop release tag." >&2 - exit 1 - fi - - git fetch --force --tags origin \ - "+refs/tags/${RELEASE_TAG}:refs/tags/${RELEASE_TAG}" \ - "+refs/tags/${TRACK_TAG}:refs/tags/${TRACK_TAG}" || true - - gh release view "$RELEASE_TAG" --repo "$REPO" \ - --json tagName,body,isDraft,isPrerelease,publishedAt,assets \ - > /tmp/desktop-release.json - - TARGET_SHA=$(git rev-list -n1 "$RELEASE_TAG") - PROMOTION_CHECK_ARGS=( - --release-json /tmp/desktop-release.json - --release-tag "$RELEASE_TAG" - --target-sha "$TARGET_SHA" - --github-output "$GITHUB_OUTPUT" - ) - if [ "$BREAK_GLASS" = "true" ]; then - PROMOTION_CHECK_ARGS+=( - --break-glass - --break-glass-confirm "$BREAK_GLASS_CONFIRM" - --break-glass-reason "$BREAK_GLASS_REASON" - ) - echo "::warning::Audited stable-promotion break glass enabled for $RELEASE_TAG: $BREAK_GLASS_REASON" - echo "Audited break glass by ${{ github.actor }} for $RELEASE_TAG: $BREAK_GLASS_REASON" >> "$GITHUB_STEP_SUMMARY" - fi - - python3 .github/scripts/check-desktop-release-promotion.py \ - "${PROMOTION_CHECK_ARGS[@]}" - - CURRENT_SHA=$(git rev-list -n1 "$TRACK_TAG" 2>/dev/null || true) - - if ! git grep -q "OMI_DESKTOP_RELEASE_TAG" "$TARGET_SHA" -- desktop/macos/Backend-Rust/src \ - || ! git grep -q "release_tag" "$TARGET_SHA" -- desktop/macos/Backend-Rust/src; then - echo "$RELEASE_TAG does not include /health release identity support." >&2 - echo "Choose a release built after the stable-promotion workflow landed; older tags cannot be promoted or rolled back safely." >&2 - exit 1 - fi - - if [ -n "$CURRENT_SHA" ] && [ "$TARGET_SHA" != "$CURRENT_SHA" ] \ - && git merge-base --is-ancestor "$TARGET_SHA" "$CURRENT_SHA"; then - echo "$RELEASE_TAG ($TARGET_SHA) is older than tracked prod ($CURRENT_SHA)." >&2 - echo "This promotion workflow is roll-forward only because both desktop update feeds choose the newest stable release." >&2 - echo "Ship a newer fixed release for prod, or handle infrastructure rollback outside this stable-promotion path." >&2 - exit 1 - fi - - { - echo "target_sha=$TARGET_SHA" - echo "target_image=gcr.io/${{ vars.GCP_PROJECT_ID }}/${SERVICE}:$TARGET_SHA" - } >> "$GITHUB_OUTPUT" - - echo "Promotion target: $RELEASE_TAG ($TARGET_SHA)" >> "$GITHUB_STEP_SUMMARY" - - - name: Preflight Omi Bot token configuration - id: app-token-preflight + - name: Generate Omi Bot token + id: app-token uses: actions/create-github-app-token@v3 with: app-id: ${{ secrets.OMI_BOT_APP_ID }} private-key: ${{ secrets.OMI_BOT_PRIVATE_KEY }} - - name: Preflight Omi Bot repository access + - name: Validate trusted qualification run for initial promotion + if: ${{ inputs.operation == 'promote' }} + id: candidate env: - GH_TOKEN: ${{ steps.app-token-preflight.outputs.token }} + GH_TOKEN: ${{ steps.app-token.outputs.token }} REPO: ${{ github.repository }} - run: gh api "repos/${REPO}" --jq .full_name >/dev/null - + RELEASE_TAG: ${{ inputs.release_tag }} + CONFIRM: ${{ inputs.confirm }} + QUALIFICATION_RUN_ID: ${{ inputs.qualification_run_id }} + run: | + set -euo pipefail + test "$CONFIRM" = promote-stable || { echo 'confirm must be exactly: promote-stable' >&2; exit 1; } + printf '%s\n' "$RELEASE_TAG" | grep -qE '^v[0-9]+\.[0-9]+(\.[0-9]+)?\+[0-9]+-macos$' || { + echo "Invalid macOS release tag: $RELEASE_TAG" >&2; exit 1; + } + git fetch --force origin "+refs/tags/${RELEASE_TAG}:refs/tags/${RELEASE_TAG}" + TARGET_SHA=$(git rev-list -n1 "$RELEASE_TAG") + [[ "$QUALIFICATION_RUN_ID" =~ ^[0-9]+$ ]] || { echo 'qualification_run_id must be numeric' >&2; exit 1; } + run=$(gh api "repos/$REPO/actions/runs/$QUALIFICATION_RUN_ID") + test "$(jq -r .conclusion <<<"$run")" = success + test "$(jq -r .repository.full_name <<<"$run")" = "$REPO" + test "$(jq -r .head_repository.full_name <<<"$run")" = "$REPO" + test "$(jq -r .event <<<"$run")" = workflow_dispatch + test "$(jq -r .path <<<"$run")" = .github/workflows/desktop_qualify_beta.yml + test "$(jq -r .head_branch <<<"$run")" = main + test "$(jq -r .name <<<"$run")" = 'Qualify Desktop Beta Candidate' + test "$(jq -r .head_sha <<<"$run")" = "$TARGET_SHA" - name: Google Auth uses: google-github-actions/auth@v3 with: credentials_json: ${{ secrets.GCP_CREDENTIALS }} - - name: Login to GCR - run: gcloud auth configure-docker - - - name: Preflight Firestore bridge release - env: - PROJECT_ID: ${{ vars.GCP_PROJECT_ID }} - FIRESTORE_DOC_ID: ${{ steps.plan.outputs.firestore_doc_id }} - EXPECTED_VERSION: ${{ steps.plan.outputs.release_version }} - EXPECTED_BUILD_NUMBER: ${{ steps.plan.outputs.release_build_number }} - run: | - set -euo pipefail - ACCESS_TOKEN=$(gcloud auth print-access-token) - DOC_URL="https://firestore.googleapis.com/v1/projects/${PROJECT_ID}/databases/(default)/documents/desktop_releases/${FIRESTORE_DOC_ID}" - FIRESTORE_DOC=$(curl -fsS \ - -H "Authorization: Bearer ${ACCESS_TOKEN}" \ - "$DOC_URL") - - FIRESTORE_DOC="$FIRESTORE_DOC" python3 - <<'PY' - import json - import os - import sys - - doc = json.loads(os.environ["FIRESTORE_DOC"]) - fields = doc.get("fields", {}) - - def string_field(name): - return fields.get(name, {}).get("stringValue", "") - - def bool_field(name): - return fields.get(name, {}).get("booleanValue") - - def int_field(name): - return fields.get(name, {}).get("integerValue") - - channel = string_field("channel") - version = string_field("version") - is_live = bool_field("is_live") - missing = [ - name - for name in ("version", "download_url", "ed_signature") - if not string_field(name) - ] - if channel not in {"beta", "stable"}: - print(f"Firestore bridge release must be beta or stable before prod promotion, got {channel!r}", file=sys.stderr) - sys.exit(1) - if version != os.environ["EXPECTED_VERSION"]: - print( - f"Firestore bridge release version mismatch: expected {os.environ['EXPECTED_VERSION']!r}, got {version!r}", - file=sys.stderr, - ) - sys.exit(1) - build_number = int_field("build_number") - if build_number is None or int(build_number) != int(os.environ["EXPECTED_BUILD_NUMBER"]): - print( - f"Firestore bridge release build_number mismatch: expected {os.environ['EXPECTED_BUILD_NUMBER']!r}, got {build_number!r}", - file=sys.stderr, - ) - sys.exit(1) - if is_live is not True: - print(f"Firestore bridge release must have is_live=true before prod promotion, got {is_live!r}", file=sys.stderr) - sys.exit(1) - if missing: - print(f"Firestore bridge release is missing required field(s): {', '.join(missing)}", file=sys.stderr) - sys.exit(1) - print(f"Firestore bridge release preflight OK: channel={channel}") - PY - - - name: Check current prod backend - id: current - env: - PROJECT_ID: ${{ vars.GCP_PROJECT_ID }} - TARGET_IMAGE: ${{ steps.plan.outputs.target_image }} - TARGET_TAG: ${{ inputs.release_tag }} - TARGET_SHA: ${{ steps.plan.outputs.target_sha }} - run: | - set -euo pipefail - - SERVICE_URL=$(gcloud run services describe "$SERVICE" \ - --region "$REGION" \ - --project "$PROJECT_ID" \ - --format='value(status.url)') - CURRENT_IMAGE=$(gcloud run services describe "$SERVICE" \ - --region "$REGION" \ - --project "$PROJECT_ID" \ - --format='value(spec.template.spec.containers[0].image)') - - HEALTH_JSON=$(curl -fsS --max-time 15 "${SERVICE_URL%/}/health" || true) - HEALTH_MATCH=false - if [ -n "$HEALTH_JSON" ]; then - HEALTH_MATCH=$(HEALTH_JSON="$HEALTH_JSON" python3 - <<'PY' - import json - import os - import sys - - try: - health = json.loads(os.environ["HEALTH_JSON"]) - except Exception: - print("false") - sys.exit(0) - - ok = ( - health.get("release_tag") == os.environ["TARGET_TAG"] - and health.get("release_sha") == os.environ["TARGET_SHA"] - and health.get("release_channel") == "stable" - ) - print("true" if ok else "false") - PY - ) - fi - - DEPLOY_NEEDED=true - if [ "$CURRENT_IMAGE" = "$TARGET_IMAGE" ] && [ "$HEALTH_MATCH" = "true" ]; then - DEPLOY_NEEDED=false - fi - - { - echo "service_url=$SERVICE_URL" - echo "current_image=$CURRENT_IMAGE" - echo "deploy_needed=$DEPLOY_NEEDED" - } >> "$GITHUB_OUTPUT" - - { - echo "Current prod image: $CURRENT_IMAGE" - echo "Target prod image: $TARGET_IMAGE" - echo "Health identity already matches target: $HEALTH_MATCH" - echo "Deploy needed: $DEPLOY_NEEDED" - } >> "$GITHUB_STEP_SUMMARY" - - - name: Checkout release source - if: steps.current.outputs.deploy_needed == 'true' - uses: actions/checkout@v7 - with: - ref: ${{ steps.plan.outputs.target_sha }} - fetch-depth: 0 - path: release-source - - - name: Set up Docker Buildx - if: steps.current.outputs.deploy_needed == 'true' - uses: docker/setup-buildx-action@v4 - - - name: Google Service Account - if: steps.current.outputs.deploy_needed == 'true' - run: echo "${{ secrets.GCP_SERVICE_ACCOUNT }}" | base64 -d > ./release-source/desktop/macos/Backend-Rust/google-credentials.json - - - name: Validate desktop OAuth base API URL - if: steps.current.outputs.deploy_needed == 'true' - env: - DESKTOP_BACKEND_BASE_API_URL: ${{ vars.DESKTOP_BACKEND_BASE_API_URL }} - run: | - set -euo pipefail - if [ -z "${DESKTOP_BACKEND_BASE_API_URL//[[:space:]]/}" ]; then - echo "DESKTOP_BACKEND_BASE_API_URL must be configured for desktop OAuth callbacks" >&2 - exit 1 - fi - - - name: Validate desktop Calendar API key secret - if: steps.current.outputs.deploy_needed == 'true' + - name: Fetch exact retained qualified manifest env: PROJECT_ID: ${{ vars.GCP_PROJECT_ID }} + RELEASE_TAG: ${{ inputs.release_tag }} run: | set -euo pipefail - gcloud secrets versions access latest \ - --project="$PROJECT_ID" \ - --secret=DESKTOP_GOOGLE_CALENDAR_API_KEY > /tmp/desktop-google-calendar-api-key + ADMIN_KEY=$(gcloud secrets versions access latest --secret=ADMIN_KEY --project "$PROJECT_ID") + curl -fsS -H "secret-key: $ADMIN_KEY" "https://api.omi.me/v2/desktop/releases/$RELEASE_TAG" > /tmp/desktop-manifest-response.json + jq '.manifest' /tmp/desktop-manifest-response.json > /tmp/desktop-release-manifest.json python3 - <<'PY' - from pathlib import Path - - value = Path("/tmp/desktop-google-calendar-api-key").read_bytes() - if value.endswith(b"\n"): - raise SystemExit("DESKTOP_GOOGLE_CALENDAR_API_KEY must not include a trailing newline") - if len(value) != 39 or not value.startswith(b"AIza"): - raise SystemExit("DESKTOP_GOOGLE_CALENDAR_API_KEY does not look like a Google API key") + import hashlib, json + response = json.load(open('/tmp/desktop-manifest-response.json')) + manifest = json.load(open('/tmp/desktop-release-manifest.json')) + if manifest.get('release_id') != __import__('os').environ['RELEASE_TAG']: + raise SystemExit('retained manifest does not match requested release') + actual = hashlib.sha256(json.dumps(manifest, sort_keys=True, separators=(',', ':')).encode()).hexdigest() + if actual != response.get('manifest_sha256'): + raise SystemExit('retained manifest canonical identity mismatch') PY - - name: Build and Push Desktop Backend Image - if: steps.current.outputs.deploy_needed == 'true' - uses: docker/build-push-action@v7 - with: - context: ./release-source/desktop/macos/Backend-Rust - file: ./release-source/desktop/macos/Backend-Rust/Dockerfile - push: true - tags: | - ${{ steps.plan.outputs.target_image }} - gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:prod - cache-from: type=registry,ref=gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:buildcache - cache-to: type=registry,ref=gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:buildcache,mode=max - - - name: Deploy Desktop Backend to Production - if: steps.current.outputs.deploy_needed == 'true' - uses: google-github-actions/deploy-cloudrun@v3 - with: - service: ${{ env.SERVICE }} - region: ${{ env.REGION }} - image: ${{ steps.plan.outputs.target_image }} - flags: '--allow-unauthenticated' - wait: true - env_vars: | - FIREBASE_PROJECT_ID=based-hardware - GOOGLE_APPLICATION_CREDENTIALS=/app/google-credentials.json - AGENT_GCS_BUCKET=based-hardware-agent - BASE_API_URL=${{ vars.DESKTOP_BACKEND_BASE_API_URL }} - OMI_DESKTOP_RELEASE_TAG=${{ inputs.release_tag }} - OMI_DESKTOP_RELEASE_SHA=${{ steps.plan.outputs.target_sha }} - OMI_DESKTOP_RELEASE_CHANNEL=stable - secrets: | - ENCRYPTION_SECRET=ENCRYPTION_SECRET:latest - GEMINI_API_KEY=DESKTOP_GEMINI_API_KEY:latest - OPENAI_API_KEY=OPENAI_API_KEY:latest - REDIS_DB_PASSWORD=DESKTOP_REDIS_DB_PASSWORD:latest - FIREBASE_API_KEY=DESKTOP_FIREBASE_API_KEY:latest - REDIS_DB_HOST=DESKTOP_REDIS_DB_HOST:latest - REDIS_DB_PORT=DESKTOP_REDIS_DB_PORT:latest - ANTHROPIC_API_KEY=DESKTOP_ANTHROPIC_API_KEY:latest - DESKTOP_LEGACY_ANTHROPIC_KEY=DESKTOP_LEGACY_ANTHROPIC_KEY:latest - GOOGLE_CALENDAR_API_KEY=DESKTOP_GOOGLE_CALENDAR_API_KEY:latest - RELEASE_SECRET=RELEASE_SECRET:latest - - - name: Route traffic to deployed desktop-backend revision - if: steps.current.outputs.deploy_needed == 'true' + - name: Verify current beta and stable pointer compare-and-swap inputs env: PROJECT_ID: ${{ vars.GCP_PROJECT_ID }} + RELEASE_TAG: ${{ inputs.release_tag }} + EXPECTED_RELEASE_ID: ${{ inputs.expected_current_release_id }} + EXPECTED_GENERATION: ${{ inputs.expected_generation }} + OPERATION: ${{ inputs.operation }} run: | set -euo pipefail - REVISION="$(gcloud run services describe "$SERVICE" \ - --project="$PROJECT_ID" \ - --region="$REGION" \ - --format='value(status.latestCreatedRevisionName)')" - if [ -z "$REVISION" ]; then - echo "Could not determine latest desktop-backend revision" >&2 - exit 1 - fi - gcloud run services update-traffic "$SERVICE" \ - --project="$PROJECT_ID" \ - --region="$REGION" \ - --to-revisions="$REVISION=100" \ - --quiet - - - name: Verify prod backend release identity - env: - PROJECT_ID: ${{ vars.GCP_PROJECT_ID }} - TARGET_TAG: ${{ inputs.release_tag }} - TARGET_SHA: ${{ steps.plan.outputs.target_sha }} - run: | - set -euo pipefail - - SERVICE_URL=$(gcloud run services describe "$SERVICE" \ - --region "$REGION" \ - --project "$PROJECT_ID" \ - --format='value(status.url)') - - for attempt in $(seq 1 30); do - HEALTH_JSON=$(curl -fsS --max-time 15 "${SERVICE_URL%/}/health" || true) - if [ -n "$HEALTH_JSON" ]; then - if HEALTH_JSON="$HEALTH_JSON" python3 - <<'PY' - import json - import os - import sys - - health = json.loads(os.environ["HEALTH_JSON"]) - expected = { - "release_tag": os.environ["TARGET_TAG"], - "release_sha": os.environ["TARGET_SHA"], - "release_channel": "stable", - } - mismatches = { - key: {"expected": value, "actual": health.get(key)} - for key, value in expected.items() - if health.get(key) != value - } - if mismatches: - print(json.dumps({"health": health, "mismatches": mismatches}, indent=2)) - sys.exit(1) - print(json.dumps(health, indent=2)) - PY - then - echo "Verified prod backend identity at $SERVICE_URL/health" >> "$GITHUB_STEP_SUMMARY" - exit 0 - fi - fi - echo "Waiting for prod backend identity ($attempt/30)..." - sleep 10 - done - - echo "Prod backend did not report the promoted release identity." >&2 - exit 1 + ACCESS_TOKEN=$(gcloud auth print-access-token) + BASE="https://firestore.googleapis.com/v1/projects/${PROJECT_ID}/databases/(default)/documents/desktop_update_channels" + curl -fsS -H "Authorization: Bearer $ACCESS_TOKEN" "$BASE/macos-beta" > /tmp/macos-beta-pointer.json + curl -sS -H "Authorization: Bearer $ACCESS_TOKEN" "$BASE/macos-stable" > /tmp/macos-stable-pointer.json + python3 .github/scripts/check_stable_pointer_precondition.py \ + --beta /tmp/macos-beta-pointer.json --stable /tmp/macos-stable-pointer.json \ + --release-id "$RELEASE_TAG" --expected-release-id "$EXPECTED_RELEASE_ID" \ + --expected-generation "$EXPECTED_GENERATION" --operation "$OPERATION" - name: Publish immutable stable repair installer env: - GH_TOKEN: ${{ github.token }} + GCS_DESKTOP_UPDATES_BUCKET: ${{ env.GCS_DESKTOP_UPDATES_BUCKET }} RELEASE_TAG: ${{ inputs.release_tag }} - TARGET_SHA: ${{ steps.plan.outputs.target_sha }} - REPO: ${{ github.repository }} run: | set -euo pipefail - mkdir -p /tmp/desktop-stable-repair-assets - gh release download "$RELEASE_TAG" --repo "$REPO" \ - --pattern 'Omi.zip' --pattern 'Omi.dmg' --pattern 'omi.dmg' --dir /tmp/desktop-stable-repair-assets - ZIP_PATH=/tmp/desktop-stable-repair-assets/Omi.zip - test -f "$ZIP_PATH" - DMG_PATHS=() - for candidate in Omi.dmg omi.dmg; do - candidate_path="/tmp/desktop-stable-repair-assets/$candidate" - if [ -f "$candidate_path" ]; then - DMG_PATHS+=("$candidate_path") - fi - done - if [ "${#DMG_PATHS[@]}" -ne 1 ]; then - echo "Expected exactly one qualified Omi.dmg or omi.dmg release asset." >&2 - exit 1 + python3 .github/scripts/desktop_repair_installer.py --manifest /tmp/desktop-release-manifest.json \ + --bucket "$GCS_DESKTOP_UPDATES_BUCKET" --output-dir /tmp/desktop-stable-repair + REPAIR_SHA=$(sha256sum /tmp/desktop-stable-repair/repair.json | cut -d' ' -f1) + target="$GCS_DESKTOP_UPDATES_BUCKET/stable/${RELEASE_TAG}/repair.json" + if gcloud storage objects describe "$target" --format='value(metadata.sha256)' > /tmp/remote-sha.txt 2>/dev/null; then + test "$(cat /tmp/remote-sha.txt)" = "$REPAIR_SHA" || { echo "Immutable stable repair metadata differs: $target" >&2; exit 1; } + else + gcloud storage cp --if-generation-match=0 --custom-metadata="sha256=$REPAIR_SHA" /tmp/desktop-stable-repair/repair.json "$target" fi - DMG_PATH="${DMG_PATHS[0]}" - ZIP_SHA=$(sha256sum "$ZIP_PATH" | cut -d' ' -f1) - DMG_SHA=$(sha256sum "$DMG_PATH" | cut -d' ' -f1) - - python3 .github/scripts/prepare-desktop-beta-promotion.py \ - --release-json /tmp/desktop-release.json \ - --release-tag "$RELEASE_TAG" \ - --target-sha "$TARGET_SHA" \ - --zip-sha256 "$ZIP_SHA" \ - --dmg-sha256 "$DMG_SHA" \ - --allow-stable-channel \ - --output /tmp/desktop-stable-release-manifest.json - python3 .github/scripts/desktop_repair_installer.py \ - --manifest /tmp/desktop-stable-release-manifest.json \ - --bucket "$GCS_DESKTOP_UPDATES_BUCKET" \ - --output-dir /tmp/desktop-stable-repair - - INSTALLER_URL=$(python3 -c "import json; print(json.load(open('/tmp/desktop-stable-repair/repair.json'))['installer_url'])") - REPAIR_MANIFEST_URL=$(python3 -c "import json; print(json.load(open('/tmp/desktop-stable-repair/latest.json'))['repair_manifest_url'])") - - upload_immutable() { - local source="$1" - local destination="$2" - local content_type="$3" - local existing - existing=$(mktemp) - if gcloud storage objects describe "$destination" >/dev/null 2>&1; then - gcloud storage cp "$destination" "$existing" - cmp -s "$source" "$existing" || { - echo "Refusing to overwrite immutable repair object: $destination" >&2 - exit 1 - } - return - fi - if ! gcloud storage cp --if-generation-match=0 \ - --cache-control='public,max-age=31536000,immutable' \ - --content-type="$content_type" "$source" "$destination"; then - gcloud storage cp "$destination" "$existing" - cmp -s "$source" "$existing" || { - echo "Immutable repair object changed during publication: $destination" >&2 - exit 1 - } - fi - } - upload_immutable "$DMG_PATH" "$GCS_DESKTOP_UPDATES_BUCKET/stable/$RELEASE_TAG/Omi.dmg" \ - 'application/x-apple-diskimage' - upload_immutable /tmp/desktop-stable-repair/repair.json \ - "$GCS_DESKTOP_UPDATES_BUCKET/stable/$RELEASE_TAG/repair.json" 'application/json; charset=utf-8' - curl -fsSI "$INSTALLER_URL" >/dev/null - curl -fsSI "$REPAIR_MANIFEST_URL" >/dev/null - - - name: Promote Firestore release stable + - name: Advance explicit stable pointer env: PROJECT_ID: ${{ vars.GCP_PROJECT_ID }} - FIRESTORE_DOC_ID: ${{ steps.plan.outputs.firestore_doc_id }} + RELEASE_TAG: ${{ inputs.release_tag }} + EXPECTED_RELEASE_ID: ${{ inputs.expected_current_release_id }} + EXPECTED_GENERATION: ${{ inputs.expected_generation }} + OPERATION: ${{ inputs.operation }} run: | set -euo pipefail - - SERVICE_URL=$(gcloud run services describe "$SERVICE" \ - --region "$REGION" \ - --project "$PROJECT_ID" \ - --format='value(status.url)') - RELEASE_SECRET=$(gcloud secrets versions access latest \ - --secret=RELEASE_SECRET \ - --project "$PROJECT_ID") - - RESPONSE=$(curl -sS -w "\n%{http_code}" \ - -X PATCH \ - -H "Content-Type: application/json" \ - -H "X-Release-Secret: $RELEASE_SECRET" \ - -d "{\"doc_id\":\"${FIRESTORE_DOC_ID}\"}" \ - "${SERVICE_URL%/}/updates/releases/promote") - - HTTP_CODE=$(printf '%s\n' "$RESPONSE" | tail -n1) - BODY=$(printf '%s\n' "$RESPONSE" | sed '$d') - if [ "$HTTP_CODE" = "200" ]; then - NEW_CHANNEL=$(BODY="$BODY" python3 - <<'PY' - import json - import os - - print(json.loads(os.environ["BODY"]).get("new_channel", "")) + ADMIN_KEY=$(gcloud secrets versions access latest --secret=ADMIN_KEY --project "$PROJECT_ID") + python3 - <<'PY' > /tmp/desktop-stable-pointer.json + import json, os + payload = {"platform": "macos", "channel": "stable", "release_id": os.environ["RELEASE_TAG"], + "expected_generation": int(os.environ["EXPECTED_GENERATION"]), "operation": os.environ["OPERATION"]} + if os.environ["EXPECTED_RELEASE_ID"]: + payload["expected_current_release_id"] = os.environ["EXPECTED_RELEASE_ID"] + print(json.dumps(payload)) PY - ) - if [ "$NEW_CHANNEL" != "stable" ]; then - echo "$BODY" >&2 - echo "Firestore release $FIRESTORE_DOC_ID promoted to $NEW_CHANNEL, not stable." >&2 - exit 1 - fi - echo "Promoted Firestore release $FIRESTORE_DOC_ID to stable" >> "$GITHUB_STEP_SUMMARY" - elif [ "$HTTP_CODE" = "400" ] && printf '%s\n' "$BODY" | grep -qi "already on stable"; then - echo "Firestore release $FIRESTORE_DOC_ID is already stable" >> "$GITHUB_STEP_SUMMARY" - else - echo "$BODY" >&2 - echo "Failed to promote Firestore release $FIRESTORE_DOC_ID (HTTP $HTTP_CODE)" >&2 - exit 1 - fi + curl -fsS -X POST -H 'Content-Type: application/json' -H "secret-key: $ADMIN_KEY" \ + --data-binary @/tmp/desktop-stable-pointer.json https://api.omi.me/v2/desktop/channels/promote \ + > /tmp/desktop-stable-pointer-response.json + python3 -m json.tool /tmp/desktop-stable-pointer-response.json - - name: Advance explicit stable pointer + - name: Bridge stable for legacy desktop clients env: PROJECT_ID: ${{ vars.GCP_PROJECT_ID }} - RELEASE_TAG: ${{ inputs.release_tag }} run: | set -euo pipefail - ADMIN_KEY=$(gcloud secrets versions access latest \ - --secret=ADMIN_KEY \ - --project "$PROJECT_ID") - python3 - "$RELEASE_TAG" <<'PY' > /tmp/desktop-stable-pointer.json + SERVICE_URL=$(gcloud run services describe desktop-backend --region us-central1 --project "$PROJECT_ID" --format='value(status.url)') + RELEASE_SECRET=$(gcloud secrets versions access latest --secret=RELEASE_SECRET --project "$PROJECT_ID") + python3 - <<'PY' > /tmp/legacy-desktop-release.json import json - import sys - print(json.dumps({"platform": "macos", "channel": "stable", "release_id": sys.argv[1]})) + manifest = json.load(open('/tmp/desktop-release-manifest.json')) + print(json.dumps({"version": manifest["version"], "build_number": manifest["build_number"], + "download_url": manifest["zip_url"], "ed_signature": manifest["ed_signature"], "changelog": manifest["changelog"], + "is_live": True, "is_critical": manifest["mandatory"], "channel": "stable"})) PY - curl -fsS \ - -X POST \ - -H "Content-Type: application/json" \ - -H "secret-key: $ADMIN_KEY" \ - --data-binary @/tmp/desktop-stable-pointer.json \ - https://api.omi.me/v2/desktop/channels/promote \ - > /tmp/desktop-stable-pointer-response.json - python3 -m json.tool /tmp/desktop-stable-pointer-response.json - echo "Advanced explicit stable pointer to $RELEASE_TAG" >> "$GITHUB_STEP_SUMMARY" + RESPONSE=$(curl -sS -w '\n%{http_code}' -X POST -H 'Content-Type: application/json' \ + -H "X-Release-Secret: $RELEASE_SECRET" --data-binary @/tmp/legacy-desktop-release.json "${SERVICE_URL%/}/updates/releases") + CODE=$(printf '%s\n' "$RESPONSE" | tail -n1) + test "$CODE" = 201 -o "$CODE" = 409 || { printf '%s\n' "$RESPONSE" >&2; exit 1; } - name: Publish latest stable repair route env: - RELEASE_TAG: ${{ inputs.release_tag }} + GCS_DESKTOP_UPDATES_BUCKET: ${{ env.GCS_DESKTOP_UPDATES_BUCKET }} run: | set -euo pipefail - for file in index.html latest.json; do - gcloud storage cp --cache-control='no-store,max-age=0' \ - --content-type="$([ "$file" = index.html ] && echo 'text/html; charset=utf-8' || echo 'application/json; charset=utf-8')" \ - "/tmp/desktop-stable-repair/$file" "$GCS_DESKTOP_UPDATES_BUCKET/stable/$file" + for file in latest.json index.html; do + gcloud storage cp --cache-control='no-store,max-age=0' "/tmp/desktop-stable-repair/$file" "$GCS_DESKTOP_UPDATES_BUCKET/stable/$file" done - LATEST_URL=$(python3 - <<'PY' - import json - from pathlib import Path - - repair_manifest_url = json.loads(Path('/tmp/desktop-stable-repair/latest.json').read_text())['repair_manifest_url'] - print(repair_manifest_url.rsplit('/', 2)[0] + '/latest.json') - PY - ) - curl -fsS "$LATEST_URL" > /tmp/desktop-stable-repair-live.json - RELEASE_TAG="$RELEASE_TAG" python3 - <<'PY' - import json - import os - from pathlib import Path - - latest = json.loads(Path('/tmp/desktop-stable-repair-live.json').read_text()) - if latest['channel'] != 'stable': - raise SystemExit('repair route must be stable only') - if latest['release_id'] != os.environ['RELEASE_TAG']: - raise SystemExit('repair route release does not match stable promotion') - PY - echo "Verified stable repair route: $LATEST_URL" >> "$GITHUB_STEP_SUMMARY" - - name: Generate Omi Bot token for release mutations - id: app-token - uses: actions/create-github-app-token@v3 - with: - app-id: ${{ secrets.OMI_BOT_APP_ID }} - private-key: ${{ secrets.OMI_BOT_PRIVATE_KEY }} - - - name: Mark GitHub Release stable + - name: Mark GitHub release live stable + if: ${{ inputs.operation == 'promote' }} env: GH_TOKEN: ${{ steps.app-token.outputs.token }} RELEASE_TAG: ${{ inputs.release_tag }} @@ -636,42 +211,29 @@ jobs: run: | set -euo pipefail gh release view "$RELEASE_TAG" --repo "$REPO" --json body -q .body > /tmp/release-body.md - python3 .github/scripts/mark-desktop-release-stable.py \ - --input /tmp/release-body.md \ - --output /tmp/release-body-stable.md - gh release edit "$RELEASE_TAG" --repo "$REPO" --notes-file /tmp/release-body-stable.md + python3 .github/scripts/mark-desktop-release-stable.py --input /tmp/release-body.md --output /tmp/release-body-stable.md + gh release edit "$RELEASE_TAG" --repo "$REPO" --title "Omi Desktop ${RELEASE_TAG#v}" --notes-file /tmp/release-body-stable.md - - name: Clear desktop update cache + - name: Verify exact pointer, hashes, and stable feed env: PROJECT_ID: ${{ vars.GCP_PROJECT_ID }} + RELEASE_TAG: ${{ inputs.release_tag }} run: | set -euo pipefail - ADMIN_KEY=$(gcloud secrets versions access latest \ - --secret=ADMIN_KEY \ - --project "$PROJECT_ID" 2>/dev/null || true) - if [ -z "$ADMIN_KEY" ]; then - echo "ADMIN_KEY unavailable; skipping desktop update cache clear." >> "$GITHUB_STEP_SUMMARY" - exit 0 - fi - curl -fsS \ - -X POST \ - -H "secret-key: $ADMIN_KEY" \ - "https://api.omi.me/v2/desktop/clear-cache" >/dev/null - echo "Cleared desktop update cache" >> "$GITHUB_STEP_SUMMARY" - - - name: Advance prod-tracking tag - env: - GH_TOKEN: ${{ steps.app-token.outputs.token }} - TARGET_SHA: ${{ steps.plan.outputs.target_sha }} - run: | - set -euo pipefail - python3 - <<'PY' > /tmp/update-desktop-backend-prod-ref.json - import json - import os - - print(json.dumps({"sha": os.environ["TARGET_SHA"], "force": True})) + ACCESS_TOKEN=$(gcloud auth print-access-token) + BASE="https://firestore.googleapis.com/v1/projects/${PROJECT_ID}/databases/(default)/documents" + curl -fsS -H "Authorization: Bearer $ACCESS_TOKEN" "$BASE/desktop_update_channels/macos-stable" > /tmp/final-stable-pointer.json + curl -fsS -H "Authorization: Bearer $ACCESS_TOKEN" "$BASE/desktop_release_manifests/$RELEASE_TAG" > /tmp/final-stable-manifest.json + python3 - <<'PY' + import json, os, sys + pointer = json.load(open('/tmp/final-stable-pointer.json')).get('fields', {}) + manifest = json.load(open('/tmp/final-stable-manifest.json')).get('fields', {}) + release_id = pointer.get('release_id', {}).get('stringValue') + if release_id != os.environ['RELEASE_TAG']: sys.exit('stable pointer does not resolve to the requested release') + expected = json.load(open('/tmp/desktop-release-manifest.json')) + for key in ('zip_sha256', 'dmg_sha256'): + if manifest.get(key, {}).get('stringValue') != expected[key]: sys.exit(f'manifest {key} mismatch') PY - if ! gh api --method PATCH "repos/${GITHUB_REPOSITORY}/git/refs/tags/${TRACK_TAG}" --input /tmp/update-desktop-backend-prod-ref.json >/dev/null; then - gh api --method POST "repos/${GITHUB_REPOSITORY}/git/refs" -f ref="refs/tags/${TRACK_TAG}" -f sha="${TARGET_SHA}" >/dev/null - fi - echo "Promoted desktop-backend to PROD - ${{ inputs.release_tag }} (${{ steps.plan.outputs.target_sha }})" >> "$GITHUB_STEP_SUMMARY" + curl -fsS 'https://api.omi.me/v2/desktop/appcast.xml?identity=stable' > /tmp/stable-appcast.xml + python3 .github/scripts/verify_stable_appcast.py --manifest /tmp/desktop-release-manifest.json --feed /tmp/stable-appcast.xml + echo "Stable pointer, exact manifest hashes, and appcast verified for $RELEASE_TAG." >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/desktop_qualify_beta.yml b/.github/workflows/desktop_qualify_beta.yml index 11aa681a4c0..214963e0a29 100644 --- a/.github/workflows/desktop_qualify_beta.yml +++ b/.github/workflows/desktop_qualify_beta.yml @@ -7,14 +7,6 @@ on: description: 'Newest v*-macos candidate tag to qualify' required: true type: string - dispatch_key: - description: 'Deterministic Codemagic handoff key; leave empty for a deduplicated manual run' - required: false - type: string - retry_nonce: - description: 'Required only to deliberately retry a terminal qualification failure' - required: false - type: string run-name: Qualify desktop beta ${{ inputs.release_tag }} @@ -22,14 +14,14 @@ permissions: contents: read concurrency: - # This is the authoritative atomic serialiser for release-body claims. No - # outside dispatcher may edit qualificationDispatch* metadata. + # Per-tag serialization plus immutable qualification evidence make retries + # safe without a second release-body claim state machine. group: desktop-beta-qualification-${{ inputs.release_tag }} cancel-in-progress: false # This workflow deliberately runs only on the dedicated, trusted macOS runner. # Do not add pull_request or push triggers: the runner has Docker and a GitHub -# App token that can edit candidate evidence and request beta promotion. +# App token that can request beta promotion after exact qualification passes. jobs: qualify: runs-on: [self-hosted, macos, omi-desktop-qualification] @@ -51,61 +43,8 @@ jobs: app-id: ${{ secrets.OMI_BOT_APP_ID }} private-key: ${{ secrets.OMI_BOT_PRIVATE_KEY }} - - name: Claim one deduplicated qualification execution - id: claim - env: - GH_TOKEN: ${{ steps.app-token.outputs.token }} - RELEASE_TAG: ${{ inputs.release_tag }} - REPO: ${{ github.repository }} - DISPATCH_KEY_INPUT: ${{ inputs.dispatch_key }} - RETRY_NONCE: ${{ inputs.retry_nonce }} - run: | - set -euo pipefail - mkdir -p /tmp/desktop-beta-qualification - if [ -n "$RETRY_NONCE" ]; then - if ! printf '%s\n' "$RETRY_NONCE" | grep -qE '^[A-Za-z0-9._:-]{1,80}$'; then - echo "retry_nonce must be a bounded single-line token" >&2 - exit 1 - fi - DISPATCH_KEY="manual:${RELEASE_TAG}:${RETRY_NONCE}" - CLAIM_ARGS=(--allow-retry) - elif [ -n "$DISPATCH_KEY_INPUT" ]; then - DISPATCH_KEY="$DISPATCH_KEY_INPUT" - CLAIM_ARGS=() - else - DISPATCH_KEY="manual:${RELEASE_TAG}" - CLAIM_ARGS=() - fi - gh release view "$RELEASE_TAG" --repo "$REPO" --json body --jq .body \ - > /tmp/desktop-beta-qualification/release-body-before-claim.md - python3 .github/scripts/desktop_qualification_dispatch.py claim \ - --input /tmp/desktop-beta-qualification/release-body-before-claim.md \ - --output /tmp/desktop-beta-qualification/release-body-after-claim.md \ - --key "$DISPATCH_KEY" --updated-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --run-id "$GITHUB_RUN_ID" --run-url "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" \ - "${CLAIM_ARGS[@]}" > /tmp/desktop-beta-qualification/claim.json - SHOULD_RUN=$(python3 -c 'import json; print(str(json.load(open("/tmp/desktop-beta-qualification/claim.json"))["should_run"]).lower())') - REASON=$(python3 -c 'import json; print(json.load(open("/tmp/desktop-beta-qualification/claim.json"))["reason"])') - if ! cmp -s /tmp/desktop-beta-qualification/release-body-before-claim.md \ - /tmp/desktop-beta-qualification/release-body-after-claim.md; then - gh release edit "$RELEASE_TAG" --repo "$REPO" \ - --notes-file /tmp/desktop-beta-qualification/release-body-after-claim.md - fi - { - echo "dispatch_key=$DISPATCH_KEY" - echo "should_run=$SHOULD_RUN" - python3 -c 'import json; print("should_promote=" + str(json.load(open("/tmp/desktop-beta-qualification/claim.json"))["should_promote"]).lower())' - } >> "$GITHUB_OUTPUT" - { - echo "## Desktop qualification dispatch" - echo "- Candidate: \`$RELEASE_TAG\`" - echo "- Dispatch key: \`$DISPATCH_KEY\`" - echo "- Decision: $REASON" - } >> "$GITHUB_STEP_SUMMARY" - - name: Download and validate newest candidate evidence id: candidate - if: steps.claim.outputs.should_run == 'true' env: GH_TOKEN: ${{ steps.app-token.outputs.token }} RELEASE_TAG: ${{ inputs.release_tag }} @@ -128,7 +67,8 @@ jobs: } git fetch --force origin "+refs/tags/${RELEASE_TAG}:refs/tags/${RELEASE_TAG}" - LATEST_TAG=$(git tag -l 'v*-macos' --sort=-v:refname | head -1) + LATEST_TAG=$(git for-each-ref --count=1 --sort=-v:refname --format='%(refname:strip=2)' 'refs/tags/v*-macos') + test -n "$LATEST_TAG" TARGET_SHA=$(git rev-list -n1 "$RELEASE_TAG") CHECKOUT_SHA=$(git rev-parse "$RELEASE_TAG") rm -rf /tmp/desktop-beta-qualification @@ -139,10 +79,13 @@ jobs: gh release download "$RELEASE_TAG" --repo "$REPO" \ --pattern 'desktop-smoke-result.json' \ --dir /tmp/desktop-beta-qualification - # Present only on releases that ship the side-by-side Omi Beta identity. gh release download "$RELEASE_TAG" --repo "$REPO" \ --pattern 'desktop-smoke-result-beta.json' \ --dir /tmp/desktop-beta-qualification 2>/dev/null || true + gh release download "$RELEASE_TAG" --repo "$REPO" \ + --pattern 'Omi.zip' --pattern 'Omi.dmg' --pattern 'omi.dmg' \ + --pattern 'Omi.Beta.zip' --pattern 'omi-beta.dmg' \ + --dir /tmp/desktop-beta-qualification/assets python3 .github/scripts/check-desktop-auto-beta-candidate.py \ --release-json /tmp/desktop-beta-qualification/release.json \ @@ -156,7 +99,6 @@ jobs: - name: Qualify exact candidate on hermetic stack id: qualify - if: steps.claim.outputs.should_run == 'true' env: GH_TOKEN: ${{ steps.app-token.outputs.token }} RELEASE_TAG: ${{ inputs.release_tag }} @@ -165,37 +107,41 @@ jobs: desktop/macos/scripts/qualify-desktop-beta.sh \ --automatic \ --no-promote \ + --github-actions-artifact \ --signed-smoke-result /tmp/desktop-beta-qualification/desktop-smoke-result.json \ --candidate-gate-result /tmp/desktop-beta-qualification/candidate-gate.json \ "$RELEASE_TAG" - - name: Record terminal qualification state - if: always() && steps.claim.outputs.should_run == 'true' + - name: Create immutable qualification evidence + id: evidence + if: steps.candidate.outcome == 'success' && steps.qualify.outcome == 'success' env: - GH_TOKEN: ${{ steps.app-token.outputs.token }} RELEASE_TAG: ${{ inputs.release_tag }} - REPO: ${{ github.repository }} - DISPATCH_KEY: ${{ steps.claim.outputs.dispatch_key }} - CANDIDATE_OUTCOME: ${{ steps.candidate.outcome }} - QUALIFICATION_OUTCOME: ${{ steps.qualify.outcome }} run: | set -euo pipefail - PASSED=false - if [ "$CANDIDATE_OUTCOME" = success ] && [ "$QUALIFICATION_OUTCOME" = success ]; then - PASSED=true - fi - gh release view "$RELEASE_TAG" --repo "$REPO" --json body --jq .body \ - > /tmp/desktop-beta-qualification/release-body-before-complete.md - python3 .github/scripts/desktop_qualification_dispatch.py complete \ - --input /tmp/desktop-beta-qualification/release-body-before-complete.md \ - --output /tmp/desktop-beta-qualification/release-body-after-complete.md \ - --key "$DISPATCH_KEY" --updated-at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --passed "$PASSED" \ - > /tmp/desktop-beta-qualification/complete.json - gh release edit "$RELEASE_TAG" --repo "$REPO" \ - --notes-file /tmp/desktop-beta-qualification/release-body-after-complete.md + TARGET_SHA=$(git rev-list -n1 "$RELEASE_TAG") + STABLE_DMG=$(find /tmp/desktop-beta-qualification/assets -maxdepth 1 -type f \( -name Omi.dmg -o -name omi.dmg \) -print -quit) + test -n "$STABLE_DMG" + python3 .github/scripts/desktop_qualification_evidence.py build \ + --release-json /tmp/desktop-beta-qualification/release.json \ + --release-tag "$RELEASE_TAG" --source-sha "$TARGET_SHA" \ + --candidate-gate /tmp/desktop-beta-qualification/candidate-gate.json \ + --asset Omi.zip=/tmp/desktop-beta-qualification/assets/Omi.zip \ + --asset "$(basename "$STABLE_DMG")=$STABLE_DMG" \ + --asset Omi.Beta.zip=/tmp/desktop-beta-qualification/assets/Omi.Beta.zip \ + --asset omi-beta.dmg=/tmp/desktop-beta-qualification/assets/omi-beta.dmg \ + --evidence /tmp/desktop-beta-qualification/qualification-evidence.json + + - name: Upload immutable qualification evidence + if: steps.evidence.outcome == 'success' + uses: actions/upload-artifact@v7 + with: + name: desktop-qualification-evidence-${{ inputs.release_tag }} + path: /tmp/desktop-beta-qualification/qualification-evidence.json + if-no-files-found: error - name: Request protected beta promotion - if: (steps.claim.outputs.should_run == 'true' && steps.candidate.outcome == 'success' && steps.qualify.outcome == 'success') || steps.claim.outputs.should_promote == 'true' + if: steps.candidate.outcome == 'success' && steps.qualify.outcome == 'success' env: GH_TOKEN: ${{ steps.app-token.outputs.token }} RELEASE_TAG: ${{ inputs.release_tag }} @@ -204,4 +150,5 @@ jobs: set -euo pipefail gh workflow run desktop_promote_beta.yml --repo "$REPO" \ -f release_tag="$RELEASE_TAG" \ - -f automatic=true + -f automatic=true \ + -f qualification_run_id="$GITHUB_RUN_ID" diff --git a/.github/workflows/gcp_backend.yml b/.github/workflows/gcp_backend.yml index 0542b364517..7ebf7f4a6cc 100644 --- a/.github/workflows/gcp_backend.yml +++ b/.github/workflows/gcp_backend.yml @@ -68,6 +68,30 @@ env: TRANSCRIPTION_CANDIDATE_TAG: stt-gate-${{ github.run_id }}-${{ github.run_attempt }} jobs: + validate-production-boundary: + if: >- + github.event.inputs.mode == 'deploy' + runs-on: ubuntu-latest + steps: + - name: Reject unsupported production deployment boundary before side effects + env: + DEPLOY_ENVIRONMENT: ${{ github.event.inputs.environment }} + DEPLOY_TARGETS: ${{ github.event.inputs.deploy_targets }} + run: | + set -euo pipefail + if [[ "$DEPLOY_ENVIRONMENT" != "development" && "$DEPLOY_ENVIRONMENT" != "prod" ]]; then + echo "Invalid environment: $DEPLOY_ENVIRONMENT. Must be 'development' or 'prod'." >&2 + exit 1 + fi + if [[ "$DEPLOY_TARGETS" != "all" && "$DEPLOY_TARGETS" != "cloud-run-only" ]]; then + echo "Invalid deploy_targets: $DEPLOY_TARGETS. Must be 'all' or 'cloud-run-only'." >&2 + exit 1 + fi + if [[ "$DEPLOY_ENVIRONMENT" == "prod" && "$DEPLOY_TARGETS" == "all" ]]; then + echo 'ERROR: environment=prod, deploy_targets=all is unsupported: transactional GKE/config rollback parity does not exist. Use cloud-run-only; direct GKE workflows remain the production path for those surfaces.' >&2 + exit 1 + fi + repair-traffic: if: >- github.ref == 'refs/heads/main' && @@ -122,6 +146,7 @@ jobs: --cloud-run-service backend-integration || true firestore_readiness: + needs: validate-production-boundary if: >- github.ref == 'refs/heads/main' && github.event.inputs.mode == 'deploy' @@ -143,11 +168,6 @@ jobs: echo "Invalid environment: $DEPLOY_ENVIRONMENT. Must be 'development' or 'prod'." exit 1 fi - if [[ "$DEPLOY_ENVIRONMENT" == "prod" ]]; then - echo "Production artifact deploys are release-record-only; use deploy-release-ring.yml with ring=prod." >&2 - exit 1 - fi - - name: Require read-only Firestore credentials env: GCP_FIRESTORE_READONLY_CREDENTIALS: ${{ secrets.GCP_FIRESTORE_READONLY_CREDENTIALS }} @@ -296,7 +316,7 @@ jobs: "The gate being unusable is the defect. Fix the gate, then close this.")" deploy: - needs: firestore_readiness + needs: [validate-production-boundary, firestore_readiness] if: >- github.ref == 'refs/heads/main' && github.event.inputs.mode == 'deploy' @@ -391,6 +411,7 @@ jobs: - name: Determine whether this deploy requests gateway-first serving id: gateway-intent + if: ${{ github.event.inputs.deploy_targets == 'all' }} run: | python3 backend/scripts/verify-llm-gateway-serving.py \ --environment="${{ vars.ENV }}" \ @@ -400,7 +421,7 @@ jobs: --intent-only >> "$GITHUB_OUTPUT" - name: Get GKE credentials for gateway serving gate - if: ${{ steps.gateway-intent.outputs.enabled == 'true' }} + if: ${{ github.event.inputs.deploy_targets == 'all' && steps.gateway-intent.outputs.enabled == 'true' }} uses: google-github-actions/get-gke-credentials@v3 with: cluster_name: ${{ vars.GKE_CLUSTER }} @@ -409,7 +430,7 @@ jobs: - name: Verify LLM gateway control plane before promotion id: gateway-serving - if: ${{ steps.gateway-intent.outputs.enabled == 'true' }} + if: ${{ github.event.inputs.deploy_targets == 'all' && steps.gateway-intent.outputs.enabled == 'true' }} run: | python3 backend/scripts/verify-llm-gateway-serving.py \ --environment="${{ vars.ENV }}" \ @@ -461,7 +482,7 @@ jobs: docker push gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${{ steps.image-tag.outputs.short_sha }} - name: Probe LLM gateway from the Cloud Run VPC before promotion - if: ${{ steps.gateway-intent.outputs.enabled == 'true' }} + if: ${{ github.event.inputs.deploy_targets == 'all' && steps.gateway-intent.outputs.enabled == 'true' }} run: | bash backend/scripts/probe-llm-gateway-from-cloud-run.sh \ --project "${{ vars.GCP_PROJECT_ID }}" \ @@ -523,7 +544,7 @@ jobs: no_traffic: true flags: >- --revision-suffix=${{ steps.image-tag.outputs.revision_suffix }} - ${{ github.event.inputs.environment == 'development' && format('--tag={0}', env.TRANSCRIPTION_CANDIDATE_TAG) || '' }} + ${{ (github.event.inputs.environment == 'development' || github.event.inputs.deploy_targets == 'cloud-run-only') && format('--tag={0}', env.TRANSCRIPTION_CANDIDATE_TAG) || '' }} ${{ steps.runtime-env.outputs.cloud_run_flags }} env_vars: ${{ steps.runtime-env.outputs.backend_env_vars }} secrets: ${{ steps.runtime-env.outputs.backend_secrets }} @@ -571,6 +592,7 @@ jobs: echo "revision=backend-sync-${{ steps.image-tag.outputs.revision_suffix }}" >> "$GITHUB_OUTPUT" - name: Provision sync-backfill platform + if: ${{ github.event.inputs.deploy_targets == 'all' }} uses: ./.github/actions/sync-backfill-lifecycle with: mode: platform @@ -582,7 +604,7 @@ jobs: alert_notification_channels: ${{ vars.SYNC_BACKFILL_ALERT_NOTIFICATION_CHANNELS }} - name: Configure production conversation-finalization Cloud Tasks queue - if: ${{ vars.ENV == 'prod' }} + if: ${{ vars.ENV == 'prod' && github.event.inputs.deploy_targets == 'all' }} env: TASKS_INVOKER_SA: ${{ steps.account-deletion-target.outputs.listen_finalization_tasks_invoker_sa }} run: | @@ -648,19 +670,13 @@ jobs: run: | python3 backend/scripts/validate-backend-runtime-env.py --env ${{ vars.ENV }} --check-workflows --check-live-cloud-run - # The manual development deployment runs this no-traffic probe before - # promotion, making it the required auth and route dry run for production. - # - # Development-only by construction. The probe reaches the candidate over a - # Cloud Run tag URL, which is a run.app URL. Production deliberately serves - # no run.app URL (`run.googleapis.com/default-url-disabled=true`, ingress - # `internal-and-cloud-load-balancing`), so a production tag resolves to no - # HTTPS endpoint and this runner sits outside the VPC that could reach one. - # Do not enable these steps for production without first giving the probe a - # route that does not depend on a public revision URL. + # Development probes the no-traffic tagged revision directly from the + # runner. Production cloud-run-only resolves the same immutable tag but + # proves it from an ephemeral VPC-connected Cloud Run Job with both Cloud + # Run IAM and Firebase authentication before any traffic movement. - name: Resolve transcription candidate URL id: transcription-candidate - if: ${{ github.event.inputs.environment == 'development' }} + if: ${{ github.event.inputs.environment == 'development' || github.event.inputs.deploy_targets == 'cloud-run-only' }} run: | candidate_url="$(python3 backend/scripts/resolve_cloud_run_tagged_url.py \ --project=${{ vars.GCP_PROJECT_ID }} \ @@ -668,7 +684,14 @@ jobs: --service=${{ env.SERVICE }} \ --revision=${{ steps.capture-backend-revision.outputs.revision }} \ --tag="$TRANSCRIPTION_CANDIDATE_TAG")" + identity_audience="$(gcloud run services describe "${{ env.SERVICE }}" \ + --project="${{ vars.GCP_PROJECT_ID }}" --region="${{ env.REGION }}" --format='value(status.url)')" + [[ -n "$identity_audience" ]] || { echo 'ERROR: canonical Cloud Run IAM audience is empty' >&2; exit 1; } + [[ "$identity_audience" != "$candidate_url" ]] || { + echo 'ERROR: canonical Cloud Run IAM audience must differ from tagged candidate URL' >&2; exit 1; + } printf 'url=%s\n' "$candidate_url" >> "$GITHUB_OUTPUT" + printf 'identity_audience=%s\n' "$identity_audience" >> "$GITHUB_OUTPUT" - name: Gate backend candidate on known audio if: ${{ github.event.inputs.environment == 'development' }} @@ -678,9 +701,37 @@ jobs: project_id: ${{ vars.GCP_PROJECT_ID }} firebase_auth_project_id: based-hardware + - name: Mint Firebase token for internal production candidate probe + if: ${{ github.event.inputs.environment == 'prod' && github.event.inputs.deploy_targets == 'cloud-run-only' }} + env: + PROBE_SECRET_PROJECT: ${{ vars.GCP_PROJECT_ID }} + FIREBASE_AUTH_PROJECT_ID: based-hardware + run: | + set -euo pipefail + umask 077 + python3 backend/scripts/firebase_release_probe_token.py \ + --secret-project "$PROBE_SECRET_PROJECT" \ + --firebase-project "$FIREBASE_AUTH_PROJECT_ID" \ + --token-output "$RUNNER_TEMP/firebase-production-candidate-token" + + - name: Gate internal production candidate on known audio from Cloud Run VPC + if: ${{ github.event.inputs.environment == 'prod' && github.event.inputs.deploy_targets == 'cloud-run-only' }} + run: | + set -euo pipefail + bash backend/scripts/probe-transcription-candidate-from-cloud-run.sh \ + --project "${{ vars.GCP_PROJECT_ID }}" \ + --region "${{ env.REGION }}" \ + --image "gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${{ steps.image-tag.outputs.short_sha }}" \ + --candidate-url "${{ steps.transcription-candidate.outputs.url }}" \ + --identity-audience "${{ steps.transcription-candidate.outputs.identity_audience }}" \ + --network "${{ vars.CLOUD_RUN_VPC_NETWORK }}" \ + --subnet "${{ vars.CLOUD_RUN_VPC_SUBNET }}" \ + --firebase-token-file "$RUNNER_TEMP/firebase-production-candidate-token" \ + --name-suffix "${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" + # This is the Cloud Run-only acceptance boundary. GKE config, secrets, # and backend-listen stay untouched until every no-traffic revision is - # proven ready (and development has passed its known-audio candidate gate). + # proven ready and the environment-specific known-audio gate has passed. - name: Accept no-traffic Cloud Run candidate run: | python3 backend/scripts/verify_backend_release_vector.py \ @@ -696,7 +747,7 @@ jobs: --evidence-path artifacts/backend-cloud-run-candidate-release-vector.json - name: Remove passed transcription candidate tag - if: ${{ success() && github.event.inputs.environment == 'development' }} + if: ${{ success() && (github.event.inputs.environment == 'development' || github.event.inputs.deploy_targets == 'cloud-run-only') }} run: | gcloud run services update-traffic ${{ env.SERVICE }} \ --project=${{ vars.GCP_PROJECT_ID }} \ @@ -706,9 +757,8 @@ jobs: - name: Remove failed transcription candidate tag # Tag is created at deploy-backend; clean it up on any later failure, - # not only after the candidate URL resolution step succeeds. Production - # never creates the tag, so there is nothing to remove there. - if: ${{ failure() && steps.deploy-backend.outcome == 'success' && github.event.inputs.environment == 'development' }} + # not only after the candidate URL resolution step succeeds. + if: ${{ failure() && steps.deploy-backend.outcome == 'success' && (github.event.inputs.environment == 'development' || github.event.inputs.deploy_targets == 'cloud-run-only') }} run: | gcloud run services update-traffic ${{ env.SERVICE }} \ --project=${{ vars.GCP_PROJECT_ID }} \ @@ -835,8 +885,11 @@ jobs: # gate prevents a partial apply from being reported as a healthy release. - name: Verify serving backend release vector id: verify-serving-release-vector - if: ${{ github.event.inputs.deploy_targets == 'all' }} run: | + CLOUD_RUN_ONLY="" + if [[ "${{ github.event.inputs.deploy_targets }}" == "cloud-run-only" ]]; then + CLOUD_RUN_ONLY="--cloud-run-only" + fi python3 backend/scripts/verify_backend_release_vector.py \ --commit-sha "${{ needs.firestore_readiness.outputs.admitted_sha }}" \ --short-sha "${{ steps.image-tag.outputs.short_sha }}" \ @@ -845,7 +898,8 @@ jobs: --project "${{ vars.GCP_PROJECT_ID }}" \ --region "${{ env.REGION }}" \ --environment "${{ vars.ENV }}" \ - --evidence-path artifacts/backend-serving-release-vector.json + --evidence-path artifacts/backend-serving-release-vector.json \ + $CLOUD_RUN_ONLY - name: Restore Cloud Run traffic snapshot after failed promotion if: ${{ failure() && steps.cloud-run-traffic-snapshot.outcome == 'success' && (steps.shift-cloud-run-traffic.outcome == 'failure' || steps.verify-serving-release-vector.outcome == 'failure') }} diff --git a/.github/workflows/gcp_backend_agent_proxy.yml b/.github/workflows/gcp_backend_agent_proxy.yml index 2f1cf87e7e1..65fba2eaa42 100644 --- a/.github/workflows/gcp_backend_agent_proxy.yml +++ b/.github/workflows/gcp_backend_agent_proxy.yml @@ -36,18 +36,30 @@ jobs: echo "Invalid environment: ${{ github.event.inputs.environment }}. Must be 'development' or 'prod'." exit 1 fi - if [[ "${{ github.event.inputs.environment }}" == "prod" ]]; then - echo "Production artifact deploys are release-record-only; use deploy-release-ring.yml with ring=prod." >&2 - exit 1 - fi - - name: Delete huge unnecessary tools folder run: rm -rf /opt/hostedtoolcache - name: Checkout uses: actions/checkout@v7 with: - ref: ${{ github.event.inputs.branch }} + ref: ${{ github.event.inputs.environment == 'prod' && 'main' || github.event.inputs.branch }} + fetch-depth: 0 + + - name: Admit checked-out production source and image identity + run: | + set -euo pipefail + CHECKED_OUT_SHA=$(git rev-parse HEAD) + if [[ "${{ github.event.inputs.environment }}" == prod ]]; then + git fetch --no-tags origin +refs/heads/main:refs/remotes/origin/main + if ! git merge-base --is-ancestor "$CHECKED_OUT_SHA" origin/main; then + echo "ERROR: checked-out HEAD $CHECKED_OUT_SHA is not an ancestor of fresh origin/main $(git rev-parse origin/main)." >&2 + exit 1 + fi + fi + IMAGE_TAG=$(git rev-parse --short=7 "$CHECKED_OUT_SHA") + echo "Admitted deployment source: $CHECKED_OUT_SHA" + echo "CHECKED_OUT_SHA=$CHECKED_OUT_SHA" >> "$GITHUB_ENV" + echo "IMAGE_TAG=$IMAGE_TAG" >> "$GITHUB_ENV" - name: Google Auth id: auth @@ -66,11 +78,11 @@ jobs: - name: Build and Push Docker image run: | - docker build -t gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7} -f backend/agent-proxy/Dockerfile . + docker build -t gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG} -f backend/agent-proxy/Dockerfile . python3 backend/scripts/runtime_image_contracts.py smoke \ --dockerfile backend/agent-proxy/Dockerfile \ - --image gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7} - docker push gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7} + --image gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG} + docker push gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG} - name: Get GKE credentials uses: google-github-actions/get-gke-credentials@v3 @@ -81,7 +93,7 @@ jobs: - name: Deploy Agent Proxy to GKE cluster using Helm run: | - helm -n ${{ vars.ENV }}-omi-backend upgrade --install ${{ vars.ENV }}-omi-agent-proxy ./backend/charts/agent-proxy -f ./backend/charts/agent-proxy/${{ vars.ENV }}_omi_agent_proxy_values.yaml --set-string "image.tag=${GITHUB_SHA::7}" + helm -n ${{ vars.ENV }}-omi-backend upgrade --install ${{ vars.ENV }}-omi-agent-proxy ./backend/charts/agent-proxy -f ./backend/charts/agent-proxy/${{ vars.ENV }}_omi_agent_proxy_values.yaml --set-string "image.tag=${IMAGE_TAG}" if ! kubectl -n ${{ vars.ENV }}-omi-backend rollout status deploy/${{ vars.ENV }}-omi-agent-proxy --timeout=300s; then python3 backend/scripts/deploy_status_report.py --env ${{ vars.ENV }} --include-gke --gke-service agent-proxy || true exit 1 diff --git a/.github/workflows/gcp_backend_listen_helm.yml b/.github/workflows/gcp_backend_listen_helm.yml index c911d1c134c..57d17b7acd6 100644 --- a/.github/workflows/gcp_backend_listen_helm.yml +++ b/.github/workflows/gcp_backend_listen_helm.yml @@ -46,15 +46,27 @@ jobs: echo "Invalid environment: ${{ github.event.inputs.environment }}. Must be 'development' or 'prod'." exit 1 fi - if [[ "${{ github.event.inputs.environment }}" == "prod" ]]; then - echo "Production artifact deploys are release-record-only; use deploy-release-ring.yml with ring=prod." >&2 - exit 1 - fi - - name: Checkout uses: actions/checkout@v7 with: - ref: ${{ github.event.inputs.branch || github.ref }} + ref: ${{ github.event.inputs.environment == 'prod' && 'main' || github.event.inputs.branch }} + fetch-depth: 0 + + - name: Admit checked-out production source and image identity + run: | + set -euo pipefail + CHECKED_OUT_SHA=$(git rev-parse HEAD) + if [[ "${{ github.event.inputs.environment }}" == prod ]]; then + git fetch --no-tags origin +refs/heads/main:refs/remotes/origin/main + if ! git merge-base --is-ancestor "$CHECKED_OUT_SHA" origin/main; then + echo "ERROR: checked-out HEAD $CHECKED_OUT_SHA is not an ancestor of fresh origin/main $(git rev-parse origin/main)." >&2 + exit 1 + fi + fi + IMAGE_TAG=$(git rev-parse --short=7 "$CHECKED_OUT_SHA") + echo "Admitted deployment source: $CHECKED_OUT_SHA" + echo "CHECKED_OUT_SHA=$CHECKED_OUT_SHA" >> "$GITHUB_ENV" + echo "IMAGE_TAG=$IMAGE_TAG" >> "$GITHUB_ENV" - name: Install Helm uses: azure/setup-helm@v5 diff --git a/.github/workflows/gcp_backend_pusher.yml b/.github/workflows/gcp_backend_pusher.yml index 97f869b7c06..afdf25f4f68 100644 --- a/.github/workflows/gcp_backend_pusher.yml +++ b/.github/workflows/gcp_backend_pusher.yml @@ -49,10 +49,6 @@ jobs: echo "Invalid environment: ${{ github.event.inputs.environment }}. Must be 'development' or 'prod'." exit 1 fi - if [[ "${{ github.event.inputs.environment }}" == "prod" ]]; then - echo "Production artifact deploys are release-record-only; use deploy-release-ring.yml with ring=prod." >&2 - exit 1 - fi if [[ "${{ env.SERVICE }}" != "pusher" && "${{ env.SERVICE }}" != "llm-gateway" ]]; then echo "Invalid service: ${{ env.SERVICE }}. Must be 'pusher' or 'llm-gateway'." exit 1 @@ -69,7 +65,24 @@ jobs: - name: Checkout uses: actions/checkout@v7 with: - ref: ${{ github.event.inputs.branch }} + ref: ${{ github.event.inputs.environment == 'prod' && 'main' || github.event.inputs.branch }} + fetch-depth: 0 + + - name: Admit checked-out production source and image identity + run: | + set -euo pipefail + CHECKED_OUT_SHA=$(git rev-parse HEAD) + if [[ "${{ github.event.inputs.environment }}" == prod ]]; then + git fetch --no-tags origin +refs/heads/main:refs/remotes/origin/main + if ! git merge-base --is-ancestor "$CHECKED_OUT_SHA" origin/main; then + echo "ERROR: checked-out HEAD $CHECKED_OUT_SHA is not an ancestor of fresh origin/main $(git rev-parse origin/main)." >&2 + exit 1 + fi + fi + IMAGE_TAG=$(git rev-parse --short=7 "$CHECKED_OUT_SHA") + echo "Admitted deployment source: $CHECKED_OUT_SHA" + echo "CHECKED_OUT_SHA=$CHECKED_OUT_SHA" >> "$GITHUB_ENV" + echo "IMAGE_TAG=$IMAGE_TAG" >> "$GITHUB_ENV" - name: Google Auth id: auth @@ -121,17 +134,17 @@ jobs: - name: Build and Push Docker image run: | if [[ "${{ env.SERVICE }}" == "llm-gateway" ]]; then - docker build -t gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7} -f backend/Dockerfile . + docker build -t gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG} -f backend/Dockerfile . python3 backend/scripts/runtime_image_contracts.py smoke \ --dockerfile backend/Dockerfile \ - --image gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7} + --image gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG} else - docker build -t gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7} -f backend/pusher/Dockerfile . + docker build -t gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG} -f backend/pusher/Dockerfile . python3 backend/scripts/runtime_image_contracts.py smoke \ --dockerfile backend/pusher/Dockerfile \ - --image gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7} + --image gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG} fi - docker push gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7} + docker push gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG} - name: Get GKE credentials uses: google-github-actions/get-gke-credentials@v3 @@ -218,7 +231,7 @@ jobs: python3 backend/scripts/validate-llm-gateway-env.py \ backend/charts/backend-listen/${{ vars.ENV }}_omi_backend_listen_values.yaml \ backend/charts/llm-gateway/${{ vars.ENV }}_omi_llm_gateway_values.yaml - if ! IMAGE_TAG="${GITHUB_SHA::7}" backend/scripts/deploy-llm-gateway.sh; then + if ! IMAGE_TAG="${IMAGE_TAG}" backend/scripts/deploy-llm-gateway.sh; then python3 backend/scripts/deploy_status_report.py --env ${{ vars.ENV }} --include-gke --gke-service llm-gateway || true echo "=== llm-gateway deployment ===" kubectl -n ${{ vars.ENV }}-omi-backend describe deploy/${{ vars.ENV }}-omi-llm-gateway || true @@ -238,7 +251,7 @@ jobs: fi python3 backend/scripts/deploy_status_report.py --env ${{ vars.ENV }} --include-gke --gke-service llm-gateway else - helm -n ${{ vars.ENV }}-omi-backend upgrade --install ${{ vars.ENV }}-omi-pusher ./backend/charts/pusher -f ./backend/charts/pusher/${{ vars.ENV }}_omi_pusher_values.yaml --set-string "image.tag=${GITHUB_SHA::7}" + helm -n ${{ vars.ENV }}-omi-backend upgrade --install ${{ vars.ENV }}-omi-pusher ./backend/charts/pusher -f ./backend/charts/pusher/${{ vars.ENV }}_omi_pusher_values.yaml --set-string "image.tag=${IMAGE_TAG}" # Must cover the chart-derived Pusher rollout budget (fourteen # availability waves at the current production HPA ceiling). if ! kubectl -n ${{ vars.ENV }}-omi-backend rollout status deploy/${{ vars.ENV }}-omi-pusher --timeout=9600s; then @@ -264,7 +277,7 @@ jobs: bash backend/scripts/probe-llm-gateway-from-cloud-run.sh \ --project "${{ vars.GCP_PROJECT_ID }}" \ --region "${{ env.REGION }}" \ - --image "gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7}" \ + --image "gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG}" \ --gateway-url "${{ steps.gateway-serving.outputs.gateway_url }}" \ --network "${{ vars.CLOUD_RUN_VPC_NETWORK }}" \ --subnet "${{ vars.CLOUD_RUN_VPC_SUBNET }}" \ @@ -289,7 +302,7 @@ jobs: restartPolicy: Never containers: - name: smoke - image: gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7} + image: gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG} command: ["sh", "-c", "python scripts/smoke-llm-gateway.py --url \"\$SMOKE_URL\" --token \"\$SMOKE_TOKEN\" --check-metrics --metrics-token \"\$METRICS_SECRET\""] env: - name: SMOKE_URL diff --git a/.github/workflows/gcp_llm_gateway.yml b/.github/workflows/gcp_llm_gateway.yml index 59b5b929fe3..d1da8217d86 100644 --- a/.github/workflows/gcp_llm_gateway.yml +++ b/.github/workflows/gcp_llm_gateway.yml @@ -37,18 +37,30 @@ jobs: echo "Invalid environment: ${{ github.event.inputs.environment }}. Must be 'development' or 'prod'." exit 1 fi - if [[ "${{ github.event.inputs.environment }}" == "prod" ]]; then - echo "Production artifact deploys are release-record-only; use deploy-release-ring.yml with ring=prod." >&2 - exit 1 - fi - - name: Delete huge unnecessary tools folder run: rm -rf /opt/hostedtoolcache - name: Checkout uses: actions/checkout@v7 with: - ref: ${{ github.event.inputs.branch }} + ref: ${{ github.event.inputs.environment == 'prod' && 'main' || github.event.inputs.branch }} + fetch-depth: 0 + + - name: Admit checked-out production source and image identity + run: | + set -euo pipefail + CHECKED_OUT_SHA=$(git rev-parse HEAD) + if [[ "${{ github.event.inputs.environment }}" == prod ]]; then + git fetch --no-tags origin +refs/heads/main:refs/remotes/origin/main + if ! git merge-base --is-ancestor "$CHECKED_OUT_SHA" origin/main; then + echo "ERROR: checked-out HEAD $CHECKED_OUT_SHA is not an ancestor of fresh origin/main $(git rev-parse origin/main)." >&2 + exit 1 + fi + fi + IMAGE_TAG=$(git rev-parse --short=7 "$CHECKED_OUT_SHA") + echo "Admitted deployment source: $CHECKED_OUT_SHA" + echo "CHECKED_OUT_SHA=$CHECKED_OUT_SHA" >> "$GITHUB_ENV" + echo "IMAGE_TAG=$IMAGE_TAG" >> "$GITHUB_ENV" - name: Google Auth id: auth @@ -87,11 +99,11 @@ jobs: - name: Build and Push Docker image run: | - docker build -t gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7} -f backend/Dockerfile . + docker build -t gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG} -f backend/Dockerfile . python3 backend/scripts/runtime_image_contracts.py smoke \ --dockerfile backend/Dockerfile \ - --image gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7} - docker push gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7} + --image gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG} + docker push gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG} - name: Get GKE credentials uses: google-github-actions/get-gke-credentials@v3 @@ -109,7 +121,7 @@ jobs: GKE_CLUSTER: ${{ vars.GKE_CLUSTER }} REGION: ${{ env.REGION }} run: | - IMAGE_TAG="${GITHUB_SHA::7}" backend/scripts/deploy-llm-gateway.sh + IMAGE_TAG="${IMAGE_TAG}" backend/scripts/deploy-llm-gateway.sh - name: Verify LLM Gateway serving data plane id: gateway-serving @@ -125,7 +137,7 @@ jobs: bash backend/scripts/probe-llm-gateway-from-cloud-run.sh \ --project "${{ vars.GCP_PROJECT_ID }}" \ --region "${{ env.REGION }}" \ - --image "gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7}" \ + --image "gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG}" \ --gateway-url "${{ steps.gateway-serving.outputs.gateway_url }}" \ --network "${{ vars.CLOUD_RUN_VPC_NETWORK }}" \ --subnet "${{ vars.CLOUD_RUN_VPC_SUBNET }}" \ @@ -139,7 +151,7 @@ jobs: METRICS_TOKEN="$(kubectl -n "$NAMESPACE" get secret ${{ vars.ENV }}-omi-backend-secrets -o jsonpath='{.data.METRICS_SECRET}' | base64 -d)" kubectl -n "$NAMESPACE" run "llm-gateway-smoke-${GITHUB_RUN_ID}" \ --rm -i --restart=Never \ - --image "gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${GITHUB_SHA::7}" \ + --image "gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${IMAGE_TAG}" \ --env "SMOKE_URL=http://${{ vars.ENV }}-omi-llm-gateway.${{ vars.ENV }}-omi-backend.svc.cluster.local:8080" \ --env "SMOKE_TOKEN=$TOKEN" \ --env "METRICS_SECRET=$METRICS_TOKEN" \ diff --git a/.github/workflows/release-record.yml b/.github/workflows/release-record.yml deleted file mode 100644 index 223d55a8b28..00000000000 --- a/.github/workflows/release-record.yml +++ /dev/null @@ -1,217 +0,0 @@ -name: Build release record - -on: - workflow_run: - workflows: ["Release Eligibility"] - types: [completed] - -# A record writer may publish artifacts but never deploys a serving ring. The -# selected source is admitted below before checkout; workflow_run's default -# branch context is never treated as the release source. -permissions: - actions: write - contents: read - id-token: write - -concurrency: - group: build-release-record-${{ github.event.workflow_run.head_sha }} - cancel-in-progress: false - -jobs: - build-record: - if: >- - github.event.workflow_run.conclusion == 'success' && - github.event.workflow_run.event == 'push' && - github.event.workflow_run.run_attempt == 1 && - github.event.workflow_run.head_branch == 'main' && - github.event.workflow_run.head_repository.full_name == github.repository - environment: beta - runs-on: ubuntu-latest - steps: - - name: Checkout exact eligible main source - uses: actions/checkout@v7 - with: - ref: ${{ github.event.workflow_run.head_sha }} - fetch-depth: 0 - - - name: Refuse superseded or non-main source - env: - RELEASE_SHA: ${{ github.event.workflow_run.head_sha }} - run: | - set -euo pipefail - git fetch --no-tags origin +refs/heads/main:refs/remotes/origin/main - test "$RELEASE_SHA" = "$(git rev-parse origin/main)" - git merge-base --is-ancestor "$RELEASE_SHA" origin/main - - - name: Authenticate as release-record writer - uses: google-github-actions/auth@v3 - with: - workload_identity_provider: ${{ vars.RELEASE_RINGS_WIF_PROVIDER }} - service_account: ${{ vars.RELEASE_RECORDS_WRITER_SERVICE_ACCOUNT }} - - - name: Set up build tooling - uses: google-github-actions/setup-gcloud@v3 - - - name: Read the serving public runtime configuration once - uses: google-github-actions/get-gke-credentials@v3 - with: - cluster_name: ${{ vars.RELEASE_GKE_CLUSTER }} - location: us-central1 - project_id: ${{ vars.RELEASE_RUNTIME_PROJECT_ID }} - - - name: Capture deployable public ConfigMap and release topology - env: - CLOUD_RUN_VPC_NETWORK: ${{ vars.CLOUD_RUN_VPC_NETWORK }} - CLOUD_RUN_VPC_SUBNET: ${{ vars.CLOUD_RUN_VPC_SUBNET }} - run: | - set -euo pipefail - test -n "$CLOUD_RUN_VPC_NETWORK" - test -n "$CLOUD_RUN_VPC_SUBNET" - python3 -m pip install -q pyyaml - mkdir -p artifacts/release-record - for ring in beta prod; do - kubectl -n "${ring}-omi-backend" get configmap "${ring}-omi-backend-config" -o json \ - | jq '{apiVersion, kind, metadata: {name: .metadata.name}, data}' \ - > "artifacts/release-record/${ring}-backend-config.yaml" - jq --arg network "$CLOUD_RUN_VPC_NETWORK" --arg subnet "$CLOUD_RUN_VPC_SUBNET" \ - '.data + {CLOUD_RUN_VPC_NETWORK: $network, CLOUD_RUN_VPC_SUBNET: $subnet}' \ - "artifacts/release-record/${ring}-backend-config.yaml" \ - > "artifacts/release-record/${ring}-public-values.json" - done - python3 - <<'PY' - import json - import yaml - - topology = yaml.safe_load(open("backend/deploy/release_rings.yaml", encoding="utf-8"))["rings"] - json.dump(topology, open("artifacts/release-record/topology.json", "w", encoding="utf-8"), sort_keys=True) - PY - - - name: Prepare Docker registry - run: gcloud auth configure-docker gcr.io --quiet - - - name: Build and publish backend image once - id: backend - uses: docker/build-push-action@v7 - with: - context: . - file: backend/Dockerfile - push: true - tags: gcr.io/${{ vars.RELEASE_ARTIFACT_PROJECT_ID }}/backend:${{ github.event.workflow_run.head_sha }} - - - name: Build and publish pusher image once - id: pusher - uses: docker/build-push-action@v7 - with: - context: . - file: backend/pusher/Dockerfile - push: true - tags: gcr.io/${{ vars.RELEASE_ARTIFACT_PROJECT_ID }}/pusher:${{ github.event.workflow_run.head_sha }} - - - name: Build and publish LLM gateway image once - id: llm_gateway - uses: docker/build-push-action@v7 - with: - context: . - file: backend/Dockerfile - push: true - tags: gcr.io/${{ vars.RELEASE_ARTIFACT_PROJECT_ID }}/llm-gateway:${{ github.event.workflow_run.head_sha }} - - - name: Build and publish agent-proxy image once - id: agent_proxy - uses: docker/build-push-action@v7 - with: - context: . - file: backend/agent-proxy/Dockerfile - push: true - tags: gcr.io/${{ vars.RELEASE_ARTIFACT_PROJECT_ID }}/agent-proxy:${{ github.event.workflow_run.head_sha }} - - - name: Resolve release secret versions - run: | - python3 -m pip install -q pyyaml - mkdir -p artifacts/release-record - python3 backend/scripts/release_rings.py resolve-secrets \ - --manifest backend/deploy/runtime_env.yaml \ - --manifest backend/charts/backend-secrets/prod_omi_backend_secrets_values.yaml \ - --project "${{ vars.RELEASE_RUNTIME_PROJECT_ID }}" \ - --output artifacts/release-record/secret-versions.json - - - name: Render fully deployable, pinned ring configuration - run: | - set -euo pipefail - for ring in beta prod; do - python3 backend/scripts/release_rings.py materialize-runtime \ - --manifest backend/deploy/runtime_env.yaml \ - --secret-versions-file artifacts/release-record/secret-versions.json \ - --public-values-file "artifacts/release-record/${ring}-public-values.json" \ - --output "artifacts/release-record/runtime-env-${ring}-pinned.yaml" - done - python3 backend/scripts/release_rings.py materialize-secrets \ - --manifest backend/charts/backend-secrets/prod_omi_backend_secrets_values.yaml \ - --secret-versions-file artifacts/release-record/secret-versions.json \ - --output artifacts/release-record/backend-secrets-pinned.yaml - for ring in beta prod; do - python3 backend/scripts/render_release_ring_config.py --ring "$ring" --source "artifacts/release-record/runtime-env-${ring}-pinned.yaml" --output "artifacts/release-record/${ring}-backend.yaml" - python3 backend/scripts/render_release_ring_config.py --ring "$ring" --source artifacts/release-record/backend-secrets-pinned.yaml --output "artifacts/release-record/${ring}-backend-secrets.yaml" - python3 backend/scripts/render_release_ring_config.py --ring "$ring" --source backend/charts/backend-listen/prod_omi_backend_listen_values.yaml --output "artifacts/release-record/${ring}-backend-listen.yaml" - python3 backend/scripts/render_release_ring_config.py --ring "$ring" --source backend/charts/pusher/prod_omi_pusher_values.yaml --output "artifacts/release-record/${ring}-pusher.yaml" - python3 backend/scripts/render_release_ring_config.py --ring "$ring" --source backend/charts/llm-gateway/prod_omi_llm_gateway_values.yaml --output "artifacts/release-record/${ring}-llm-gateway.yaml" - python3 backend/scripts/render_release_ring_config.py --ring "$ring" --source backend/charts/agent-proxy/prod_omi_agent_proxy_values.yaml --output "artifacts/release-record/${ring}-agent-proxy.yaml" - done - - - name: Upload immutable configuration objects and create record - id: record - env: - RELEASE_BUCKET: ${{ vars.RELEASE_RECORDS_BUCKET }} - RELEASE_SHA: ${{ github.event.workflow_run.head_sha }} - ELIGIBILITY_RUN_ID: ${{ github.event.workflow_run.id }} - BACKEND_DIGEST: ${{ steps.backend.outputs.digest }} - PUSHER_DIGEST: ${{ steps.pusher.outputs.digest }} - LLM_GATEWAY_DIGEST: ${{ steps.llm_gateway.outputs.digest }} - AGENT_PROXY_DIGEST: ${{ steps.agent_proxy.outputs.digest }} - ARTIFACT_PROJECT: ${{ vars.RELEASE_ARTIFACT_PROJECT_ID }} - run: | - set -euo pipefail - test -n "$RELEASE_BUCKET" - release_id="$(date -u +%Y%m%d).${GITHUB_RUN_ID}" - config_args=() - for ring in beta prod; do - for component in backend backend-config backend-secrets backend-listen pusher llm-gateway agent-proxy; do - source="artifacts/release-record/${ring}-${component}.yaml" - object="gs://${RELEASE_BUCKET}/config/${release_id}/${ring}/${component}.yaml" - gcloud storage cp --if-generation-match=0 "$source" "$object" - digest="$(sha256sum "$source" | awk '{print $1}')" - config_args+=(--rendered-config "${ring}/${component}=${object}#sha256:${digest}") - done - done - python3 backend/scripts/release_rings.py create-record \ - --release-id "$release_id" \ - --git-sha "$RELEASE_SHA" \ - --eligibility-run-id "$ELIGIBILITY_RUN_ID" \ - --image "backend=gcr.io/${ARTIFACT_PROJECT}/backend@${BACKEND_DIGEST}" \ - --image "backend-listen=gcr.io/${ARTIFACT_PROJECT}/backend@${BACKEND_DIGEST}" \ - --image "pusher=gcr.io/${ARTIFACT_PROJECT}/pusher@${PUSHER_DIGEST}" \ - --image "llm-gateway=gcr.io/${ARTIFACT_PROJECT}/llm-gateway@${LLM_GATEWAY_DIGEST}" \ - --image "agent-proxy=gcr.io/${ARTIFACT_PROJECT}/agent-proxy@${AGENT_PROXY_DIGEST}" \ - "${config_args[@]}" \ - --secret-versions-file artifacts/release-record/secret-versions.json \ - --topology-file artifacts/release-record/topology.json \ - --output artifacts/release-record/record.json - python3 backend/scripts/release_rings.py validate-record --record artifacts/release-record/record.json - gcloud storage cp --if-generation-match=0 artifacts/release-record/record.json "gs://${RELEASE_BUCKET}/records/${release_id}.json" - printf 'release_id=%s\n' "$release_id" >> "$GITHUB_OUTPUT" - - - name: Persist record build evidence - uses: actions/upload-artifact@v7 - with: - name: release-record-${{ steps.record.outputs.release_id }} - path: artifacts/release-record - if-no-files-found: error - retention-days: 30 - - - name: Auto-deploy the newest record to beta - env: - GH_TOKEN: ${{ github.token }} - run: | - gh workflow run deploy-release-ring.yml --ref main \ - -f ring=beta \ - -f release_id="${{ steps.record.outputs.release_id }}" diff --git a/.github/workflows/repo-checks.yml b/.github/workflows/repo-checks.yml index e57629ff503..aaf0679a907 100644 --- a/.github/workflows/repo-checks.yml +++ b/.github/workflows/repo-checks.yml @@ -143,15 +143,9 @@ jobs: - name: Check deployment concurrency policy run: python3 .github/scripts/check-deployment-concurrency.py - - name: Check desktop release terminology - run: python3 .github/scripts/check-desktop-release-terminology.py - - name: Test desktop qualification metadata writer run: python3 desktop/macos/scripts/release-keyvalue.py self-test - - name: Test desktop release promotion gate - run: python3 .github/scripts/test_check_desktop_release_promotion.py - - name: Test automatic desktop beta candidate gate run: python3 .github/scripts/test_check_desktop_auto_beta_candidate.py diff --git a/.pr-body.release-evidence-10082.md b/.pr-body.release-evidence-10082.md deleted file mode 100644 index 7913d817a35..00000000000 --- a/.pr-body.release-evidence-10082.md +++ /dev/null @@ -1,104 +0,0 @@ -## Summary - -Closes #10082 - -Adds a break-glass path for promoting a signed macOS candidate to beta when the qualification gate itself is broken — the case we actually hit. - -**Scope changed after review**, twice: - -1. The first pass built this as a parallel subsystem with an approval ceremony. It is now a relaxed precondition on the existing promotion path. -2. An audit of every deploy surface found the same footgun on two more. Those now have hatches too. - -## Every gated surface now has a hatch - -The audit found the posture was bimodal and arbitrary: one surface welded shut with no hatch, twenty with no gate at all, and which was which bore no relation to blast radius. - -| Surface | Before | Now | -|---|---|---| -| Desktop candidate (`desktop_auto_release`) | Swift source gate; `force_release` explicitly did **not** bypass it | `release_mode=break_glass` | -| Desktop beta | fail-closed qualification, no hatch | `desktop_emergency_promote_beta` | -| Desktop stable | already had `break_glass_confirm` | unchanged | -| **Backend prod** (`gcp_backend`) | required a *successful* eligibility run on the exact SHA, **zero bypass inputs** | `skip_eligibility_proof` + `break_glass_confirm` + reason | - -Two of these could block a hotfix with no recovery at all: - -- **Backend prod**: a flaky or red check *permanently poisons that SHA* — rerunning doesn't make the original run succeed, so the only recovery was landing another commit on `main` and waiting for a full green run. That's a bigger blast radius than the beta gate that cost a day. -- **Desktop candidate**: `plan-desktop-release.py` checked the source gate *before* the `force_release` branch, so a flaky `Desktop Swift Build & Tests` blocked candidate creation outright, and everything downstream with it. - -**Hatches relax evidence requirements only.** The backend hatch moves the ancestry-of-`main` check *outside* the break-glass branch so it is always enforced — unreviewed code still cannot reach production. Each hatch opens a tracking issue; repeated use means the gate is the defect. - -`AGENTS.md` now carries one table of every hatch. Not knowing a hatch exists is the same outage as not having one. - -Untouched: `gcp_backend_auto_dev` is `workflow_run`-only and manual `gcp_backend` is already its fallback. The 20 ungated `gcp_*` services need gates decided first — separately, not here. - -## The design - -Emergency promotion is ordinary promotion with the T2 qualification gate relaxed. It is not a separate mutation path. - -- `_build_channel_pointer`, `_build_beta_rollback_pointer`, and `_build_emergency_macos_beta_pointer` collapse into one `_build_pointer` driven by a `TRANSITIONS` policy table. It is the single pointer-mutation authority; the three transitions differ only in which preconditions they declare. -- The three emergency endpoints become `emergency: true` on the existing `POST /v2/desktop/channels/promote`. Beta-only, and requires a compare-and-swap on both current release id and generation, so it cannot silently clobber a concurrent normal promotion. -- The resulting pointer carries `emergency: true`. The audit record lives on the pointer rather than in a parallel collection that can diverge from it. -- Normal promotion is unchanged and still fail-closed. - -## Why the ceremony was removed - -Removed: two-approver comment parsing, the four-hour expiry window, incident binding, behavioral-evidence fetch, responder notification, the one-hour monitor, the operation-id reconciliation protocol, and the recovery job that existed to repair state the subsystem itself created. - -This codebase is mostly agent-driven, and **ceremony is not a control against an automated actor.** An agent fills in a rationale, an incident id, and an approval comment as readily as a human — faster, and correctly. Those fields imposed real cost on a small team mid-incident while filtering nothing. Eight of this branch's original commits were fixes to that machinery rather than to release behavior, which is the clearest signal it cost more than the risk it insured. - -What remains are the guards a machine can actually check: tag-to-SHA binding, artifact digests recomputed from the published assets, and the pointer CAS. - -The workflow now reads the live pointer itself rather than requiring an operator to paste release id and generation mid-incident. - -## The escape hatch is a signal, not a solution - -Every run opens a `qualification-gate-failure` issue recording tag, SHA, operator, reason, and run URL. If those accumulate, the qualification gate is the defect and this hatch is masking it. Fix the gate, then close the issue. - -## Also in this PR - -Deletes `release_blessing.py` and `check-python-backend-blessing.py` plus their tests (502 lines). The latter is referenced only by its own test — no workflow, no Makefile, no `checks-manifest.yaml` entry — and `run_checks.py` resolves checks by explicit id without globbing, so nothing could execute it. - -The qualification reliability amendment keeps every break-glass workflow unchanged while making its automatic evidence boundary fail closed: fault qualification waits for the bridge's live signed-in, onboarded, non-restoring owner snapshot (not only the UI's `appState: main`), and release preflight validates the fetched publication timestamp plus signed ZIP, DMG, and signed-smoke assets with SHA-256 digests. - -## Product invariants affected - -- INV-AUTH-1 - -## Verification - -- `cd backend && python3 -m pytest tests/unit/test_desktop_update_channels.py tests/unit/test_desktop_updates.py tests/unit/test_desktop_release_scripts.py` → **133 passed** -- New `TestEmergencyBetaPromotionRules` covers: promotes an unqualified build and marks the pointer `emergency`; normal promotion still rejects that same build; stale release-id and stale generation CAS both rejected; roll-forward only; never reaches stable. -- `python3 .github/scripts/check-release-process-guards.py` → passes -- `python3 .github/scripts/test_desktop_qualification_dispatch.py` → passes -- `python3 .github/scripts/check-desktop-release-terminology.py` → passes -- `python3 backend/scripts/check_route_policy_baseline.py` → 0 baseline additions, 0 new missing entries after removing the three routes -- Workflow validated as YAML (466 → 179 lines, one job, 8 steps) -- `python3 .github/scripts/test_plan_desktop_release.py` → passes, including two new cases: `break_glass` releases over a blocked source check, and does not set the bypass flag when the check passes. The existing assertion that `force_release` still blocks is unchanged. -- `python3 .github/scripts/test_check_backend_deploy_source_admission.py` → 23 pass, including 3 new break-glass contract tests (confirm string + reason required; ancestry enforced outside the branch; bypass recorded as an issue). These read workflow source, so they are **static tripwires, not behavioral coverage** — Actions cannot be driven from a unit test. -- `bash desktop/macos/tests/test-omi-ctl-wait-ready.sh` → rejects signed-out, restoring, owner-not-ready, and stale snapshots; accepts only the live authenticated owner-ready bridge snapshot. -- `python3 desktop/macos/scripts/release-keyvalue.py self-test` → validates passing release JSON and rejects missing/malformed publication timestamps, missing signed-smoke assets, and missing artifact digests. -- `bash desktop/macos/tests/test-fault-suite-launch-env.sh && bash desktop/macos/tests/test-qualify-desktop-beta-contract.sh` → passes; confirms the fault harness invokes the strengthened readiness command and qualification bootstrap contracts remain intact. -- `ENCRYPTION_SECRET=*** backend/.venv/bin/python -m pytest backend/tests/unit/test_desktop_release_scripts.py backend/tests/unit/test_desktop_update_channels.py` → 43 passed; covers qualified qualification-claim retries re-dispatching beta promotion without rerunning the runner. -- `actionlint .github/workflows/desktop_qualify_beta.yml` → passes. -- `make preflight` with this PR body → 36 deterministic checks passed. -- `python3 .github/scripts/check_backend_deploy_source_admission.py` → passes. Note it validates structural wiring, not bypass semantics — hence the added contract tests. -- `check_agents_md_lean.py` → within budget - -## Limitations - -- **No hatch has been fired against real infrastructure.** These change production deploy controls — beta promotion, backend prod deploy, desktop candidate creation. They need a real incident or a deliberate dry run before they are trusted. -- **`gcp_backend.yml` is prod backend deploy.** Per `AGENTS.md` that is a wide-blast-radius surface needing explicit sign-off; this is not auto-mergeable. -- Full backend pytest selection could not run in this worktree (no project-local Python environment; restricted network blocks the documented 3.11 bootstrap). The three affected files were run directly. - -## Failure class - -Failure-Class: FC-split-mutation-authority - -The original defect and this rewrite share a cause: pointer-mutation authority was split across parallel implementations. Consolidating to a single `_build_pointer` removes the class rather than adding another observer of it. - -## Follow-ups (not in this PR) - -Deliberately excluded to keep this reviewable: - -- 22 `gcp_*` deploy workflows, 3,766 lines, zero `workflow_call` — the job scaffold is copy-pasted ~20×. A reusable workflow targets ~1,200 lines and removes the drift risk of hardening one deploy path and missing the others. -- ~1,300 lines implementing the same scan → count → compare-to-JSON-baseline ratchet five separate times, across 8 baseline formats. One runner plus a rule table targets ~300. diff --git a/AGENTS.md b/AGENTS.md index 2da881fb81e..9c8168c47bd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -131,8 +131,6 @@ Click at coordinates: `cliclick c:X,Y`. Mac screenshots: `screencapture -x /tmp/ | Blocked on | Hatch | |---|---| | Desktop candidate won't cut (`Desktop Swift Build & Tests` red/flaky) | `desktop_auto_release.yml` with `release_mode=break_glass` | -| Beta qualification broken | `desktop_emergency_promote_beta.yml` (`confirm=emergency-promote-beta`) | -| Stable promotion blocked | `desktop_promote_prod.yml` with `break_glass_confirm` | | Backend deploy has no Release Eligibility proof | `gcp_backend.yml` with `skip_eligibility_proof=true`, `break_glass_confirm=deploy-without-proof`, `break_glass_reason` | Hatches relax *evidence* requirements only. They never relax that code is merged to `main` first, and never reach stable/prod pointers without their own explicit confirm. diff --git a/app/.env.template b/app/.env.template index f046e363485..3830e2c1127 100644 --- a/app/.env.template +++ b/app/.env.template @@ -5,5 +5,4 @@ GOOGLE_CLIENT_ID= GOOGLE_CLIENT_SECRET= USE_WEB_AUTH=false USE_AUTH_CUSTOM_TOKEN=false -STAGING_API_URL= POSTHOG_API_KEY= diff --git a/app/config/client_env_policy.yaml b/app/config/client_env_policy.yaml index 3a13d33ee69..8c81394e18e 100644 --- a/app/config/client_env_policy.yaml +++ b/app/config/client_env_policy.yaml @@ -2,7 +2,6 @@ "public_client_env": { "allowed": [ "PUBLIC_API_BASE_URL", - "PUBLIC_STAGING_API_URL", "PUBLIC_USE_WEB_AUTH", "PUBLIC_USE_AUTH_CUSTOM_TOKEN", "PUBLIC_GOOGLE_MAPS_API_KEY", @@ -41,7 +40,6 @@ }, "public_client_env_sources": { "allowed": [ - "STAGING_API_URL" ] }, "restricted_public_client_keys": { diff --git a/app/lib/backend/preferences.dart b/app/lib/backend/preferences.dart index b9e8659d988..4f16c40dcb5 100644 --- a/app/lib/backend/preferences.dart +++ b/app/lib/backend/preferences.dart @@ -735,15 +735,6 @@ class SharedPreferencesUtil { bool get companionAssociationPrompted => getBool('companionAssociationPrompted'); - //------------------------ TestFlight API Environment ----------------------// - - /// Legacy display preference only. TestFlight routing is always the beta ring. - String get testFlightApiEnvironment => getString('testFlightApiEnvironment', defaultValue: 'staging'); - - set testFlightApiEnvironment(String value) => saveString('testFlightApiEnvironment', value); - - bool get testFlightUseStagingApi => testFlightApiEnvironment == 'staging'; - //--------------------------- Announcements ---------------------------------// // Last known app version - used to detect app upgrades diff --git a/app/lib/env/dev_env.dart b/app/lib/env/dev_env.dart index b9d6546c802..b40bb3dce55 100644 --- a/app/lib/env/dev_env.dart +++ b/app/lib/env/dev_env.dart @@ -51,8 +51,4 @@ final class DevEnv implements EnvFields { @override @EnviedField(varName: 'USE_AUTH_CUSTOM_TOKEN', obfuscate: false, defaultValue: false) final bool? useAuthCustomToken = _DevEnv.useAuthCustomToken; - - @override - @EnviedField(varName: 'STAGING_API_URL', obfuscate: true) - final String? stagingApiUrl = _DevEnv.stagingApiUrl; } diff --git a/app/lib/env/env.dart b/app/lib/env/env.dart index f99ceebc770..7f18eabcca9 100644 --- a/app/lib/env/env.dart +++ b/app/lib/env/env.dart @@ -1,4 +1,6 @@ abstract class Env { + static const productionApiBaseUrl = 'https://api.omi.me/'; + static const productionAgentProxyWsUrl = 'wss://agent.omi.me/v1/agent/ws'; static late final EnvFields _instance; static String? _apiBaseUrlOverride; static String? _agentProxyWsUrlOverride; @@ -12,6 +14,10 @@ abstract class Env { _apiBaseUrlOverride = url; } + static void clearApiBaseUrlOverrideForTesting() { + _apiBaseUrlOverride = null; + } + static void overrideAgentProxyWsUrl(String url) { _agentProxyWsUrlOverride = url; } @@ -23,37 +29,30 @@ abstract class Env { // static String? get apiBaseUrl => 'https://omi-backend.ngrok.app/'; static String? get apiBaseUrl => _apiBaseUrlOverride ?? _instance.apiBaseUrl; - /// Staging API URL from STAGING_API_URL env var. Null when not configured. - static String? get stagingApiUrl { - final url = _instance.stagingApiUrl; - if (url == null || url.isEmpty) return null; - return url; - } - - /// Whether STAGING_API_URL is configured in the environment. - static bool get isStagingConfigured => stagingApiUrl != null; - - static bool get isUsingStagingApi { - final effective = apiBaseUrl; - final staging = stagingApiUrl; - if (effective == null || staging == null) return false; - return _normalizeUrl(effective) == _normalizeUrl(staging); - } - - static String _normalizeUrl(String url) { - var s = url.trim().toLowerCase(); - while (s.endsWith('/')) { - s = s.substring(0, s.length - 1); + /// Production-family packages have one pinned backend authority. This runs + /// during startup so a misconfigured signing group fails before networking. + static void validateStartupRouting({required bool productionFamily, String? configuredApiBaseUrl}) { + if (!productionFamily) return; + final normalized = (configuredApiBaseUrl ?? apiBaseUrl ?? '').trim().replaceFirst(RegExp(r'/+$'), ''); + if (normalized != productionApiBaseUrl.replaceFirst(RegExp(r'/+$'), '')) { + throw StateError('Production packages require API_BASE_URL=https://api.omi.me/'); + } + if (_agentProxyWsUrlFor(normalized) != productionAgentProxyWsUrl) { + throw StateError('Production packages require the production agent WebSocket endpoint.'); } - return s; } + static void requireProductionRouting() => validateStartupRouting(productionFamily: true); + /// WebSocket URL for the agent proxy service. /// Derives from apiBaseUrl: api.omi.me → agent.omi.me, api.omiapi.com → agent.omiapi.com. /// Can be overridden via Env.overrideAgentProxyWsUrl() for local testing. static String get agentProxyWsUrl { if (_agentProxyWsUrlOverride != null) return _agentProxyWsUrlOverride!; - final base = apiBaseUrl ?? 'https://api.omi.me'; + return _agentProxyWsUrlFor(apiBaseUrl ?? productionApiBaseUrl); + } + + static String _agentProxyWsUrlFor(String base) { final host = Uri.parse(base).host.replaceFirst('api.', 'agent.'); return 'wss://$host/v1/agent/ws'; } @@ -97,6 +96,4 @@ abstract class EnvFields { bool? get useWebAuth; bool? get useAuthCustomToken; - - String? get stagingApiUrl; } diff --git a/app/lib/env/prod_env.dart b/app/lib/env/prod_env.dart index e46dd204c93..4a1a954517c 100644 --- a/app/lib/env/prod_env.dart +++ b/app/lib/env/prod_env.dart @@ -51,8 +51,4 @@ final class ProdEnv implements EnvFields { @override @EnviedField(varName: 'USE_AUTH_CUSTOM_TOKEN', obfuscate: false, defaultValue: false) final bool? useAuthCustomToken = _ProdEnv.useAuthCustomToken; - - @override - @EnviedField(varName: 'STAGING_API_URL', obfuscate: true) - final String? stagingApiUrl = _ProdEnv.stagingApiUrl; } diff --git a/app/lib/main.dart b/app/lib/main.dart index 19ac8f1c1d7..6e0bf04eb58 100644 --- a/app/lib/main.dart +++ b/app/lib/main.dart @@ -33,6 +33,7 @@ import 'package:omi/env/prod_env.dart'; import 'package:omi/firebase_options_dev.dart' as dev; import 'package:omi/firebase_options_prod.dart' as prod; import 'package:omi/flavors.dart'; +import 'package:omi/startup_routing.dart'; import 'package:omi/l10n/app_localizations.dart'; import 'package:omi/pages/apps/providers/add_app_provider.dart'; import 'package:omi/pages/conversation_detail/conversation_detail_provider.dart'; @@ -73,10 +74,9 @@ import 'package:omi/services/services.dart'; import 'package:omi/services/wals.dart'; import 'package:omi/utils/debug_log_manager.dart'; import 'package:omi/utils/debugging/crashlytics_manager.dart'; +import 'package:omi/utils/environment_detector.dart'; import 'package:omi/utils/analytics/rage_click_context_tracker.dart'; import 'package:omi/utils/l10n_extensions.dart'; -import 'package:omi/utils/environment_detector.dart'; -import 'package:omi/pages/settings/developer.dart'; import 'package:omi/utils/logger.dart'; import 'package:omi/utils/platform/platform_service.dart'; import 'package:omi/utils/platform/platform_manager.dart'; @@ -127,6 +127,7 @@ Future _init() async { } else { Env.init(DevEnv()); } + validateApplicationStartupRouting(); FlutterForegroundTask.initCommunicationPort(); @@ -156,18 +157,10 @@ Future _init() async { await SharedPreferencesUtil.init(); - // TestFlight environment detection — must be after SharedPreferencesUtil.init() + // TestFlight remains a distribution/telemetry signal; production-family + // builds always use the established production backend. if (F.env == Environment.prod) { - final isTestFlight = await EnvironmentDetector.isTestFlight(); - if (EnvironmentDetector.shouldUseBetaReleaseRing(isTestFlight)) { - Env.isTestFlight = isTestFlight; - final beta = Env.stagingApiUrl; - if (beta == null) { - throw StateError('Beta release build requires STAGING_API_URL to point at the beta release ring'); - } - Env.overrideApiBaseUrl(beta); - debugPrint('Beta release build detected: using beta release ring ($beta)'); - } + Env.isTestFlight = await EnvironmentDetector.isTestFlight(); } bool isAuth = (await AuthService.instance.getIdToken()) != null; @@ -405,41 +398,7 @@ class _MyAppState extends State with WidgetsBindingObserver { ErrorWidget.builder = (errorDetails) { return CustomErrorWidget(errorMessage: errorDetails.exceptionAsString()); }; - Widget content; - if (Env.isUsingStagingApi) { - final topPadding = MediaQuery.of(context).padding.top; - content = Column( - children: [ - GestureDetector( - onTap: () { - MyApp.navigatorKey.currentState?.push( - MaterialPageRoute(builder: (context) => const DeveloperSettingsPage()), - ); - }, - child: Container( - width: double.infinity, - padding: EdgeInsets.only(top: topPadding + 4, bottom: 4), - color: Colors.orange.shade800, - child: Text( - context.l10n.staging.toUpperCase(), - textAlign: TextAlign.center, - style: const TextStyle( - color: Colors.white, - fontSize: 12, - fontWeight: FontWeight.w600, - decoration: TextDecoration.none, - ), - ), - ), - ), - Expanded( - child: MediaQuery.removePadding(context: context, removeTop: true, child: child!), - ), - ], - ); - } else { - content = child!; - } + final content = child!; return PlatformService.isIOS && Env.posthogApiKey != null ? RageClickContextTracker(child: content) : content; diff --git a/app/lib/pages/settings/developer.dart b/app/lib/pages/settings/developer.dart index bf3d9042666..65238fae1ed 100644 --- a/app/lib/pages/settings/developer.dart +++ b/app/lib/pages/settings/developer.dart @@ -387,34 +387,6 @@ class _DeveloperSettingsPageState extends State<_DeveloperSettingsPageView> { ); } - void _showApiSwitchDialog(BuildContext context, String targetEnvironment) { - final targetName = targetEnvironment == 'production' ? context.l10n.production : context.l10n.staging; - showDialog( - context: context, - builder: (ctx) => AlertDialog( - backgroundColor: const Color(0xFF1C1C1E), - title: Text(context.l10n.switchApiConfirmTitle, style: const TextStyle(color: Colors.white)), - content: Text(context.l10n.switchApiConfirmBody(targetName), style: const TextStyle(color: Colors.white70)), - actions: [ - TextButton( - onPressed: () => Navigator.of(ctx).pop(), - child: Text(context.l10n.cancel, style: const TextStyle(color: Colors.grey)), - ), - TextButton( - onPressed: () async { - Navigator.of(ctx).pop(); - final message = context.l10n.apiEnvSavedRestartRequired; - await SharedPreferencesUtil().saveString('testFlightApiEnvironment', targetEnvironment); - if (!context.mounted) return; - AppSnackbar.showSnackbar(message, duration: const Duration(seconds: 5)); - }, - child: Text(context.l10n.switchAndRestart, style: const TextStyle(color: Colors.orange)), - ), - ], - ), - ); - } - Widget _buildManualFirmwareFlash(DeviceProvider provider) { return _buildSectionContainer( children: [ @@ -514,8 +486,9 @@ class _DeveloperSettingsPageState extends State<_DeveloperSettingsPageView> { _buildNavItem( icon: FontAwesomeIcons.list, title: context.l10n.conversationDisplay, - onTap: () => Navigator.of(context) - .push(MaterialPageRoute(builder: (context) => const ConversationDisplaySettings())), + onTap: () => Navigator.of( + context, + ).push(MaterialPageRoute(builder: (context) => const ConversationDisplaySettings())), ), const SizedBox(height: 12), @@ -1759,151 +1732,6 @@ class _DeveloperSettingsPageState extends State<_DeveloperSettingsPageView> { ), ), - // API Environment Section (TestFlight only, requires STAGING_API_URL env var) - if (Env.isTestFlight && Env.isStagingConfigured) ...[ - const SizedBox(height: 32), - _buildSectionHeader(context.l10n.apiEnvironment, subtitle: context.l10n.apiEnvironmentDescription), - Container( - padding: const EdgeInsets.all(16), - decoration: BoxDecoration( - color: const Color(0xFF1C1C1E), - borderRadius: BorderRadius.circular(14), - ), - child: Column( - crossAxisAlignment: CrossAxisAlignment.start, - children: [ - Container( - decoration: BoxDecoration( - color: const Color(0xFF2A2A2E), - borderRadius: BorderRadius.circular(10), - ), - child: Row( - children: [ - Expanded( - child: GestureDetector( - onTap: () { - if (!SharedPreferencesUtil().testFlightUseStagingApi) return; - _showApiSwitchDialog(context, 'production'); - }, - child: Container( - padding: const EdgeInsets.symmetric(vertical: 12), - decoration: BoxDecoration( - color: !SharedPreferencesUtil().testFlightUseStagingApi - ? const Color(0xFF22C55E) - : Colors.transparent, - borderRadius: BorderRadius.circular(10), - ), - child: Column( - children: [ - Text( - context.l10n.production, - style: TextStyle( - color: !SharedPreferencesUtil().testFlightUseStagingApi - ? Colors.white - : Colors.grey.shade400, - fontSize: 14, - fontWeight: FontWeight.w600, - ), - ), - const SizedBox(height: 2), - Text( - 'api.omi.me', - style: TextStyle( - color: !SharedPreferencesUtil().testFlightUseStagingApi - ? Colors.white70 - : Colors.grey.shade600, - fontSize: 11, - ), - ), - ], - ), - ), - ), - ), - Expanded( - child: GestureDetector( - onTap: () { - if (SharedPreferencesUtil().testFlightUseStagingApi) return; - _showApiSwitchDialog(context, 'staging'); - }, - child: Container( - padding: const EdgeInsets.symmetric(vertical: 12), - decoration: BoxDecoration( - color: SharedPreferencesUtil().testFlightUseStagingApi - ? Colors.orange.shade800 - : Colors.transparent, - borderRadius: BorderRadius.circular(10), - ), - child: Column( - children: [ - Text( - context.l10n.staging, - style: TextStyle( - color: SharedPreferencesUtil().testFlightUseStagingApi - ? Colors.white - : Colors.grey.shade400, - fontSize: 14, - fontWeight: FontWeight.w600, - ), - ), - const SizedBox(height: 2), - Text( - Uri.parse(Env.stagingApiUrl!).host, - style: TextStyle( - color: SharedPreferencesUtil().testFlightUseStagingApi - ? Colors.white70 - : Colors.grey.shade600, - fontSize: 11, - ), - ), - ], - ), - ), - ), - ), - ], - ), - ), - const SizedBox(height: 8), - Row( - children: [ - FaIcon(FontAwesomeIcons.circleInfo, color: Colors.grey.shade600, size: 12), - const SizedBox(width: 6), - Text( - context.l10n.switchRequiresRestart, - style: TextStyle(color: Colors.grey.shade600, fontSize: 12), - ), - ], - ), - ], - ), - ), - if (SharedPreferencesUtil().testFlightUseStagingApi) ...[ - const SizedBox(height: 8), - Container( - margin: const EdgeInsets.symmetric(horizontal: 4), - padding: const EdgeInsets.all(12), - decoration: BoxDecoration( - color: Colors.orange.shade900.withValues(alpha: 0.3), - borderRadius: BorderRadius.circular(8), - border: Border.all(color: Colors.orange.shade700.withValues(alpha: 0.5)), - ), - child: Row( - children: [ - Icon(Icons.warning_amber_rounded, color: Colors.orange.shade300, size: 20), - const SizedBox(width: 8), - Expanded( - child: Text( - context.l10n.stagingDisclaimer, - style: TextStyle(color: Colors.orange.shade300, fontSize: 12), - ), - ), - ], - ), - ), - ], - ], - // Manual Firmware Flash (only when device connected) Builder( builder: (context) { diff --git a/app/lib/startup_routing.dart b/app/lib/startup_routing.dart new file mode 100644 index 00000000000..98a7b813d85 --- /dev/null +++ b/app/lib/startup_routing.dart @@ -0,0 +1,10 @@ +import 'package:omi/env/env.dart'; +import 'package:omi/flavors.dart'; + +/// The startup boundary used by [main] before any networked service starts. +void validateApplicationStartupRouting({Environment? environment, String? configuredApiBaseUrl}) { + Env.validateStartupRouting( + productionFamily: (environment ?? F.env) == Environment.prod, + configuredApiBaseUrl: configuredApiBaseUrl, + ); +} diff --git a/app/lib/utils/environment_detector.dart b/app/lib/utils/environment_detector.dart index 62a4265bb11..a47a3014250 100644 --- a/app/lib/utils/environment_detector.dart +++ b/app/lib/utils/environment_detector.dart @@ -5,14 +5,6 @@ import 'package:flutter/services.dart'; class EnvironmentDetector { static const _channel = MethodChannel('com.omi/environment'); - static const bool isBuildForBetaRing = bool.fromEnvironment('OMI_BETA_RELEASE_RING'); - - /// iOS identifies TestFlight at runtime; Android internal releases carry a - /// compile-time identity. Neither production store channel may opt in by a - /// user preference, which keeps release-ring routing deterministic. - static bool shouldUseBetaReleaseRing(bool isTestFlight, {bool? betaBuild}) { - return isTestFlight || (betaBuild ?? isBuildForBetaRing); - } static Future isTestFlight() async { if (!Platform.isIOS) return false; diff --git a/app/test.sh b/app/test.sh index 2589974b80e..1bf0aa930ae 100755 --- a/app/test.sh +++ b/app/test.sh @@ -45,7 +45,6 @@ if [[ ${#missing_files[@]} -gt 0 ]]; then fi echo "USE_WEB_AUTH=true" >> .dev.env echo "USE_AUTH_CUSTOM_TOKEN=true" >> .dev.env - echo "STAGING_API_URL=" >> .dev.env flutter pub get flutter pub run build_runner build --delete-conflicting-outputs diff --git a/app/test/providers/capture_provider_test.dart b/app/test/providers/capture_provider_test.dart index 72eb3426b1d..50bfc4fb2d0 100644 --- a/app/test/providers/capture_provider_test.dart +++ b/app/test/providers/capture_provider_test.dart @@ -107,8 +107,6 @@ class _TestEnvFields implements EnvFields { bool? get useWebAuth => false; @override bool? get useAuthCustomToken => false; - @override - String? get stagingApiUrl => null; } void main() { diff --git a/app/test/unit/env_empty_staging_test.dart b/app/test/unit/env_empty_staging_test.dart deleted file mode 100644 index 50ff5dcb906..00000000000 --- a/app/test/unit/env_empty_staging_test.dart +++ /dev/null @@ -1,55 +0,0 @@ -import 'package:flutter_test/flutter_test.dart'; -import 'package:omi/env/env.dart'; - -/// EnvFields stub with an empty-string stagingApiUrl to exercise the isEmpty path. -/// Must live in a separate test file because Env._instance is late final. -class _EmptyStagingEnvFields implements EnvFields { - @override - String? get stagingApiUrl => ''; // STAGING_API_URL present but empty - - @override - String? get openAIAPIKey => null; - @override - String? get posthogApiKey => null; - @override - String? get apiBaseUrl => 'https://api.prod.example.com/'; - @override - String? get googleMapsApiKey => null; - @override - String? get intercomAppId => null; - @override - String? get intercomIOSApiKey => null; - @override - String? get intercomAndroidApiKey => null; - @override - String? get googleClientId => null; - @override - String? get googleClientSecret => null; - @override - bool? get useWebAuth => false; - @override - bool? get useAuthCustomToken => false; -} - -void main() { - setUpAll(() { - Env.init(_EmptyStagingEnvFields()); - }); - - group('Env.stagingApiUrl with empty STAGING_API_URL', () { - test('returns null when env var is empty string', () { - expect(Env.stagingApiUrl, isNull); - }); - - test('isStagingConfigured is false when env var is empty', () { - expect(Env.isStagingConfigured, isFalse); - }); - }); - - group('Env.isUsingStagingApi with empty STAGING_API_URL', () { - test('false even when override matches empty staging', () { - Env.overrideApiBaseUrl(''); - expect(Env.isUsingStagingApi, isFalse); - }); - }); -} diff --git a/app/test/unit/env_staging_test.dart b/app/test/unit/env_staging_test.dart deleted file mode 100644 index 7950eb3533e..00000000000 --- a/app/test/unit/env_staging_test.dart +++ /dev/null @@ -1,65 +0,0 @@ -import 'package:flutter_test/flutter_test.dart'; -import 'package:omi/env/env.dart'; - -/// EnvFields stub with an explicit stagingApiUrl to exercise the isTrue path. -/// Must live in a separate test file because Env._instance is late final. -class _StagingEnvFields implements EnvFields { - @override - String? get stagingApiUrl => 'https://staging.omiapi.com/'; - - @override - String? get openAIAPIKey => null; - @override - String? get posthogApiKey => null; - @override - String? get apiBaseUrl => 'https://api.prod.example.com/'; - @override - String? get googleMapsApiKey => null; - @override - String? get intercomAppId => null; - @override - String? get intercomIOSApiKey => null; - @override - String? get intercomAndroidApiKey => null; - @override - String? get googleClientId => null; - @override - String? get googleClientSecret => null; - @override - bool? get useWebAuth => false; - @override - bool? get useAuthCustomToken => false; -} - -void main() { - setUpAll(() { - Env.init(_StagingEnvFields()); - }); - - group('Env.stagingApiUrl with STAGING_API_URL configured', () { - test('returns the env var value', () { - expect(Env.stagingApiUrl, 'https://staging.omiapi.com/'); - }); - - test('isStagingConfigured is true', () { - expect(Env.isStagingConfigured, isTrue); - }); - }); - - group('Env.isUsingStagingApi with explicit stagingApiUrl', () { - test('true when override matches configured stagingApiUrl', () { - Env.overrideApiBaseUrl('https://staging.omiapi.com/'); - expect(Env.isUsingStagingApi, isTrue); - }); - - test('true with normalisation — trailing slash and case differences', () { - Env.overrideApiBaseUrl('https://Staging.OmiApi.com'); - expect(Env.isUsingStagingApi, isTrue); - }); - - test('false when override points to a different URL', () { - Env.overrideApiBaseUrl('https://api.prod.example.com/'); - expect(Env.isUsingStagingApi, isFalse); - }); - }); -} diff --git a/app/test/unit/env_test.dart b/app/test/unit/env_test.dart index d2f180c5af6..31932acd6f6 100644 --- a/app/test/unit/env_test.dart +++ b/app/test/unit/env_test.dart @@ -1,19 +1,19 @@ import 'package:flutter_test/flutter_test.dart'; import 'package:omi/env/env.dart'; +import 'package:omi/flavors.dart'; +import 'package:omi/startup_routing.dart'; +import 'dart:io'; /// Minimal EnvFields stub for testing Env logic in isolation. /// Since Env._instance is late final (can only be set once per process), /// we test with a single init and exercise the override/flag mechanisms. class _TestEnvFields implements EnvFields { - @override - String? get stagingApiUrl => null; // STAGING_API_URL not configured - @override String? get openAIAPIKey => null; @override String? get posthogApiKey => null; @override - String? get apiBaseUrl => 'https://api.prod.example.com/'; + String? get apiBaseUrl => 'https://api.omi.me/'; @override String? get googleMapsApiKey => null; @override @@ -38,30 +38,6 @@ void main() { Env.init(_TestEnvFields()); }); - group('Env.stagingApiUrl', () { - test('returns null when STAGING_API_URL is not configured', () { - expect(Env.stagingApiUrl, isNull); - }); - }); - - group('Env.isStagingConfigured', () { - test('false when STAGING_API_URL is not set', () { - expect(Env.isStagingConfigured, isFalse); - }); - }); - - group('Env.isUsingStagingApi', () { - test('false when staging is not configured', () { - Env.overrideApiBaseUrl('https://api.omiapi.com/'); - expect(Env.isUsingStagingApi, isFalse); - }); - - test('false when override points to non-staging URL', () { - Env.overrideApiBaseUrl('https://api.prod.example.com/'); - expect(Env.isUsingStagingApi, isFalse); - }); - }); - group('Env.isTestFlight', () { test('can be set to false', () { Env.isTestFlight = false; @@ -80,6 +56,48 @@ void main() { test('returns override when set', () { Env.overrideApiBaseUrl('https://override.example.com/'); expect(Env.apiBaseUrl, 'https://override.example.com/'); + Env.clearApiBaseUrlOverrideForTesting(); + }); + + test('TestFlight production startup accepts the production API and WebSocket', () { + validateApplicationStartupRouting(environment: Environment.prod, configuredApiBaseUrl: 'https://api.omi.me/'); + expect(Env.productionAgentProxyWsUrl, 'wss://agent.omi.me/v1/agent/ws'); }); + + test('Android production startup accepts the production API and WebSocket', () { + validateApplicationStartupRouting(environment: Environment.prod, configuredApiBaseUrl: 'https://api.omi.me/'); + expect(Env.productionAgentProxyWsUrl, 'wss://agent.omi.me/v1/agent/ws'); + }); + + test('production startup rejects legacy Beta, dev, staging, and arbitrary endpoints', () { + for (final endpoint in [ + 'https://api-beta.omi.me/', + 'https://api.omi.dev/', + 'https://staging.example.test/', + 'https://arbitrary.example.test/', + ]) { + expect( + () => validateApplicationStartupRouting(environment: Environment.prod, configuredApiBaseUrl: endpoint), + throwsStateError, + reason: endpoint, + ); + } + }); + + test('development startup remains configurable', () { + expect( + () => validateApplicationStartupRouting( + environment: Environment.dev, configuredApiBaseUrl: 'https://api.omi.dev/'), + returnsNormally, + ); + }); + }); + + test('main invokes the production startup routing seam before services initialize', () { + // Static wiring tripwire: the behavioral cases above call the exact seam. + final mainSource = File('lib/main.dart').readAsStringSync(); + expect(mainSource, contains('validateApplicationStartupRouting();')); + expect(mainSource.indexOf('validateApplicationStartupRouting();'), + lessThan(mainSource.indexOf('ServiceManager.init()'))); }); } diff --git a/app/test/unit/environment_detector_test.dart b/app/test/unit/environment_detector_test.dart deleted file mode 100644 index c9d40a03683..00000000000 --- a/app/test/unit/environment_detector_test.dart +++ /dev/null @@ -1,16 +0,0 @@ -import 'package:flutter_test/flutter_test.dart'; -import 'package:omi/utils/environment_detector.dart'; - -void main() { - test('TestFlight builds always select the beta release ring', () { - expect(EnvironmentDetector.shouldUseBetaReleaseRing(true, betaBuild: false), isTrue); - }); - - test('Android internal build identity selects beta without TestFlight', () { - expect(EnvironmentDetector.shouldUseBetaReleaseRing(false, betaBuild: true), isTrue); - }); - - test('store build identity remains out of the beta release ring', () { - expect(EnvironmentDetector.shouldUseBetaReleaseRing(false, betaBuild: false), isFalse); - }); -} diff --git a/app/test/unit/testflight_preferences_test.dart b/app/test/unit/testflight_preferences_test.dart deleted file mode 100644 index 955eb09356e..00000000000 --- a/app/test/unit/testflight_preferences_test.dart +++ /dev/null @@ -1,40 +0,0 @@ -import 'package:flutter_test/flutter_test.dart'; -import 'package:shared_preferences/shared_preferences.dart'; -import 'package:omi/backend/preferences.dart'; - -void main() { - group('SharedPreferencesUtil TestFlight API Environment', () { - setUp(() async { - SharedPreferences.setMockInitialValues({}); - await SharedPreferencesUtil.init(); - }); - - test('default testFlightApiEnvironment is beta staging', () { - expect(SharedPreferencesUtil().testFlightApiEnvironment, 'staging'); - }); - - test('default testFlightUseStagingApi is true', () { - expect(SharedPreferencesUtil().testFlightUseStagingApi, isTrue); - }); - - test('setting production flips testFlightUseStagingApi to false', () { - SharedPreferencesUtil().testFlightApiEnvironment = 'production'; - expect(SharedPreferencesUtil().testFlightApiEnvironment, 'production'); - expect(SharedPreferencesUtil().testFlightUseStagingApi, isFalse); - }); - - test('setting staging flips testFlightUseStagingApi to true', () { - SharedPreferencesUtil().testFlightApiEnvironment = 'production'; - expect(SharedPreferencesUtil().testFlightUseStagingApi, isFalse); - // Switch back - SharedPreferencesUtil().testFlightApiEnvironment = 'staging'; - expect(SharedPreferencesUtil().testFlightUseStagingApi, isTrue); - }); - - test('persists through saveString', () async { - final result = await SharedPreferencesUtil().saveString('testFlightApiEnvironment', 'production'); - expect(result, isTrue); - expect(SharedPreferencesUtil().testFlightApiEnvironment, 'production'); - }); - }); -} diff --git a/backend/charts/agent-proxy/dev_omi_agent_proxy_values.yaml b/backend/charts/agent-proxy/dev_omi_agent_proxy_values.yaml index 2498c718ba9..df7b8f12b49 100644 --- a/backend/charts/agent-proxy/dev_omi_agent_proxy_values.yaml +++ b/backend/charts/agent-proxy/dev_omi_agent_proxy_values.yaml @@ -2,7 +2,6 @@ replicaCount: 1 image: repository: gcr.io/based-hardware-dev/agent-proxy - digest: "" pullPolicy: Always tag: "" diff --git a/backend/charts/agent-proxy/prod_omi_agent_proxy_values.yaml b/backend/charts/agent-proxy/prod_omi_agent_proxy_values.yaml index a84e31510b0..7a4aa1745ff 100644 --- a/backend/charts/agent-proxy/prod_omi_agent_proxy_values.yaml +++ b/backend/charts/agent-proxy/prod_omi_agent_proxy_values.yaml @@ -2,7 +2,6 @@ replicaCount: 1 image: repository: gcr.io/based-hardware/agent-proxy - digest: "" pullPolicy: Always tag: "" diff --git a/backend/charts/agent-proxy/templates/deployment.yaml b/backend/charts/agent-proxy/templates/deployment.yaml index 4e339b3733f..fb5b106681d 100644 --- a/backend/charts/agent-proxy/templates/deployment.yaml +++ b/backend/charts/agent-proxy/templates/deployment.yaml @@ -30,7 +30,7 @@ spec: serviceAccountName: {{ include "agent-proxy.serviceAccountName" . }} containers: - name: {{ .Chart.Name }} - image: "{{ required "image.repository is required" .Values.image.repository }}{{ if .Values.image.digest }}@{{ .Values.image.digest }}{{ else }}:{{ required "image.tag is required" .Values.image.tag }}{{ end }}" + image: "{{ required "image.repository is required" .Values.image.repository }}:{{ required "image.tag is required" .Values.image.tag }}" imagePullPolicy: {{ .Values.image.pullPolicy }} {{- with .Values.env }} env: diff --git a/backend/charts/backend-listen/dev_omi_backend_listen_values.yaml b/backend/charts/backend-listen/dev_omi_backend_listen_values.yaml index acb393bc8a0..ac370127816 100644 --- a/backend/charts/backend-listen/dev_omi_backend_listen_values.yaml +++ b/backend/charts/backend-listen/dev_omi_backend_listen_values.yaml @@ -9,7 +9,6 @@ replicaCount: 1 # This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/ image: repository: gcr.io/based-hardware-dev/backend - digest: "" # This sets the pull policy for images. pullPolicy: Always # Build-specific; CI/manual deploy must pass --set-string image.tag=. diff --git a/backend/charts/backend-listen/prod_omi_backend_listen_values.yaml b/backend/charts/backend-listen/prod_omi_backend_listen_values.yaml index 5e7e7e3811e..4820af85ae0 100644 --- a/backend/charts/backend-listen/prod_omi_backend_listen_values.yaml +++ b/backend/charts/backend-listen/prod_omi_backend_listen_values.yaml @@ -9,7 +9,6 @@ replicaCount: 1 # This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/ image: repository: gcr.io/based-hardware/backend - digest: "" # This sets the pull policy for images. pullPolicy: Always # Build-specific; CI/manual deploy must pass --set-string image.tag=. diff --git a/backend/charts/backend-listen/templates/deployment.yaml b/backend/charts/backend-listen/templates/deployment.yaml index 45045b6ab59..7e1e7663205 100644 --- a/backend/charts/backend-listen/templates/deployment.yaml +++ b/backend/charts/backend-listen/templates/deployment.yaml @@ -53,7 +53,7 @@ spec: securityContext: {{- toYaml . | nindent 12 }} {{- end }} - image: "{{ required "image.repository is required" .Values.image.repository }}{{ if .Values.image.digest }}@{{ .Values.image.digest }}{{ else }}:{{ required "image.tag is required" .Values.image.tag }}{{ end }}" + image: "{{ required "image.repository is required" .Values.image.repository }}:{{ required "image.tag is required" .Values.image.tag }}" imagePullPolicy: {{ .Values.image.pullPolicy }} {{- with .Values.envFrom }} envFrom: diff --git a/backend/charts/backend-secrets/templates/externalsecret.yaml b/backend/charts/backend-secrets/templates/externalsecret.yaml index f04d3ae38cd..4164c24b99a 100644 --- a/backend/charts/backend-secrets/templates/externalsecret.yaml +++ b/backend/charts/backend-secrets/templates/externalsecret.yaml @@ -15,7 +15,4 @@ spec: - secretKey: {{ .secretKey }} remoteRef: key: {{ .remoteKey }} - {{- with .version }} - version: {{ . }} - {{- end }} {{- end }} diff --git a/backend/charts/llm-gateway/dev_omi_llm_gateway_values.yaml b/backend/charts/llm-gateway/dev_omi_llm_gateway_values.yaml index 5c07e4d9581..eed4f680492 100644 --- a/backend/charts/llm-gateway/dev_omi_llm_gateway_values.yaml +++ b/backend/charts/llm-gateway/dev_omi_llm_gateway_values.yaml @@ -2,7 +2,6 @@ replicaCount: 1 image: repository: gcr.io/based-hardware-dev/llm-gateway - digest: "" pullPolicy: Always tag: "" diff --git a/backend/charts/llm-gateway/prod_omi_llm_gateway_values.yaml b/backend/charts/llm-gateway/prod_omi_llm_gateway_values.yaml index 10081fb6188..a9e221b0959 100644 --- a/backend/charts/llm-gateway/prod_omi_llm_gateway_values.yaml +++ b/backend/charts/llm-gateway/prod_omi_llm_gateway_values.yaml @@ -2,7 +2,6 @@ replicaCount: 2 image: repository: gcr.io/based-hardware/llm-gateway - digest: "" pullPolicy: Always tag: "" diff --git a/backend/charts/llm-gateway/templates/deployment.yaml b/backend/charts/llm-gateway/templates/deployment.yaml index 012ca1eb89b..4446e510049 100644 --- a/backend/charts/llm-gateway/templates/deployment.yaml +++ b/backend/charts/llm-gateway/templates/deployment.yaml @@ -38,7 +38,7 @@ spec: securityContext: {{- toYaml . | nindent 12 }} {{- end }} - image: "{{ required "image.repository is required" .Values.image.repository }}{{ if .Values.image.digest }}@{{ .Values.image.digest }}{{ else }}:{{ required "image.tag is required" .Values.image.tag }}{{ end }}" + image: "{{ required "image.repository is required" .Values.image.repository }}:{{ required "image.tag is required" .Values.image.tag }}" imagePullPolicy: {{ .Values.image.pullPolicy }} command: ["uvicorn"] args: ["llm_gateway.main:app", "--host", "0.0.0.0", "--port", "{{ .Values.service.port }}", "--loop", "uvloop"] diff --git a/backend/charts/pusher/dev_omi_pusher_values.yaml b/backend/charts/pusher/dev_omi_pusher_values.yaml index 5aec08d46c4..83a3d2db599 100644 --- a/backend/charts/pusher/dev_omi_pusher_values.yaml +++ b/backend/charts/pusher/dev_omi_pusher_values.yaml @@ -14,7 +14,6 @@ replicaCount: 1 # This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/ image: repository: gcr.io/based-hardware-dev/pusher - digest: "" # This sets the pull policy for images. pullPolicy: Always # Overrides the image tag whose default is the chart appVersion. diff --git a/backend/charts/pusher/prod_omi_pusher_values.yaml b/backend/charts/pusher/prod_omi_pusher_values.yaml index 28a17e043cc..adce8b50ca5 100644 --- a/backend/charts/pusher/prod_omi_pusher_values.yaml +++ b/backend/charts/pusher/prod_omi_pusher_values.yaml @@ -15,7 +15,6 @@ replicaCount: 1 # This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/ image: repository: gcr.io/based-hardware/pusher - digest: "" # This sets the pull policy for images. pullPolicy: Always # Overrides the image tag whose default is the chart appVersion. diff --git a/backend/charts/pusher/templates/deployment.yaml b/backend/charts/pusher/templates/deployment.yaml index d59bc6c9d03..fd9926564e1 100644 --- a/backend/charts/pusher/templates/deployment.yaml +++ b/backend/charts/pusher/templates/deployment.yaml @@ -54,7 +54,7 @@ spec: securityContext: {{- toYaml . | nindent 12 }} {{- end }} - image: "{{ required "image.repository is required" .Values.image.repository }}{{ if .Values.image.digest }}@{{ .Values.image.digest }}{{ else }}:{{ required "image.tag is required" .Values.image.tag }}{{ end }}" + image: "{{ required "image.repository is required" .Values.image.repository }}:{{ required "image.tag is required" .Values.image.tag }}" imagePullPolicy: {{ .Values.image.pullPolicy }} {{- with .Values.envFrom }} envFrom: diff --git a/backend/database/desktop_update_channels.py b/backend/database/desktop_update_channels.py index a8d74617725..b2d02d1d60b 100644 --- a/backend/database/desktop_update_channels.py +++ b/backend/database/desktop_update_channels.py @@ -4,7 +4,6 @@ import re from typing import Any, cast from urllib.parse import urlparse -from uuid import uuid4 from google.cloud.firestore import transactional @@ -12,7 +11,6 @@ CHANNELS_COLLECTION = "desktop_update_channels" MANIFESTS_COLLECTION = "desktop_release_manifests" -ROLLBACK_AUDITS_COLLECTION = "desktop_update_channel_rollback_audits" VALID_CHANNELS = frozenset({"stable", "beta"}) VALID_PLATFORMS = frozenset({"macos", "windows", "linux"}) SHA40_RE = re.compile(r"^[0-9a-f]{40}$", re.IGNORECASE) @@ -113,13 +111,18 @@ def normalize_release_manifest(data: dict[str, Any]) -> dict[str, Any]: "build_number": _positive_int(data, "build_number"), "zip_url": _https_url(data, "zip_url", required=True), "dmg_url": _https_url(data, "dmg_url", required=platform == "macos"), + "beta_zip_url": _https_url(data, "beta_zip_url", required=platform == "macos"), + "beta_dmg_url": _https_url(data, "beta_dmg_url", required=platform == "macos"), "ed_signature": _required_string(data, "ed_signature"), + "beta_ed_signature": _required_string(data, "beta_ed_signature"), "published_at": published_at, "changelog": [item.strip() for item in changelog if item.strip()], "mandatory": data.get("mandatory") is True, "source_sha": _digest(data, "source_sha", SHA40_RE, required=True), "zip_sha256": _digest(data, "zip_sha256", SHA256_RE, required=False), "dmg_sha256": _digest(data, "dmg_sha256", SHA256_RE, required=False), + "beta_zip_sha256": _digest(data, "beta_zip_sha256", SHA256_RE, required=platform == "macos"), + "beta_dmg_sha256": _digest(data, "beta_dmg_sha256", SHA256_RE, required=platform == "macos"), "qualification": cast(dict[str, Any], qualification), } return manifest @@ -145,12 +148,11 @@ def register_release_manifest(data: dict[str, Any], *, firestore_client: Any = N #: Pointer transitions differ only in which preconditions they enforce, so the #: policy is data and :func:`_build_pointer` is the single mutation authority. -#: ``direction`` is the permitted build-number movement; ``require_qualified`` -#: is the T2 evidence gate that emergency beta promotion is allowed to skip. +#: ``direction`` is the permitted build-number movement; every transition +#: requires the manifest's passed T2 qualification evidence. TRANSITIONS: dict[str, dict[str, Any]] = { "promote": {"direction": "forward", "require_qualified": True, "require_current_release_id": False}, - "rollback": {"direction": "backward", "require_qualified": True, "require_current_release_id": True}, - "emergency": {"direction": "forward", "require_qualified": False, "require_current_release_id": True}, + "repoint": {"direction": "either", "require_qualified": True, "require_current_release_id": True}, } @@ -168,24 +170,26 @@ def _build_pointer( ) -> dict[str, Any]: """Build the next channel pointer under the named transition policy. - This is the only place a channel pointer is constructed. Emergency beta - promotion is an ordinary promotion that skips the T2 qualification gate; - it is not a separate mutation path. + This is the only place a channel pointer is constructed. A repoint is an + explicit compare-and-swap to an existing qualified manifest. """ policy = TRANSITIONS[transition] if manifest["platform"] != platform: raise ValueError("release manifest platform does not match pointer platform") - if transition != "promote" and (platform, channel) != ("macos", "beta"): - raise ValueError(f"{transition} is only permitted for the macos beta channel") - if policy["require_qualified"]: qualification = cast(dict[str, Any], manifest["qualification"]) if qualification.get("passed") is not True or str(qualification.get("tier", "")).upper() != "T2": raise ValueError("release manifest is missing passed T2 qualification evidence") current_release_id = current.get("release_id") - if policy["require_current_release_id"]: + # An acknowledged pointer target is a safe exact retry. It still had to + # resolve to this qualified immutable manifest above, but does not require + # callers to guess the generation created by a lost response. + if current_release_id == release_id: + return current + + if policy["require_current_release_id"] or expected_current_release_id is not None: if current_release_id != expected_current_release_id: raise ValueError( f"current release mismatch: expected {expected_current_release_id}, " @@ -196,11 +200,6 @@ def _build_pointer( if expected_generation is not None and expected_generation != current_generation: raise ValueError(f"generation mismatch: expected {expected_generation}, current {current_generation}") - # Re-promoting the live release is a no-op only where no explicit - # compare-and-swap was requested; the CAS transitions must still advance. - if not policy["require_current_release_id"] and current_release_id == release_id: - return current - current_build_raw = current.get("build_number") if policy["direction"] == "forward": if current_build_raw is not None and manifest["build_number"] <= _generation(current_build_raw): @@ -208,8 +207,6 @@ def _build_pointer( f"channel pointers are roll-forward only: current build {_generation(current_build_raw)}, " f"requested build {manifest['build_number']}" ) - elif release_id == current_release_id or manifest["build_number"] >= _generation(current_build_raw): - raise ValueError("rollback target must be an earlier qualified beta release") pointer = { "platform": platform, @@ -220,10 +217,6 @@ def _build_pointer( "generation": current_generation + 1, "updated_at": updated_at or datetime.now(timezone.utc), } - if transition == "emergency": - # The audit trail lives on the pointer itself: one field, queryable with - # the pointer, rather than a parallel collection that can diverge. - pointer["emergency"] = True return pointer @@ -272,15 +265,10 @@ def promote_channel( *, expected_generation: int | None = None, expected_current_release_id: str | None = None, - emergency: bool = False, + operation: str = "promote", firestore_client: Any = None, ) -> dict[str, Any]: - """Advance a channel pointer. - - ``emergency=True`` is the beta-only break-glass path: it skips the T2 - qualification gate and therefore requires an explicit compare-and-swap on - both the current release id and generation. - """ + """Advance or explicitly repoint a channel pointer to a qualified manifest.""" platform = platform.strip().lower() channel = channel.strip().lower() release_id = release_id.strip() @@ -290,8 +278,10 @@ def promote_channel( raise ValueError("invalid channel") if not release_id: raise ValueError("release_id is required") - if emergency and (expected_current_release_id is None or expected_generation is None): - raise ValueError("emergency promotion requires expected_current_release_id and expected_generation") + if operation not in TRANSITIONS: + raise ValueError("invalid pointer operation") + if operation == "repoint" and (expected_current_release_id is None or expected_generation is None): + raise ValueError("repoint requires expected_current_release_id and expected_generation") client = firestore_client if firestore_client is not None else get_firestore_client() pointer_ref = client.collection(CHANNELS_COLLECTION).document(f"{platform}-{channel}") @@ -301,7 +291,7 @@ def promote_channel( transaction, pointer_ref, manifest_ref, - transition="emergency" if emergency else "promote", + transition=operation, platform=platform, channel=channel, release_id=release_id, @@ -310,95 +300,6 @@ def promote_channel( ) -@transactional -def _rollback_macos_beta_transaction( - transaction: Any, - pointer_ref: Any, - manifest_ref: Any, - audit_ref: Any, - *, - release_id: str, - expected_current_release_id: str, - expected_generation: int, - audit_id: str, - occurred_at: datetime, -) -> dict[str, Any]: - manifest_snapshot = manifest_ref.get(transaction=transaction) - if not getattr(manifest_snapshot, "exists", False): - raise ValueError("rollback target release manifest does not exist") - raw_manifest: object = manifest_snapshot.to_dict() - manifest_data = cast(dict[str, Any], raw_manifest) if isinstance(raw_manifest, dict) else {} - manifest = normalize_release_manifest(manifest_data) - - pointer_snapshot = pointer_ref.get(transaction=transaction) - if not getattr(pointer_snapshot, "exists", False): - raise ValueError("current macos beta pointer does not exist") - current_raw: object = pointer_snapshot.to_dict() - current = cast(dict[str, Any], current_raw) if isinstance(current_raw, dict) else {} - pointer = _build_pointer( - current, - manifest, - transition="rollback", - platform="macos", - channel="beta", - release_id=release_id, - expected_current_release_id=expected_current_release_id, - expected_generation=expected_generation, - updated_at=occurred_at, - ) - audit = { - "audit_id": audit_id, - "operation": "macos_beta_rollback", - "platform": "macos", - "channel": "beta", - "previous_release_id": expected_current_release_id, - "previous_generation": expected_generation, - "target_release_id": release_id, - "generation": pointer["generation"], - "occurred_at": occurred_at, - } - # create() provides an immutable, append-only audit record. All reads above - # occur before this first transactional write. - transaction.create(audit_ref, audit) - transaction.set(pointer_ref, pointer) - return {"pointer": pointer, "audit": audit} - - -def rollback_macos_beta_channel( - release_id: str, - *, - expected_current_release_id: str, - expected_generation: int, - firestore_client: Any = None, -) -> dict[str, Any]: - """Atomically roll macOS beta back to an earlier, qualified registered release only.""" - release_id = release_id.strip() - expected_current_release_id = expected_current_release_id.strip() - if not release_id: - raise ValueError("release_id is required") - if not expected_current_release_id: - raise ValueError("expected_current_release_id is required") - if expected_generation < 0: - raise ValueError("expected_generation must be a non-negative integer") - - client = firestore_client if firestore_client is not None else get_firestore_client() - pointer_ref = client.collection(CHANNELS_COLLECTION).document("macos-beta") - manifest_ref = client.collection(MANIFESTS_COLLECTION).document(release_id) - audit_id = uuid4().hex - audit_ref = client.collection(ROLLBACK_AUDITS_COLLECTION).document(audit_id) - return _rollback_macos_beta_transaction( - client.transaction(), - pointer_ref, - manifest_ref, - audit_ref, - release_id=release_id, - expected_current_release_id=expected_current_release_id, - expected_generation=expected_generation, - audit_id=audit_id, - occurred_at=datetime.now(timezone.utc), - ) - - def get_channel_release(platform: str, channel: str, *, firestore_client: Any = None) -> dict[str, Any] | None: """Resolve one explicit channel pointer to its immutable manifest.""" if platform not in VALID_PLATFORMS or channel not in VALID_CHANNELS: @@ -433,3 +334,17 @@ def get_channel_release(platform: str, channel: str, *, firestore_client: Any = }, "manifest": manifest, } + + +def get_release_manifest(release_id: str, *, firestore_client: Any = None) -> dict[str, Any] | None: + """Read one retained immutable manifest without consulting release metadata.""" + release_id = release_id.strip() + if not release_id: + raise ValueError("release_id is required") + client = firestore_client if firestore_client is not None else get_firestore_client() + snapshot = client.collection(MANIFESTS_COLLECTION).document(release_id).get() + if not getattr(snapshot, "exists", False): + return None + raw: object = snapshot.to_dict() + data = cast(dict[str, Any], raw) if isinstance(raw, dict) else {} + return normalize_release_manifest(data) diff --git a/backend/deploy/release_rings.yaml b/backend/deploy/release_rings.yaml deleted file mode 100644 index f322ed99684..00000000000 --- a/backend/deploy/release_rings.yaml +++ /dev/null @@ -1,33 +0,0 @@ -# Stateless release-ring topology. This is intentionally not a second runtime -# manifest: release records render immutable copies of the referenced values -# and runtime config before beta/prod deployment. -schema_version: 1 -rings: - beta: - environment: beta - namespace: beta-omi-backend - shared_data_stores: true - shared_gpu_endpoints: true - endpoints: - api: https://api-beta.omi.me - pusher: http://pusher-beta.omi.me - agent: wss://agent-beta.omi.me - cloud_run_services: - backend: backend-beta - backend-sync: backend-sync-beta - backend-sync-backfill: backend-sync-backfill-beta - backend-integration: backend-integration-beta - prod: - environment: prod - namespace: prod-omi-backend - shared_data_stores: true - shared_gpu_endpoints: true - endpoints: - api: https://api.omi.me - pusher: http://pusher.omi.me - agent: wss://agent.omi.me - cloud_run_services: - backend: backend - backend-sync: backend-sync - backend-sync-backfill: backend-sync-backfill - backend-integration: backend-integration diff --git a/backend/models/memory_apply.py b/backend/models/memory_apply.py index 6db724a2b7a..de05e86ba88 100644 --- a/backend/models/memory_apply.py +++ b/backend/models/memory_apply.py @@ -296,7 +296,7 @@ def _apply_update_memory_item( promotion_audit: Optional[Dict[str, Any]] = None, extra_updates: Optional[Dict[str, Any]] = None, ) -> MemoryItem: - now = datetime.now(timezone.utc) + now = max(datetime.now(timezone.utc), existing.captured_at, existing.updated_at) if patch.target_tier is not None: tier = patch.target_tier else: diff --git a/backend/route_policy_manifest.yaml b/backend/route_policy_manifest.yaml index 10adf3db240..c86e168c8a7 100644 --- a/backend/route_policy_manifest.yaml +++ b/backend/route_policy_manifest.yaml @@ -140,8 +140,8 @@ routes: deprecation: state: active - route_type: http - method: POST - path: /v2/desktop/channels/rollback + method: GET + path: /v2/desktop/releases/{release_id} policy: review_status: reviewed auth: diff --git a/backend/routers/updates.py b/backend/routers/updates.py index 13efcb5c783..0d506cb4172 100644 --- a/backend/routers/updates.py +++ b/backend/routers/updates.py @@ -1,5 +1,7 @@ from html import escape as html_escape import hmac +import hashlib +import json import logging import os import random @@ -13,9 +15,9 @@ from database.desktop_previews import delist_preview, get_current_preview, get_preview_manifest, publish_preview from database.desktop_update_channels import ( + get_release_manifest, promote_channel, register_release_manifest, - rollback_macos_beta_channel, ) from database.desktop_update_policy import default_desktop_update_policy, get_desktop_update_policy from database.redis_db import delete_generic_cache @@ -65,13 +67,18 @@ class DesktopReleaseManifestRequest(BaseModel): build_number: int = Field(gt=0) zip_url: str dmg_url: Optional[str] = None + beta_zip_url: Optional[str] = None + beta_dmg_url: Optional[str] = None ed_signature: str + beta_ed_signature: Optional[str] = None published_at: str changelog: List[str] = Field(default_factory=list) mandatory: bool = False source_sha: str zip_sha256: Optional[str] = None dmg_sha256: Optional[str] = None + beta_zip_sha256: Optional[str] = None + beta_dmg_sha256: Optional[str] = None qualification: Dict[str, Any] = Field(default_factory=dict) @@ -81,19 +88,7 @@ class DesktopChannelPromotionRequest(BaseModel): release_id: str expected_generation: Optional[int] = Field(default=None, ge=0) expected_current_release_id: Optional[str] = None - #: Beta-only break-glass: skip the T2 qualification gate. Requires both - #: compare-and-swap fields above so the bypass cannot race normal promotion. - emergency: bool = False - - -class DesktopBetaRollbackRequest(BaseModel): - """Emergency-only rollback request; the literal fields prevent cross-channel reuse.""" - - platform: Literal["macos"] - channel: Literal["beta"] - release_id: str = Field(min_length=1) - expected_current_release_id: str = Field(min_length=1) - expected_generation: int = Field(ge=0) + operation: Literal["promote", "repoint"] = "promote" class DesktopPreviewPublishRequest(BaseModel): @@ -282,14 +277,7 @@ async def _find_desktop_release_by_tag(tag_name: str) -> Optional[Dict]: async def _resolve_beta_identity_dmg(entry: Dict) -> Optional[str]: """Beta-identity DMG URL for this entry's release, or None when it predates the dual-identity pipeline.""" - url = _get_asset_download_url(entry["release"], {BETA_IDENTITY_DMG_ASSET}) - if url: - return url - tag = (entry.get("version_info") or {}).get("tag_name") or entry["release"].get("tag_name", "") - gh_release = await _find_desktop_release_by_tag(tag) - if not gh_release: - return None - return _get_asset_download_url(gh_release, {BETA_IDENTITY_DMG_ASSET}) + return _get_asset_download_url(entry["release"], {BETA_IDENTITY_DMG_ASSET}) async def _resolve_beta_identity_enclosure(entry: Dict) -> Optional[tuple]: @@ -303,13 +291,6 @@ async def _resolve_beta_identity_enclosure(entry: Dict) -> Optional[tuple]: metadata = entry.get("metadata") or {} url = _get_asset_download_url(release, {BETA_IDENTITY_SPARKLE_ASSET}) signature = (metadata.get("betaEdSignature") or "").strip() - if not url: - tag = (entry.get("version_info") or {}).get("tag_name") or release.get("tag_name", "") - gh_release = await _find_desktop_release_by_tag(tag) - if not gh_release: - return None - url = _get_asset_download_url(gh_release, {BETA_IDENTITY_SPARKLE_ASSET}) - signature = (extract_key_value_pairs(gh_release.get("body", "")).get("betaEdSignature") or "").strip() if not url or not signature: return None return url, signature @@ -376,6 +357,10 @@ def _pointer_release_to_entry(release: Dict[str, Any], channel: str, source: str assets = [{"name": "Omi.zip", "browser_download_url": manifest["zip_url"]}] if manifest.get("dmg_url"): assets.append({"name": "Omi.dmg", "browser_download_url": manifest["dmg_url"]}) + if manifest.get("beta_zip_url"): + assets.append({"name": "Omi.Beta.zip", "browser_download_url": manifest["beta_zip_url"]}) + if manifest.get("beta_dmg_url"): + assets.append({"name": "omi-beta.dmg", "browser_download_url": manifest["beta_dmg_url"]}) return { "channel": channel, "source": source, @@ -392,6 +377,9 @@ def _pointer_release_to_entry(release: Dict[str, Any], channel: str, source: str }, "metadata": { "edSignature": manifest["ed_signature"], + "betaEdSignature": manifest["beta_ed_signature"], + "betaZipSha256": manifest.get("beta_zip_sha256"), + "betaDmgSha256": manifest.get("beta_dmg_sha256"), "changelog": manifest.get("changelog", []), "mandatory": "true" if manifest.get("mandatory") else "false", "sourceSha": manifest["source_sha"], @@ -993,15 +981,21 @@ async def register_desktop_release(request: DesktopReleaseManifestRequest, secre return {"success": True, "manifest": manifest} +@router.get("/v2/desktop/releases/{release_id}") +async def get_desktop_release_manifest(release_id: str, secret_key: str = Header(...)): + """Return the retained manifest used for a pointer transition, not GitHub metadata.""" + if secret_key != os.getenv('ADMIN_KEY'): + raise HTTPException(status_code=403, detail='You are not authorized to perform this action') + manifest = await run_blocking(db_executor, get_release_manifest, release_id) + if manifest is None: + raise HTTPException(status_code=404, detail='desktop release manifest not found') + canonical = json.dumps(manifest, sort_keys=True, separators=(",", ":")).encode("utf-8") + return {"success": True, "manifest": manifest, "manifest_sha256": hashlib.sha256(canonical).hexdigest()} + + @router.post("/v2/desktop/channels/promote") async def promote_desktop_channel(request: DesktopChannelPromotionRequest, secret_key: str = Header(...)): - """Atomically advance one explicit channel pointer to a registered release. - - ``emergency`` is the beta-only break-glass path: it skips the T2 - qualification gate and requires an explicit compare-and-swap on both the - current release id and generation, so a concurrent normal promotion cannot - be silently clobbered. - """ + """Atomically advance or repoint one explicit qualified channel pointer.""" if secret_key != os.getenv('ADMIN_KEY'): raise HTTPException(status_code=403, detail='You are not authorized to perform this action') try: @@ -1013,7 +1007,7 @@ async def promote_desktop_channel(request: DesktopChannelPromotionRequest, secre request.release_id, expected_generation=request.expected_generation, expected_current_release_id=request.expected_current_release_id, - emergency=request.emergency, + operation=request.operation, ) except ValueError as exc: raise HTTPException(status_code=409, detail=str(exc)) from exc @@ -1023,22 +1017,3 @@ async def promote_desktop_channel(request: DesktopChannelPromotionRequest, secre live_cache_key(request.platform, request.channel), ) return {"success": True, "pointer": pointer} - - -@router.post("/v2/desktop/channels/rollback") -async def rollback_macos_beta_channel_endpoint(request: DesktopBetaRollbackRequest, secret_key: str = Header(...)): - """Emergency-only, compare-and-swap rollback for the macOS beta pointer.""" - if secret_key != os.getenv('ADMIN_KEY'): - raise HTTPException(status_code=403, detail='You are not authorized to perform this action') - try: - result = await run_blocking( - db_executor, - rollback_macos_beta_channel, - request.release_id, - expected_current_release_id=request.expected_current_release_id, - expected_generation=request.expected_generation, - ) - except ValueError as exc: - raise HTTPException(status_code=409, detail=str(exc)) from exc - await run_blocking(db_executor, delete_generic_cache, live_cache_key("macos", "beta")) - return {"success": True, **result} diff --git a/backend/scripts/deploy_status_report.py b/backend/scripts/deploy_status_report.py index 2930daa9c23..fb018bb01e0 100755 --- a/backend/scripts/deploy_status_report.py +++ b/backend/scripts/deploy_status_report.py @@ -46,7 +46,7 @@ class CloudRunFetchError: def main() -> int: parser = argparse.ArgumentParser(description='Read-only deploy rollout/status reporter for Omi services.') - parser.add_argument('--env', choices=('dev', 'beta', 'prod'), required=True) + parser.add_argument('--env', choices=('dev', 'prod'), required=True) parser.add_argument('--project', help='GCP project id for live Cloud Run reads.') parser.add_argument('--region', default=DEFAULT_REGION) parser.add_argument('--namespace', help='Kubernetes namespace. Defaults to -omi-backend.') diff --git a/backend/scripts/preflight-cloud-run-deploy.py b/backend/scripts/preflight-cloud-run-deploy.py index 0869bd5c6d2..b40d29a1bd1 100755 --- a/backend/scripts/preflight-cloud-run-deploy.py +++ b/backend/scripts/preflight-cloud-run-deploy.py @@ -38,7 +38,7 @@ class RuntimeBindingContract: def main() -> int: parser = argparse.ArgumentParser(description='Preflight checks for backend Cloud Run deploys.') - parser.add_argument('--env', choices=('dev', 'beta', 'prod'), required=True) + parser.add_argument('--env', choices=('dev', 'prod'), required=True) parser.add_argument('--project', required=True) parser.add_argument('--region', default=DEFAULT_REGION) parser.add_argument('--manifest', type=Path, default=DEFAULT_MANIFEST) diff --git a/backend/scripts/probe-transcription-candidate-from-cloud-run.sh b/backend/scripts/probe-transcription-candidate-from-cloud-run.sh new file mode 100755 index 00000000000..f093e499c91 --- /dev/null +++ b/backend/scripts/probe-transcription-candidate-from-cloud-run.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# Launch the smallest VPC-connected Cloud Run Job that can reach an internal +# tagged backend candidate. The job gets only run.invoker on that one service; +# all temporary IAM and job state is removed on every exit path. +set -euo pipefail + +PROJECT="" +REGION="" +IMAGE="" +CANDIDATE_URL="" +IDENTITY_AUDIENCE="" +NETWORK="" +SUBNET="" +FIREBASE_TOKEN_FILE="" +NAME_SUFFIX="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --project) PROJECT="$2"; shift 2 ;; + --region) REGION="$2"; shift 2 ;; + --image) IMAGE="$2"; shift 2 ;; + --candidate-url) CANDIDATE_URL="$2"; shift 2 ;; + --identity-audience) IDENTITY_AUDIENCE="$2"; shift 2 ;; + --network) NETWORK="$2"; shift 2 ;; + --subnet) SUBNET="$2"; shift 2 ;; + --firebase-token-file) FIREBASE_TOKEN_FILE="$2"; shift 2 ;; + --name-suffix) NAME_SUFFIX="$2"; shift 2 ;; + *) echo "ERROR: unknown argument: $1" >&2; exit 2 ;; + esac +done + +for required in PROJECT REGION IMAGE CANDIDATE_URL IDENTITY_AUDIENCE NETWORK SUBNET FIREBASE_TOKEN_FILE NAME_SUFFIX; do + [[ -n "${!required}" ]] || { echo "ERROR: --${required,,} is required" >&2; exit 2; } +done +[[ "$NAME_SUFFIX" =~ ^[a-z0-9-]+$ ]] || { echo 'ERROR: --name-suffix must contain lowercase letters, digits, and hyphens' >&2; exit 2; } +[[ -f "$FIREBASE_TOKEN_FILE" ]] || { echo 'ERROR: Firebase token file is missing' >&2; exit 2; } + +# The tagged candidate has a public run.app hostname. With all traffic routed +# through the selected VPC subnet, Private Google Access is the required +# private egress path; fail before creating any ephemeral probe resource. +PRIVATE_GOOGLE_ACCESS="$(gcloud compute networks subnets describe "$SUBNET" --project="$PROJECT" --region="$REGION" \ + --format='value(privateIpGoogleAccess)')" +[[ "$PRIVATE_GOOGLE_ACCESS" == "True" ]] || { + echo "ERROR: subnet $SUBNET must enable Private Google Access for all-traffic Cloud Run probe egress" >&2 + exit 1 +} + +NAME_TOKEN="$(python3 -c 'import hashlib, sys; print(hashlib.sha256(sys.argv[1].encode()).hexdigest()[:20])' "$NAME_SUFFIX")" +JOB_NAME="backend-candidate-vpc-probe-${NAME_TOKEN}" +SERVICE_ACCOUNT_NAME="bcp-${NAME_TOKEN}" +[[ ${#SERVICE_ACCOUNT_NAME} -le 30 ]] || { echo 'ERROR: derived service account ID exceeds 30 characters' >&2; exit 2; } +SERVICE_ACCOUNT="${SERVICE_ACCOUNT_NAME}@${PROJECT}.iam.gserviceaccount.com" +FIREBASE_TOKEN="$(<"$FIREBASE_TOKEN_FILE")" +[[ -n "$FIREBASE_TOKEN" ]] || { echo 'ERROR: Firebase token is empty' >&2; exit 2; } + +cleanup() { + FIREBASE_TOKEN="" + gcloud run jobs delete "$JOB_NAME" --project="$PROJECT" --region="$REGION" --quiet >/dev/null 2>&1 || true + gcloud run services remove-iam-policy-binding backend --project="$PROJECT" --region="$REGION" \ + --member="serviceAccount:${SERVICE_ACCOUNT}" --role=roles/run.invoker --quiet >/dev/null 2>&1 || true + gcloud iam service-accounts delete "$SERVICE_ACCOUNT" --project="$PROJECT" --quiet >/dev/null 2>&1 || true +} +trap cleanup EXIT + +gcloud iam service-accounts create "$SERVICE_ACCOUNT_NAME" --project="$PROJECT" \ + --display-name="Ephemeral backend candidate probe ${NAME_SUFFIX}" --quiet +gcloud run services add-iam-policy-binding backend --project="$PROJECT" --region="$REGION" \ + --member="serviceAccount:${SERVICE_ACCOUNT}" --role=roles/run.invoker --quiet + +gcloud run jobs deploy "$JOB_NAME" --project="$PROJECT" --region="$REGION" --image="$IMAGE" \ + --service-account="$SERVICE_ACCOUNT" --network="$NETWORK" --subnet="$SUBNET" --vpc-egress=all-traffic \ + --set-env-vars="CANDIDATE_API_URL=${CANDIDATE_URL},CLOUD_RUN_IDENTITY_AUDIENCE=${IDENTITY_AUDIENCE},FIREBASE_PROBE_TOKEN=${FIREBASE_TOKEN}" \ + --command=python --args=scripts/run_vpc_transcription_candidate_probe.py --task-timeout=120s --max-retries=0 --quiet +FIREBASE_TOKEN="" +gcloud run jobs execute "$JOB_NAME" --project="$PROJECT" --region="$REGION" --wait --quiet diff --git a/backend/scripts/release_ring_gke_snapshot.py b/backend/scripts/release_ring_gke_snapshot.py deleted file mode 100644 index 4d277e8a844..00000000000 --- a/backend/scripts/release_ring_gke_snapshot.py +++ /dev/null @@ -1,318 +0,0 @@ -#!/usr/bin/env python3 -"""Capture, restore, and verify the release ring's pre-mutation GKE state.""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import subprocess -import sys -from datetime import UTC, datetime -from pathlib import Path -from typing import Any, Callable, Mapping, Sequence, cast - -SCHEMA_VERSION = 1 - -CommandRunner = Callable[[Sequence[str], str | None], str] - - -def _run(command: Sequence[str], input_text: str | None = None) -> str: - completed = subprocess.run(command, check=True, capture_output=True, text=True, input=input_text) - return completed.stdout - - -def _is_not_found(exc: subprocess.CalledProcessError) -> bool: - evidence = f"{getattr(exc, 'stderr', '')} {getattr(exc, 'output', '')}".lower() - return 'not found' in evidence or 'notfound' in evidence or 'release: not found' in evidence - - -def _load_json(text: str, *, label: str) -> Any: - try: - return json.loads(text) - except json.JSONDecodeError as exc: - raise ValueError(f'{label} returned invalid JSON') from exc - - -def _canonical_json(value: Mapping[str, Any]) -> str: - return json.dumps(value, sort_keys=True, separators=(',', ':')) - - -def _sha256_text(value: str) -> str: - return hashlib.sha256(value.encode('utf-8')).hexdigest() - - -def _sanitize_config_map(value: object, *, namespace: str, name: str) -> dict[str, Any]: - if not isinstance(value, Mapping): - raise ValueError(f'{name}: ConfigMap is not an object') - metadata = value.get('metadata') - if not isinstance(metadata, Mapping) or metadata.get('name') != name: - raise ValueError(f'{name}: ConfigMap identity does not match the requested object') - observed_namespace = metadata.get('namespace') - if observed_namespace not in (None, namespace): - raise ValueError(f'{name}: ConfigMap namespace does not match {namespace}') - sanitized: dict[str, Any] = { - 'apiVersion': 'v1', - 'kind': 'ConfigMap', - 'metadata': {'name': name, 'namespace': namespace}, - 'data': {}, - 'binaryData': {}, - } - for field in ('data', 'binaryData'): - raw = value.get(field) - if raw is None: - continue - if not isinstance(raw, Mapping) or any( - not isinstance(key, str) or not isinstance(item, str) for key, item in raw.items() - ): - raise ValueError(f'{name}: ConfigMap {field} must contain string values') - sanitized[field] = dict(sorted(cast(Mapping[str, str], raw).items())) - return sanitized - - -def _capture_config_map(*, namespace: str, name: str, runner: CommandRunner) -> dict[str, Any]: - command = ['kubectl', '-n', namespace, 'get', 'configmap', name, '-o', 'json'] - try: - raw = runner(command, None) - except subprocess.CalledProcessError as exc: - if _is_not_found(exc): - return {'name': name, 'present': False} - raise - config_map = _sanitize_config_map(_load_json(raw, label=f'{name} ConfigMap'), namespace=namespace, name=name) - return { - 'name': name, - 'present': True, - 'sha256': _sha256_text(_canonical_json(config_map)), - 'object': config_map, - } - - -def _capture_helm_release(*, namespace: str, name: str, runner: CommandRunner) -> dict[str, Any]: - try: - history_text = runner(['helm', '-n', namespace, 'history', name, '--output', 'json'], None) - except subprocess.CalledProcessError as exc: - if _is_not_found(exc): - return {'name': name, 'present': False} - raise - history = _load_json(history_text, label=f'{name} Helm history') - if not isinstance(history, list): - raise ValueError(f'{name}: Helm history is not a list') - deployed_revisions: list[int] = [] - for entry in history: - if not isinstance(entry, Mapping) or entry.get('status') != 'deployed': - continue - raw_revision = entry.get('revision') - try: - revision = int(raw_revision) - except (TypeError, ValueError): - raise ValueError(f'{name}: deployed Helm revision is invalid') from None - if revision <= 0: - raise ValueError(f'{name}: deployed Helm revision is invalid') - deployed_revisions.append(revision) - if not deployed_revisions: - raise ValueError(f'{name}: Helm release exists without a deployed revision') - revision = max(deployed_revisions) - manifest = runner(['helm', '-n', namespace, 'get', 'manifest', name, '--revision', str(revision)], None) - if not manifest.strip(): - raise ValueError(f'{name}: deployed Helm manifest is empty') - return { - 'name': name, - 'present': True, - 'revision': revision, - 'manifest_sha256': _sha256_text(manifest), - } - - -def capture_snapshot( - *, - namespace: str, - config_map_name: str, - releases: Mapping[str, str], - runner: CommandRunner = _run, -) -> dict[str, Any]: - if not namespace or not config_map_name: - raise ValueError('namespace and ConfigMap name are required') - if not releases or any(not component or not name for component, name in releases.items()): - raise ValueError('at least one named Helm release is required') - return { - 'schema_version': SCHEMA_VERSION, - 'scope': 'release ring GKE pre-mutation state', - 'captured_at': datetime.now(UTC).isoformat(), - 'namespace': namespace, - 'config_map': _capture_config_map(namespace=namespace, name=config_map_name, runner=runner), - 'helm_releases': { - component: _capture_helm_release(namespace=namespace, name=name, runner=runner) - for component, name in releases.items() - }, - } - - -def _restore_config_map(snapshot: Mapping[str, Any], *, namespace: str, runner: CommandRunner) -> dict[str, Any]: - name = snapshot.get('name') - if not isinstance(name, str) or not name: - raise ValueError('GKE snapshot contains an invalid ConfigMap name') - if snapshot.get('present') is True: - raw_object = snapshot.get('object') - config_map = _sanitize_config_map(raw_object, namespace=namespace, name=name) - expected_sha = snapshot.get('sha256') - if expected_sha != _sha256_text(_canonical_json(config_map)): - raise ValueError(f'{name}: ConfigMap snapshot digest does not match its object') - runner(['kubectl', '-n', namespace, 'apply', '-f', '-'], _canonical_json(config_map)) - observed = runner(['kubectl', '-n', namespace, 'get', 'configmap', name, '-o', 'json'], None) - observed_map = _sanitize_config_map( - _load_json(observed, label=f'{name} restored ConfigMap'), namespace=namespace, name=name - ) - if _sha256_text(_canonical_json(observed_map)) != expected_sha: - raise ValueError(f'{name}: observed ConfigMap does not match the snapshot') - return {'result': 'restored', 'sha256': expected_sha} - - runner(['kubectl', '-n', namespace, 'delete', 'configmap', name, '--ignore-not-found=true'], None) - try: - runner(['kubectl', '-n', namespace, 'get', 'configmap', name, '-o', 'json'], None) - except subprocess.CalledProcessError as exc: - if _is_not_found(exc): - return {'result': 'deleted'} - raise - raise ValueError(f'{name}: ConfigMap still exists after delete') - - -def _restore_helm_release(snapshot: Mapping[str, Any], *, namespace: str, runner: CommandRunner) -> dict[str, Any]: - name = snapshot.get('name') - if not isinstance(name, str) or not name: - raise ValueError('GKE snapshot contains an invalid Helm release name') - if snapshot.get('present') is True: - revision = snapshot.get('revision') - digest = snapshot.get('manifest_sha256') - if not isinstance(revision, int) or revision <= 0 or not isinstance(digest, str) or len(digest) != 64: - raise ValueError(f'{name}: Helm snapshot is incomplete') - runner( - ['helm', '-n', namespace, 'rollback', name, str(revision), '--wait', '--timeout', '1800s'], - None, - ) - observed_manifest = runner(['helm', '-n', namespace, 'get', 'manifest', name], None) - if _sha256_text(observed_manifest) != digest: - raise ValueError(f'{name}: observed Helm manifest does not match the snapshot') - return {'result': 'restored', 'revision': revision, 'manifest_sha256': digest} - - try: - runner(['helm', '-n', namespace, 'status', name, '--output', 'json'], None) - except subprocess.CalledProcessError as exc: - if _is_not_found(exc): - return {'result': 'absent'} - raise - runner(['helm', '-n', namespace, 'uninstall', name, '--wait', '--timeout', '1800s'], None) - try: - runner(['helm', '-n', namespace, 'status', name, '--output', 'json'], None) - except subprocess.CalledProcessError as exc: - if _is_not_found(exc): - return {'result': 'deleted'} - raise - raise ValueError(f'{name}: Helm release still exists after uninstall') - - -def restore_snapshot(snapshot: Mapping[str, Any], *, runner: CommandRunner = _run) -> dict[str, Any]: - if snapshot.get('schema_version') != SCHEMA_VERSION: - raise ValueError('GKE snapshot has an unsupported schema version') - namespace = snapshot.get('namespace') - config_map = snapshot.get('config_map') - releases = snapshot.get('helm_releases') - if not isinstance(namespace, str) or not namespace: - raise ValueError('GKE snapshot is missing namespace') - if not isinstance(config_map, Mapping) or not isinstance(releases, Mapping): - raise ValueError('GKE snapshot is incomplete') - - results: dict[str, dict[str, Any]] = {} - - def attempt(component: str, operation: Callable[[], dict[str, Any]]) -> None: - try: - results[component] = operation() - except (OSError, ValueError, subprocess.CalledProcessError) as exc: - if isinstance(exc, subprocess.CalledProcessError): - error = f'command exited with code {exc.returncode}' - else: - error = str(exc) - results[component] = {'result': 'failed', 'error': error} - - secret_snapshot = releases.get('backend-secrets') - if isinstance(secret_snapshot, Mapping): - attempt( - 'backend-secrets', - lambda: _restore_helm_release(secret_snapshot, namespace=namespace, runner=runner), - ) - attempt('backend-config', lambda: _restore_config_map(config_map, namespace=namespace, runner=runner)) - for component, raw_release in releases.items(): - if component == 'backend-secrets': - continue - if not isinstance(component, str) or not isinstance(raw_release, Mapping): - results[str(component)] = {'result': 'failed', 'error': 'invalid Helm release snapshot'} - continue - attempt( - component, - lambda raw_release=raw_release: _restore_helm_release(raw_release, namespace=namespace, runner=runner), - ) - - failures = sorted(component for component, result in results.items() if result.get('result') == 'failed') - return { - 'schema_version': SCHEMA_VERSION, - 'scope': 'release ring GKE restoration', - 'result': 'pass' if not failures else 'fail', - 'failed_components': failures, - 'components': results, - } - - -def _parse_releases(values: Sequence[str]) -> dict[str, str]: - releases: dict[str, str] = {} - for value in values: - component, separator, name = value.partition('=') - if not separator or not component or not name or component in releases: - raise ValueError(f'--release must be unique COMPONENT=NAME: {value!r}') - releases[component] = name - return releases - - -def _read_snapshot(path: Path) -> dict[str, Any]: - loaded = _load_json(path.read_text(encoding='utf-8'), label=str(path)) - if not isinstance(loaded, dict): - raise ValueError('GKE snapshot must be an object') - return cast(dict[str, Any], loaded) - - -def _write_json(path: Path, value: Mapping[str, Any]) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(f'{json.dumps(value, indent=2, sort_keys=True)}\n', encoding='utf-8') - - -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__) - commands = parser.add_subparsers(dest='command', required=True) - capture = commands.add_parser('capture') - capture.add_argument('--namespace', required=True) - capture.add_argument('--config-map', required=True) - capture.add_argument('--release', action='append', default=[]) - capture.add_argument('--output', type=Path, required=True) - restore = commands.add_parser('restore') - restore.add_argument('--snapshot', type=Path, required=True) - restore.add_argument('--evidence-path', type=Path, required=True) - args = parser.parse_args() - try: - if args.command == 'capture': - releases = _parse_releases(args.release) - snapshot = capture_snapshot( - namespace=args.namespace, - config_map_name=args.config_map, - releases=releases, - ) - _write_json(args.output, snapshot) - return 0 - evidence = restore_snapshot(_read_snapshot(args.snapshot)) - _write_json(args.evidence_path, evidence) - print(json.dumps(evidence, indent=2, sort_keys=True)) - return 0 if evidence['result'] == 'pass' else 1 - except (OSError, ValueError, subprocess.CalledProcessError) as exc: - print(f'ERROR: {exc}', file=sys.stderr) - return 1 - - -if __name__ == '__main__': - raise SystemExit(main()) diff --git a/backend/scripts/release_rings.py b/backend/scripts/release_rings.py deleted file mode 100644 index 1fee183bdc0..00000000000 --- a/backend/scripts/release_rings.py +++ /dev/null @@ -1,510 +0,0 @@ -#!/usr/bin/env python3 -"""Create and validate immutable inputs for stateless release-ring deployments. - -This module deliberately owns data shapes only. GCS provides immutable object -storage/CAS and GitHub Actions provides per-ring serialization; this code must -not grow into a release coordinator or reconciliation service. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import re -import subprocess -import sys -from datetime import UTC, datetime -from pathlib import Path -from typing import Any, Iterable, Mapping - -SCHEMA_VERSION = 1 -RINGS = frozenset({"beta", "prod"}) -# These are the images the release builder creates. Config-only resources are -# deliberately separate so a record can deploy the ConfigMap and ExternalSecret -# that the stateless workloads consume, without inventing a fake image for them. -COMPONENTS = ("backend", "backend-listen", "pusher", "llm-gateway", "agent-proxy") -CONFIG_COMPONENTS = COMPONENTS + ("backend-config", "backend-secrets") -CLOUD_RUN_SERVICES = ("backend", "backend-sync", "backend-sync-backfill", "backend-integration") -RECEIPT_STATES = frozenset({"started", "verified", "restored", "partial_mutation"}) -_DIGEST_RE = re.compile(r"^[^@\s]+@sha256:[0-9a-f]{64}$") -_SHA_RE = re.compile(r"^[0-9a-f]{40}$") -_SECRET_VERSION_RE = re.compile(r"^projects/[^/]+/secrets/[^/]+/versions/[1-9][0-9]*$") -_OBJECT_RE = re.compile(r"^gs://[^/]+/.+#sha256:[0-9a-f]{64}$") -_RELEASE_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{2,127}$") - - -def canonical_json(value: Mapping[str, Any]) -> str: - return json.dumps(value, indent=2, sort_keys=True) + "\n" - - -def sha256_file(path: Path) -> str: - digest = hashlib.sha256() - with path.open("rb") as handle: - for block in iter(lambda: handle.read(1024 * 1024), b""): - digest.update(block) - return digest.hexdigest() - - -def parse_assignments(values: Iterable[str], *, label: str) -> dict[str, str]: - parsed: dict[str, str] = {} - for value in values: - name, separator, item = value.partition("=") - if not separator or not name or not item: - raise ValueError(f"{label} must use NAME=VALUE: {value!r}") - if name in parsed: - raise ValueError(f"{label} repeats {name!r}") - parsed[name] = item - return parsed - - -def build_record( - *, - release_id: str, - git_sha: str, - eligibility_run_id: str, - images: Mapping[str, str], - rendered_config: Mapping[str, str], - secret_versions: Mapping[str, str], - topology: Mapping[str, Any], - created_at: str | None = None, -) -> dict[str, Any]: - record = { - "schema_version": SCHEMA_VERSION, - "release_id": release_id, - "git_sha": git_sha, - "eligibility_run_id": eligibility_run_id, - "images": dict(sorted(images.items())), - "rendered_config": dict(sorted(rendered_config.items())), - "secret_versions": dict(sorted(secret_versions.items())), - "topology": topology, - "created_at": created_at or datetime.now(UTC).isoformat(), - } - errors = validate_record(record) - if errors: - raise ValueError("invalid release record: " + "; ".join(errors)) - return record - - -def validate_record(record: object) -> list[str]: - if not isinstance(record, Mapping): - return ["record must be an object"] - errors: list[str] = [] - if record.get("schema_version") != SCHEMA_VERSION: - errors.append(f"schema_version must be {SCHEMA_VERSION}") - release_id = record.get("release_id") - if not isinstance(release_id, str) or not _RELEASE_ID_RE.fullmatch(release_id): - errors.append("release_id must be a durable identifier") - git_sha = record.get("git_sha") - if not isinstance(git_sha, str) or not _SHA_RE.fullmatch(git_sha): - errors.append("git_sha must be a lowercase full commit SHA") - run_id = record.get("eligibility_run_id") - if not isinstance(run_id, str) or not run_id.isdecimal() or int(run_id) <= 0: - errors.append("eligibility_run_id must be a positive decimal GitHub run ID") - - images = record.get("images") - if not isinstance(images, Mapping): - errors.append("images must be an object") - else: - for component in COMPONENTS: - image = images.get(component) - if not isinstance(image, str) or not _DIGEST_RE.fullmatch(image): - errors.append(f"images.{component} must be an immutable OCI digest") - unexpected = sorted(str(name) for name in images if name not in COMPONENTS) - if unexpected: - errors.append(f"images contains unsupported components: {', '.join(unexpected)}") - - config = record.get("rendered_config") - required_config = {f"{ring}/{component}" for ring in RINGS for component in CONFIG_COMPONENTS} - if not isinstance(config, Mapping): - errors.append("rendered_config must be an object") - else: - missing = sorted(required_config - set(config)) - if missing: - errors.append(f"rendered_config is missing: {', '.join(missing)}") - extra = sorted(str(name) for name in config if name not in required_config) - if extra: - errors.append(f"rendered_config has unsupported entries: {', '.join(extra)}") - for name, reference in config.items(): - if not isinstance(reference, str) or not _OBJECT_RE.fullmatch(reference): - errors.append(f"rendered_config.{name} must be an immutable GCS object reference") - - secrets = record.get("secret_versions") - if not isinstance(secrets, Mapping) or not secrets: - errors.append("secret_versions must be a non-empty object") - else: - for name, reference in secrets.items(): - if not isinstance(name, str) or not name: - errors.append("secret_versions contains an empty name") - if not isinstance(reference, str) or not _SECRET_VERSION_RE.fullmatch(reference): - errors.append(f"secret_versions.{name} must pin a numeric Secret Manager version") - - topology = record.get("topology") - if not isinstance(topology, Mapping): - errors.append("topology must be an object") - else: - prod_topology = topology.get("prod") - prod_services = prod_topology.get("cloud_run_services") if isinstance(prod_topology, Mapping) else None - for ring in RINGS: - ring_topology = topology.get(ring) - if not isinstance(ring_topology, Mapping): - errors.append(f"topology.{ring} must be an object") - continue - namespace = ring_topology.get("namespace") - if not isinstance(namespace, str) or namespace != f"{ring}-omi-backend": - errors.append(f"topology.{ring}.namespace must be {ring}-omi-backend") - services = ring_topology.get("cloud_run_services") - if not isinstance(services, Mapping): - errors.append(f"topology.{ring}.cloud_run_services must be an object") - continue - for service in CLOUD_RUN_SERVICES: - name = services.get(service) - if not isinstance(name, str) or not name: - errors.append(f"topology.{ring}.cloud_run_services.{service} must be a non-empty name") - if ring == "beta" and services == prod_services: - errors.append("topology.beta.cloud_run_services must not reuse prod service identities") - - if _contains_mutable_reference(record): - errors.append("release records must not contain mutable 'latest' references") - return errors - - -def _contains_mutable_reference(value: object) -> bool: - if isinstance(value, str): - return value.lower() == "latest" or ":latest" in value.lower() or "/versions/latest" in value.lower() - if isinstance(value, Mapping): - return any(_contains_mutable_reference(item) for item in value.values()) - if isinstance(value, list): - return any(_contains_mutable_reference(item) for item in value) - return False - - -def build_active_pointer( - *, - ring: str, - release_id: str, - existing: Mapping[str, Any] | None, - hold: bool = False, - updated_at: str | None = None, -) -> dict[str, Any]: - if ring not in RINGS: - raise ValueError(f"unknown ring {ring!r}") - if not _RELEASE_ID_RE.fullmatch(release_id): - raise ValueError("release_id must be a durable identifier") - current = existing.get("current_release_id") if isinstance(existing, Mapping) else None - held = set(existing.get("held_release_ids", []) if isinstance(existing, Mapping) else []) - if hold: - held.add(release_id) - next_current = current if isinstance(current, str) else None - previous = existing.get("previous_verified_release_id") if isinstance(existing, Mapping) else None - else: - if release_id in held: - raise ValueError(f"held release {release_id!r} cannot be promoted") - next_current = release_id - previous = current if isinstance(current, str) and current != release_id else None - return { - "schema_version": SCHEMA_VERSION, - "ring": ring, - "current_release_id": next_current, - "previous_verified_release_id": previous, - "held_release_ids": sorted(held), - "updated_at": updated_at or datetime.now(UTC).isoformat(), - } - - -def build_receipt( - *, - ring: str, - release_id: str, - run_id: str, - state: str, - snapshot_reference: str, - components: Mapping[str, str], - created_at: str | None = None, -) -> dict[str, Any]: - if ring not in RINGS: - raise ValueError(f"unknown ring {ring!r}") - if state not in RECEIPT_STATES: - raise ValueError(f"unsupported receipt state {state!r}") - if not run_id.isdecimal() or int(run_id) <= 0: - raise ValueError("run_id must be a positive decimal GitHub run ID") - if not _OBJECT_RE.fullmatch(snapshot_reference): - raise ValueError("snapshot_reference must be an immutable GCS object reference") - if not components or any(not name or not value for name, value in components.items()): - raise ValueError("components must contain non-empty observed deployment details") - return { - "schema_version": SCHEMA_VERSION, - "ring": ring, - "release_id": release_id, - "run_id": run_id, - "state": state, - "snapshot_reference": snapshot_reference, - "components": dict(sorted(components.items())), - "created_at": created_at or datetime.now(UTC).isoformat(), - } - - -def resolve_secret_versions(*, manifests: Iterable[Path], project: str) -> dict[str, str]: - """Resolve every manifest Secret Manager reference to an immutable version. - - PyYAML is intentionally imported only for this release-build command. The - normal record validation path remains stdlib-only for CI and incident use. - """ - try: - import yaml - except ImportError as exc: # pragma: no cover - exercised in Actions - raise ValueError("resolve-secrets requires PyYAML") from exc - names: set[str] = set() - for manifest in manifests: - loaded = yaml.safe_load(manifest.read_text(encoding="utf-8")) - names.update(_secret_names(loaded)) - resolved: dict[str, str] = {} - for name in names: - command = [ - "gcloud", - "secrets", - "versions", - "describe", - "latest", - f"--secret={name}", - f"--project={project}", - "--format=value(name)", - ] - completed = subprocess.run(command, check=True, capture_output=True, text=True) - reference = completed.stdout.strip() - if not _SECRET_VERSION_RE.fullmatch(reference): - raise ValueError(f"{name}: gcloud did not resolve a numeric secret version") - resolved[name] = reference - return resolved - - -def materialize_secret_versions(*, manifest: Path, secret_versions: Mapping[str, str]) -> dict[str, Any]: - """Replace every mutable manifest binding with the record's numeric version.""" - try: - import yaml - except ImportError as exc: # pragma: no cover - exercised in Actions - raise ValueError("materialize-secrets requires PyYAML") from exc - loaded = yaml.safe_load(manifest.read_text(encoding="utf-8")) - if not isinstance(loaded, dict): - raise ValueError("runtime manifest must be a YAML object") - - def rewrite(value: object) -> object: - if isinstance(value, list): - return [rewrite(item) for item in value] - if not isinstance(value, Mapping): - return value - result = {str(name): rewrite(item) for name, item in value.items()} - secret = result.get("secret") or result.get("remoteKey") - version = result.get("version") - if isinstance(secret, str) and (version is None or version == "latest"): - reference = secret_versions.get(secret) - if not isinstance(reference, str) or not _SECRET_VERSION_RE.fullmatch(reference): - raise ValueError(f"{secret}: record does not contain a numeric secret version") - result["version"] = reference.rsplit("/", 1)[1] - return result - - materialized = rewrite(loaded) - if not isinstance(materialized, dict): # defensive narrowing for type checkers - raise ValueError("runtime manifest did not materialize to an object") - return materialized - - -def _secret_names(value: object) -> set[str]: - names: set[str] = set() - if isinstance(value, Mapping): - secret = value.get("secret") or value.get("remoteKey") - if isinstance(secret, str): - names.add(secret) - for item in value.values(): - names.update(_secret_names(item)) - elif isinstance(value, list): - for item in value: - names.update(_secret_names(item)) - return names - - -def materialize_runtime_config( - *, manifest: Path, secret_versions: Mapping[str, str], public_values: Mapping[str, str] -) -> dict[str, Any]: - """Produce a deployable runtime manifest with neither `latest` nor `env_var`. - - Public values are captured from the serving ConfigMap (plus the two VPC - settings) while building the record. A deploy or rollback never consults - mutable GitHub variables again. - """ - materialized = materialize_secret_versions(manifest=manifest, secret_versions=secret_versions) - - def rewrite(value: object) -> object: - if isinstance(value, list): - return [rewrite(item) for item in value] - if not isinstance(value, Mapping): - return value - result = {str(name): rewrite(item) for name, item in value.items()} - env_var = result.get("env_var") - if isinstance(env_var, str): - replacement = public_values.get(env_var) - if replacement is None: - replacement = result.get("default") - if replacement is None: - raise ValueError(f"{env_var}: release record has no public value") - result.pop("env_var", None) - result.pop("default", None) - result["value"] = str(replacement) - return result - - rewritten = rewrite(materialized) - if not isinstance(rewritten, dict): # pragma: no cover - defended above - raise ValueError("runtime manifest did not materialize to an object") - return rewritten - - -def _read_json(path: Path) -> dict[str, Any]: - loaded = json.loads(path.read_text(encoding="utf-8")) - if not isinstance(loaded, dict): - raise ValueError(f"{path} must contain a JSON object") - return loaded - - -def _write_json(path: Path | None, value: Mapping[str, Any]) -> None: - rendered = canonical_json(value) - if path is None: - print(rendered, end="") - else: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(rendered, encoding="utf-8") - - -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__) - commands = parser.add_subparsers(dest="command", required=True) - - create = commands.add_parser("create-record", help="create a deployable immutable release record") - create.add_argument("--release-id", required=True) - create.add_argument("--git-sha", required=True) - create.add_argument("--eligibility-run-id", required=True) - create.add_argument("--image", action="append", default=[]) - create.add_argument("--rendered-config", action="append", default=[]) - create.add_argument("--secret-versions-file", type=Path, required=True) - create.add_argument("--topology-file", type=Path, required=True) - create.add_argument("--created-at") - create.add_argument("--output", type=Path, required=True) - - validate = commands.add_parser("validate-record", help="fail closed on an invalid release record") - validate.add_argument("--record", type=Path, required=True) - - pointer = commands.add_parser("active-pointer", help="build a CAS-ready ring active pointer") - pointer.add_argument("--ring", choices=sorted(RINGS), required=True) - pointer.add_argument("--release-id", required=True) - pointer.add_argument("--existing", type=Path) - pointer.add_argument("--hold", action="store_true") - pointer.add_argument("--output", type=Path, required=True) - - receipt = commands.add_parser("receipt", help="build an append-only ring deployment receipt") - receipt.add_argument("--ring", choices=sorted(RINGS), required=True) - receipt.add_argument("--release-id", required=True) - receipt.add_argument("--run-id", required=True) - receipt.add_argument("--state", choices=sorted(RECEIPT_STATES), required=True) - receipt.add_argument("--snapshot-reference", required=True) - receipt.add_argument("--component", action="append", default=[]) - receipt.add_argument("--output", type=Path, required=True) - - secrets = commands.add_parser("resolve-secrets", help="resolve manifest secret names to numeric versions") - secrets.add_argument("--manifest", type=Path, action="append", required=True) - secrets.add_argument("--project", required=True) - secrets.add_argument("--output", type=Path, required=True) - - materialize = commands.add_parser( - "materialize-secrets", help="write a runtime manifest with record-pinned versions" - ) - materialize.add_argument("--manifest", type=Path, required=True) - materialize.add_argument("--secret-versions-file", type=Path, required=True) - materialize.add_argument("--output", type=Path, required=True) - - runtime = commands.add_parser("materialize-runtime", help="write a fully pinned Cloud Run runtime manifest") - runtime.add_argument("--manifest", type=Path, required=True) - runtime.add_argument("--secret-versions-file", type=Path, required=True) - runtime.add_argument("--public-values-file", type=Path, required=True) - runtime.add_argument("--output", type=Path, required=True) - - args = parser.parse_args() - try: - if args.command == "create-record": - secret_versions = _read_json(args.secret_versions_file) - _write_json( - args.output, - build_record( - release_id=args.release_id, - git_sha=args.git_sha, - eligibility_run_id=args.eligibility_run_id, - images=parse_assignments(args.image, label="--image"), - rendered_config=parse_assignments(args.rendered_config, label="--rendered-config"), - secret_versions={str(name): str(value) for name, value in secret_versions.items()}, - topology=_read_json(args.topology_file), - created_at=args.created_at, - ), - ) - elif args.command == "validate-record": - errors = validate_record(_read_json(args.record)) - if errors: - for error in errors: - print(f"ERROR: {error}", file=sys.stderr) - return 1 - print("release record is valid") - elif args.command == "active-pointer": - _write_json( - args.output, - build_active_pointer( - ring=args.ring, - release_id=args.release_id, - existing=_read_json(args.existing) if args.existing else None, - hold=args.hold, - ), - ) - elif args.command == "receipt": - _write_json( - args.output, - build_receipt( - ring=args.ring, - release_id=args.release_id, - run_id=args.run_id, - state=args.state, - snapshot_reference=args.snapshot_reference, - components=parse_assignments(args.component, label="--component"), - ), - ) - elif args.command == "resolve-secrets": - _write_json(args.output, resolve_secret_versions(manifests=args.manifest, project=args.project)) - elif args.command == "materialize-secrets": - try: - import yaml - except ImportError as exc: # pragma: no cover - exercised in Actions - raise ValueError("materialize-secrets requires PyYAML") from exc - materialized = materialize_secret_versions( - manifest=args.manifest, - secret_versions=_read_json(args.secret_versions_file), - ) - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(yaml.safe_dump(materialized, sort_keys=False), encoding="utf-8") - else: - try: - import yaml - except ImportError as exc: # pragma: no cover - exercised in Actions - raise ValueError("materialize-runtime requires PyYAML") from exc - public_values = _read_json(args.public_values_file) - if not all(isinstance(name, str) and isinstance(value, str) for name, value in public_values.items()): - raise ValueError("public values must be a string-to-string JSON object") - materialized = materialize_runtime_config( - manifest=args.manifest, - secret_versions=_read_json(args.secret_versions_file), - public_values={str(name): str(value) for name, value in public_values.items()}, - ) - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(yaml.safe_dump(materialized, sort_keys=False), encoding="utf-8") - except (OSError, ValueError, json.JSONDecodeError, subprocess.CalledProcessError) as exc: - print(f"ERROR: {exc}", file=sys.stderr) - return 1 - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/backend/scripts/render_backend_runtime_env.py b/backend/scripts/render_backend_runtime_env.py index 09da479af47..8dc1c1e722e 100644 --- a/backend/scripts/render_backend_runtime_env.py +++ b/backend/scripts/render_backend_runtime_env.py @@ -20,7 +20,7 @@ def _as_config_dict(value: object) -> ConfigDict | None: def main() -> int: parser = argparse.ArgumentParser(description='Render backend Cloud Run runtime env from the manifest.') - parser.add_argument('--env', choices=('dev', 'beta', 'prod'), required=True) + parser.add_argument('--env', choices=('dev', 'prod'), required=True) parser.add_argument( '--job', help='render only this Cloud Run job and the shared network flags; services remain full-environment only', diff --git a/backend/scripts/render_release_ring_config.py b/backend/scripts/render_release_ring_config.py deleted file mode 100644 index 6031e502e3d..00000000000 --- a/backend/scripts/render_release_ring_config.py +++ /dev/null @@ -1,85 +0,0 @@ -#!/usr/bin/env python3 -"""Render a ring-specific immutable values/config object from a checked-in base. - -Beta intentionally reuses the production datastore and the existing GPU -endpoints in v1. Only the stateless release names, endpoints, and runtime stage -are rewritten here. GPU routing is deliberately not changed by this renderer; -see #10057 before treating the full transcription path as beta-safe. -""" - -from __future__ import annotations - -import argparse -from pathlib import Path -from typing import Any - -import yaml - -BETA_REPLACEMENTS = ( - ("prod-omi-backend", "beta-omi-backend"), - ("prod-omi", "beta-omi"), - ("pusher.omi.me", "pusher-beta.omi.me"), - ("agent.omi.me", "agent-beta.omi.me"), - ("api.omi.me", "api-beta.omi.me"), - ("prod-pusher-ilb-ip-address", "beta-pusher-ilb-ip-address"), - ("prod-agent-proxy-ip-address", "beta-agent-proxy-ip-address"), - ("agent-proxy-cert", "beta-agent-proxy-cert"), - ("prod-omi-self-hosted-llm-ip-address", "beta-omi-self-hosted-llm-ip-address"), -) - - -def render(value: Any, *, ring: str) -> Any: - if ring == "prod": - return value - if isinstance(value, dict): - return {key: render(item, ring=ring) for key, item in value.items()} - if isinstance(value, list): - return [render(item, ring=ring) for item in value] - if not isinstance(value, str): - return value - if value == "prod": - return "beta" - rendered = value - for old, new in BETA_REPLACEMENTS: - rendered = rendered.replace(old, new) - return rendered - - -def render_manifest(value: Any, *, ring: str) -> Any: - """Render a source-controlled config and make beta a real env lane. - - The source manifest keeps dev/prod as its hand-maintained declarations. - The release record stores the fully materialized beta declaration, so the - beta lane is reviewable and rollbackable without maintaining a second - mutable copy of thousands of runtime settings. - """ - rendered = render(value, ring=ring) - if ring == "beta" and isinstance(rendered, dict): - environments = rendered.get("environments") - if isinstance(environments, dict) and "prod" in environments: - environments["beta"] = environments.pop("prod") - return rendered - - -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--ring", choices=("beta", "prod"), required=True) - parser.add_argument("--source", type=Path, required=True) - parser.add_argument("--output", type=Path, required=True) - args = parser.parse_args() - - try: - source = yaml.safe_load(args.source.read_text(encoding="utf-8")) - if source is None: - raise ValueError("source config is empty") - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text( - yaml.safe_dump(render_manifest(source, ring=args.ring), sort_keys=False), encoding="utf-8" - ) - except (OSError, ValueError, yaml.YAMLError) as exc: - parser.error(str(exc)) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/backend/scripts/run_vpc_transcription_candidate_probe.py b/backend/scripts/run_vpc_transcription_candidate_probe.py new file mode 100755 index 00000000000..e2cc096c5d9 --- /dev/null +++ b/backend/scripts/run_vpc_transcription_candidate_probe.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python3 +"""Run the known-audio candidate gate from a VPC-connected Cloud Run Job. + +The temporary job requests the traffic-tagged candidate, but mints its Cloud +Run identity token for the canonical receiving service audience. The deploy runner supplies the short-lived Firebase token +only as a job environment value; neither credential is emitted in the report. +""" + +from __future__ import annotations + +import os +import urllib.parse +import urllib.request +from pathlib import Path + +from transcription_capability_probe import ProbeConfig, build_report + +METADATA_IDENTITY_URL = "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity" + + +def _required_env(name: str) -> str: + value = os.environ.get(name, "").strip() + if not value: + raise RuntimeError(f"missing {name}") + return value + + +def _identity_token(audience: str) -> str: + request = urllib.request.Request( + f"{METADATA_IDENTITY_URL}?audience={urllib.parse.quote(audience, safe='')}", + headers={"Metadata-Flavor": "Google"}, + ) + with urllib.request.urlopen(request, timeout=20) as response: + token = response.read(8193).decode("utf-8").strip() + if not token or len(token) > 8192: + raise RuntimeError("invalid Cloud Run identity token") + return token + + +def _required_https_url(name: str) -> str: + value = _required_env(name) + parsed = urllib.parse.urlparse(value) + if parsed.scheme != "https" or not parsed.netloc or parsed.params or parsed.query or parsed.fragment: + raise RuntimeError(f"invalid {name}") + return value.rstrip("/") + + +def main() -> int: + candidate_url = _required_https_url("CANDIDATE_API_URL") + identity_audience = _required_https_url("CLOUD_RUN_IDENTITY_AUDIENCE") + firebase_token = _required_env("FIREBASE_PROBE_TOKEN") + identity_token = _identity_token(identity_audience) + report = build_report( + ProbeConfig( + fixture_path=Path("testing/release_fixtures/transcription-release-probe.wav"), + manifest_path=Path("testing/release_fixtures/transcription-release-probe.json"), + api_url=candidate_url, + bearer_token=firebase_token, + cloud_run_identity_token=identity_token, + timeout_seconds=30.0, + ) + ) + firebase_token = "" + identity_token = "" + return 0 if report["status"] == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/scripts/transcription_capability_probe.py b/backend/scripts/transcription_capability_probe.py index 920ba8cbfc3..eafe0ad9bc0 100644 --- a/backend/scripts/transcription_capability_probe.py +++ b/backend/scripts/transcription_capability_probe.py @@ -49,6 +49,7 @@ class ProbeConfig: manifest_path: Path api_url: str | None bearer_token: str | None + cloud_run_identity_token: str | None timeout_seconds: float @@ -114,6 +115,7 @@ def config_from_args(args: argparse.Namespace) -> ProbeConfig: manifest_path=args.manifest_path, api_url=api_url.rstrip('/') if api_url else None, bearer_token=_read_bearer_token(args.bearer_token_file), + cloud_run_identity_token=_read_bearer_token(args.cloud_run_identity_token_file), timeout_seconds=args.timeout_seconds, ) @@ -261,6 +263,11 @@ def _probe_full_route( 'Accept': 'application/json', 'Authorization': f'Bearer {config.bearer_token}', 'Content-Type': content_type, + **( + {'X-Serverless-Authorization': f'Bearer {config.cloud_run_identity_token}'} + if config.cloud_run_identity_token + else {} + ), }, ), config.timeout_seconds, @@ -314,6 +321,7 @@ def parse_args(argv: Sequence[str]) -> argparse.Namespace: parser.add_argument('--fixture-path', type=Path, default=DEFAULT_FIXTURE_PATH) parser.add_argument('--manifest-path', type=Path, default=DEFAULT_MANIFEST_PATH) parser.add_argument('--bearer-token-file', type=Path, required=True) + parser.add_argument('--cloud-run-identity-token-file', type=Path) parser.add_argument('--timeout-seconds', type=float, default=30.0) parser.add_argument('--json-only', action='store_true', help='Emit only the redacted machine-readable report.') return parser.parse_args(argv) diff --git a/backend/scripts/validate-backend-runtime-env.py b/backend/scripts/validate-backend-runtime-env.py index 288406c36e8..33c9ffd5969 100755 --- a/backend/scripts/validate-backend-runtime-env.py +++ b/backend/scripts/validate-backend-runtime-env.py @@ -68,8 +68,8 @@ def _as_config_list(value: object) -> list[Any] | None: def main() -> int: - parser = argparse.ArgumentParser(description='Validate runtime manifests against GKE and Cloud Run state.') - parser.add_argument('--env', choices=('dev', 'beta', 'prod'), required=True) + parser = argparse.ArgumentParser(description='Validate backend runtime env manifests against GKE and Cloud Run.') + parser.add_argument('--env', choices=('dev', 'prod'), required=True) parser.add_argument('--manifest', type=Path, default=DEFAULT_MANIFEST) parser.add_argument( '--cloud-run-state', @@ -431,7 +431,7 @@ def _validate_prerecorded_stt_contract(env: str, env_config: ConfigDict) -> list cloud_run = _as_config_dict(env_config.get('cloud_run')) or {} cloud_run_services = _as_config_dict(cloud_run.get('services')) or {} required_cloud_run_scopes: set[str] = set() - if env in {'dev', 'beta', 'prod'}: + if env in {'dev', 'prod'}: for service in ('backend', 'backend-sync', 'backend-integration'): if service not in cloud_run_services: continue diff --git a/backend/scripts/verify_pusher_config_references.py b/backend/scripts/verify_pusher_config_references.py index 2a703ea7231..c513df548e3 100644 --- a/backend/scripts/verify_pusher_config_references.py +++ b/backend/scripts/verify_pusher_config_references.py @@ -16,7 +16,7 @@ import yaml ROOT = Path(__file__).resolve().parents[2] -ENVIRONMENTS = {"beta", "dev", "prod"} +ENVIRONMENTS = {"dev", "prod"} RUNTIME_ENV_MANIFEST = ROOT / "backend/deploy/runtime_env.yaml" Reference = tuple[str, str, str | None] Binding = tuple[str, str, str] @@ -55,13 +55,9 @@ def walk(item: Any) -> None: return found -def render( - environment: str, - image_tag: str = "contract-test", - values_file: Path | None = None, -) -> list[dict[str, Any]]: +def render(environment: str, image_tag: str = "contract-test") -> list[dict[str, Any]]: chart = ROOT / "backend/charts/pusher" - values = values_file or chart / f"{environment}_omi_pusher_values.yaml" + values = chart / f"{environment}_omi_pusher_values.yaml" result = subprocess.run( [ "helm", @@ -82,22 +78,18 @@ def render( return [doc for doc in yaml.safe_load_all(result.stdout) if isinstance(doc, dict)] -def rendered_pusher_deployment(environment: str, values_file: Path | None = None) -> dict[str, Any]: - rendered = render(environment) if values_file is None else render(environment, values_file=values_file) +def rendered_pusher_deployment(environment: str) -> dict[str, Any]: + rendered = render(environment) deployments = [doc for doc in rendered if doc.get("kind") == "Deployment"] if len(deployments) != 1: raise RuntimeError("pusher render did not contain exactly one Deployment") return deployments[0] -def pusher_references( - environment: str, - deployment: dict[str, Any] | None = None, - values_file: Path | None = None, -) -> set[Reference]: +def pusher_references(environment: str, deployment: dict[str, Any] | None = None) -> set[Reference]: expected_configmap = f"{environment}-omi-backend-config" expected_secret = f"{environment}-omi-backend-secrets" - deployment = deployment or rendered_pusher_deployment(environment, values_file=values_file) + deployment = deployment or rendered_pusher_deployment(environment) refs = references(deployment.get("spec", {}).get("template", {}).get("spec", {})) object_refs = {(kind, name) for kind, name, _key in refs} missing = {("configmap", expected_configmap), ("secret", expected_secret)} - object_refs @@ -299,11 +291,10 @@ def main() -> int: parser = argparse.ArgumentParser() parser.add_argument("--environment", required=True, choices=sorted(ENVIRONMENTS)) parser.add_argument("--namespace", required=True) - parser.add_argument("--values-file", type=Path) parser.add_argument("--render-only", action="store_true") args = parser.parse_args() - deployment = rendered_pusher_deployment(args.environment, values_file=args.values_file) - refs = pusher_references(args.environment, deployment, values_file=args.values_file) + deployment = rendered_pusher_deployment(args.environment) + refs = pusher_references(args.environment, deployment) failures = validate_dev_pusher_binding_contract(deployment) if args.environment == "dev" else [] if not args.render_only and not failures: failures.extend(verify(args.namespace, refs)) diff --git a/backend/tests/unit/test_desktop_release_scripts.py b/backend/tests/unit/test_desktop_release_scripts.py index 70f0e7c9d13..66e6ee33021 100644 --- a/backend/tests/unit/test_desktop_release_scripts.py +++ b/backend/tests/unit/test_desktop_release_scripts.py @@ -1,13 +1,15 @@ import importlib.util +import json from pathlib import Path +import tempfile import pytest +from database.desktop_update_channels import _build_pointer, normalize_release_manifest REPO_ROOT = Path(__file__).resolve().parents[3] SCRIPTS = REPO_ROOT / ".github" / "scripts" PROMOTE_BETA_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "desktop_promote_beta.yml" PROMOTE_PROD_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "desktop_promote_prod.yml" -EMERGENCY_BETA_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "desktop_emergency_promote_beta.yml" QUALIFY_BETA_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "desktop_qualify_beta.yml" CODEMAGIC_CONFIG = REPO_ROOT / "codemagic.yaml" @@ -22,9 +24,8 @@ def _load(name: str, filename: str): mark_beta = _load("mark_desktop_release_beta", "mark-desktop-release-beta.py") prepare_beta = _load("prepare_desktop_beta_promotion", "prepare-desktop-beta-promotion.py") -nominate_stable = _load("nominate_desktop_stable_candidate", "nominate-desktop-stable-candidate.py") repair_installer = _load("desktop_repair_installer", "desktop_repair_installer.py") -qualification_dispatch = _load("desktop_qualification_dispatch", "desktop_qualification_dispatch.py") +qualification_evidence = _load("desktop_qualification_evidence", "desktop_qualification_evidence.py") def _release(body: str | None = None): @@ -34,6 +35,7 @@ def _release(body: str | None = None): isLive: false channel: candidate edSignature: signature +betaEdSignature: beta-signature changelog: Fixed updates|Improved recovery qualifiedBeta: true qualifiedBetaAt: 2026-07-09T12:00:00Z @@ -50,11 +52,47 @@ def _release(body: str | None = None): "assets": [ {"name": "Omi.zip", "url": "https://example.com/Omi.zip"}, {"name": "omi.dmg", "url": "https://example.com/omi.dmg"}, + {"name": "Omi.Beta.zip", "url": "https://example.com/Omi.Beta.zip"}, + {"name": "omi-beta.dmg", "url": "https://example.com/omi-beta.dmg"}, {"name": evidence, "url": "https://example.com/evidence.json"}, ], } +def _evidence(): + return { + "schema_version": 1, + "release_id": "v0.12.64+12064-macos", + "source_sha": "a" * 40, + "source_qualification": {"passed": True, "tier": "T2", "subject": "source-built named-bundle"}, + "signed_artifact_verification": {"passed": True, "subject": "exact signed ZIP/DMG bytes"}, + "artifacts": { + "Omi.zip": {"url": "https://example.com/Omi.zip", "sha256": "b" * 64, "signature": "signature"}, + "omi.dmg": {"url": "https://example.com/omi.dmg", "sha256": "c" * 64}, + "Omi.Beta.zip": { + "url": "https://example.com/Omi.Beta.zip", + "sha256": "d" * 64, + "signature": "beta-signature", + }, + "omi-beta.dmg": {"url": "https://example.com/omi-beta.dmg", "sha256": "e" * 64}, + }, + } + + +def _prepare(release=None, *, allow_stable_channel=False): + return prepare_beta.prepare_manifest( + _release() if release is None else release, + "v0.12.64+12064-macos", + "a" * 40, + "b" * 64, + "c" * 64, + beta_zip_sha256="d" * 64, + beta_dmg_sha256="e" * 64, + qualification_evidence=_evidence(), + allow_stable_channel=allow_stable_channel, + ) + + def test_mark_beta_changes_only_visibility_fields(): result = mark_beta.mark_beta(_release()["body"]) assert "isLive: true" in result @@ -63,50 +101,155 @@ def test_mark_beta_changes_only_visibility_fields(): def test_prepare_manifest_requires_exact_qualification_and_assets(): - manifest = prepare_beta.prepare_manifest( - _release(), - "v0.12.64+12064-macos", - "a" * 40, - "b" * 64, - "c" * 64, - ) + manifest = _prepare() assert manifest["build_number"] == 12064 assert manifest["qualification"]["tier"] == "T2" - assert manifest["qualification"]["qualified_at"] == "2026-07-09T12:00:00Z" + assert manifest["qualification"]["source"] == "trusted_github_actions_artifact" + assert manifest["qualification"]["evidence_asset"] == "qualification-evidence.json" + assert manifest["qualification"]["source_subject"] == "source-built named-bundle" + assert manifest["qualification"]["signed_artifact_subject"] == "exact signed ZIP/DMG bytes" assert manifest["changelog"] == ["Fixed updates", "Improved recovery"] -def test_stable_repair_bundle_uses_an_immutable_gcs_artifact(): - manifest = prepare_beta.prepare_manifest( - _release(), - "v0.12.64+12064-macos", - "a" * 40, - "b" * 64, - "c" * 64, +def test_beta_static_redirect_uses_beta_identity_dmg_on_every_surface(): + workflow = PROMOTE_BETA_WORKFLOW.read_text(encoding="utf-8") + redirect = workflow.split("Update existing beta download redirect", 1)[1] + assert redirect.count('manifest["beta_dmg_url"]') == 1 + assert redirect.count("['beta_dmg_url']") == 2 + assert 'manifest["dmg_url"]' not in redirect + assert "['dmg_url']" not in redirect + + +def test_prepare_manifest_rejects_caller_hashes_that_do_not_match_trusted_evidence(): + """Promotion can only bind bytes independently recorded by qualification.""" + evidence = { + "schema_version": 1, + "release_id": "v0.12.64+12064-macos", + "source_sha": "a" * 40, + "artifacts": { + "Omi.zip": {"url": "https://example.com/Omi.zip", "sha256": "b" * 64, "signature": "signature"}, + "omi.dmg": {"url": "https://example.com/omi.dmg", "sha256": "c" * 64}, + "Omi.Beta.zip": { + "url": "https://example.com/Omi.Beta.zip", + "sha256": "d" * 64, + "signature": "beta-signature", + }, + "omi-beta.dmg": {"url": "https://example.com/omi-beta.dmg", "sha256": "e" * 64}, + }, + "source_qualification": {"passed": True, "tier": "T2", "subject": "source-built named-bundle"}, + "signed_artifact_verification": {"passed": True, "subject": "exact signed ZIP/DMG bytes"}, + } + + with pytest.raises(ValueError, match="qualification evidence"): + prepare_beta.prepare_manifest( + _release(), + "v0.12.64+12064-macos", + "a" * 40, + "1" * 64, + "2" * 64, + beta_zip_sha256="3" * 64, + beta_dmg_sha256="4" * 64, + qualification_evidence=evidence, + ) + + +def test_qualified_artifact_replacement_is_rejected_before_beta_or_stable_pointering(): + release = _release() + release["assets"] = [ + {"name": name, "url": f"https://example.com/{name}", "digest": ""} + for name in ("Omi.zip", "omi.dmg", "Omi.Beta.zip", "omi-beta.dmg") + ] + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + paths = {} + for name, content in ( + ("Omi.zip", b"stable zip"), + ("omi.dmg", b"stable dmg"), + ("Omi.Beta.zip", b"beta zip"), + ("omi-beta.dmg", b"beta dmg"), + ): + path = root / name + path.write_bytes(content) + paths[name] = path + gate = root / "gate.json" + gate.write_text(json.dumps({"passed": True, "release_tag": release["tagName"]})) + evidence = qualification_evidence.build_evidence( + release, release["tagName"], "a" * 40, {**paths, "__candidate_gate__": gate} + ) + paths["Omi.Beta.zip"].write_bytes(b"replacement") + with pytest.raises(ValueError, match="Omi.Beta.zip hash differs"): + qualification_evidence.verify_evidence( + evidence, + release, + release["tagName"], + "a" * 40, + {name: qualification_evidence.file_sha256(path) for name, path in paths.items()}, + ) + + +def test_local_candidate_evidence_beta_stable_repoint_and_retry_simulation(): + """No-cloud release-path simulation keeps both pointers bound to exact bytes.""" + manifest = normalize_release_manifest(_prepare()) + beta = _build_pointer( + {}, + manifest, + transition="promote", + platform="macos", + channel="beta", + release_id=manifest["release_id"], + expected_generation=0, + ) + stable = _build_pointer( + {}, + manifest, + transition="promote", + platform="macos", + channel="stable", + release_id=manifest["release_id"], + expected_generation=0, ) + retry = _build_pointer( + stable, + manifest, + transition="promote", + platform="macos", + channel="stable", + release_id=manifest["release_id"], + expected_generation=0, + ) + assert retry is stable + retained = dict(manifest, release_id="v0.12.63+12063-macos", version="0.12.63+12063", build_number=12063) + repointed = _build_pointer( + stable, + retained, + transition="repoint", + platform="macos", + channel="stable", + release_id=retained["release_id"], + expected_generation=stable["generation"], + expected_current_release_id=stable["release_id"], + ) + assert beta["release_id"] == manifest["release_id"] + assert repointed["release_id"] == retained["release_id"] + assert manifest["beta_zip_sha256"] == "d" * 64 + assert manifest["beta_ed_signature"] == "beta-signature" + + +def test_stable_repair_bundle_uses_the_retained_manifest_installer_identity(): + manifest = _prepare() bundle = repair_installer.build_repair_bundle(manifest, "gs://omi_macos_updates") - assert bundle["artifact_object"] == "stable/v0.12.64+12064-macos/Omi.dmg" assert bundle["repair_object"] == "stable/v0.12.64+12064-macos/repair.json" assert bundle["repair"]["channel"] == "stable" assert bundle["repair"]["installer_sha256"] == "c" * 64 - assert bundle["repair"]["installer_url"] == ( - "https://storage.googleapis.com/omi_macos_updates/stable/v0.12.64+12064-macos/Omi.dmg" - ) - assert "example.com" not in bundle["latest"]["installer_url"] + assert bundle["repair"]["installer_url"] == "https://example.com/omi.dmg" assert "/Applications" in bundle["landing_page"] @pytest.mark.parametrize("field, value", [("platform", "windows"), ("dmg_sha256", "not-a-digest")]) def test_stable_repair_bundle_rejects_incomplete_or_wrong_platform_manifest(field, value): - manifest = prepare_beta.prepare_manifest( - _release(), - "v0.12.64+12064-macos", - "a" * 40, - "b" * 64, - "c" * 64, - ) + manifest = _prepare() manifest[field] = value with pytest.raises(ValueError): @@ -114,20 +257,14 @@ def test_stable_repair_bundle_rejects_incomplete_or_wrong_platform_manifest(fiel def test_stable_repair_bundle_requires_the_release_publication_time(): - manifest = prepare_beta.prepare_manifest( - _release(), - "v0.12.64+12064-macos", - "a" * 40, - "b" * 64, - "c" * 64, - ) + manifest = _prepare() manifest.pop("published_at") with pytest.raises(ValueError, match="published_at"): repair_installer.build_repair_bundle(manifest, "gs://omi_macos_updates") -def test_prepare_manifest_accepts_legacy_qualification_metadata(): +def test_prepare_manifest_ignores_mutable_legacy_qualification_metadata(): release = _release() release["body"] = ( release["body"] @@ -137,15 +274,8 @@ def test_prepare_manifest_accepts_legacy_qualification_metadata(): .replace("qualifiedBetaTier:", "blessedTier:") .replace("qualifiedBetaEvidence:", "blessedEvidence:") ) - manifest = prepare_beta.prepare_manifest( - release, - "v0.12.64+12064-macos", - "a" * 40, - "b" * 64, - "c" * 64, - ) - assert manifest["qualification"]["blessed_at"] == "2026-07-09T12:00:00Z" - assert "qualified_at" not in manifest["qualification"] + manifest = _prepare(release) + assert manifest["qualification"]["source"] == "trusted_github_actions_artifact" def test_prepare_manifest_allows_an_already_stable_release_only_for_repair_retries(): @@ -155,99 +285,26 @@ def test_prepare_manifest_allows_an_already_stable_release_only_for_repair_retri ) with pytest.raises(SystemExit, match="candidate or beta"): - prepare_beta.prepare_manifest( - release, - "v0.12.64+12064-macos", - "a" * 40, - "b" * 64, - "c" * 64, - ) + _prepare(release) - manifest = prepare_beta.prepare_manifest( - release, - "v0.12.64+12064-macos", - "a" * 40, - "b" * 64, - "c" * 64, - allow_stable_channel=True, - ) + manifest = _prepare(release, allow_stable_channel=True) assert manifest["release_id"] == "v0.12.64+12064-macos" def test_prepare_manifest_rejects_unqualified_candidate(): release = _release() - release["body"] = release["body"].replace("qualifiedBeta: true", "qualifiedBeta: false") - with pytest.raises(SystemExit, match="qualifiedBeta"): + evidence = _evidence() + evidence["source_qualification"] = {"passed": False, "tier": "T2"} + with pytest.raises(ValueError, match="source-built named-bundle T2"): prepare_beta.prepare_manifest( release, "v0.12.64+12064-macos", "a" * 40, "b" * 64, "c" * 64, - ) - - -def test_qualified_beta_can_be_nominated_as_stable_candidate(): - release = _release() - release["body"] = ( - release["body"].replace("isLive: false", "isLive: true").replace("channel: candidate", "channel: beta") - ) - body = nominate_stable.nominate( - release, - release_tag="v0.12.64+12064-macos", - target_sha="a" * 40, - beta_release_id="v0.12.64+12064-macos", - beta_source_sha="a" * 40, - nominator="release-operator", - rationale="soak gates passed", - soak_review="24h reviewed", - telemetry_review="crash telemetry reviewed", - release_notes_review="stable notes reviewed", - nominated_at="2026-07-10T12:00:00Z", - ) - assert "stableCandidate: true" in body - assert "stableCandidateQualificationEvidence: qualification-evidence.json" in body - - -def test_stable_nomination_rejects_non_current_beta(): - release = _release() - release["body"] = ( - release["body"].replace("isLive: false", "isLive: true").replace("channel: candidate", "channel: beta") - ) - with pytest.raises(SystemExit, match="beta pointer"): - nominate_stable.nominate( - release, - release_tag="v0.12.64+12064-macos", - target_sha="a" * 40, - beta_release_id="v0.12.63+12063-macos", - beta_source_sha="a" * 40, - nominator="release-operator", - rationale="soak gates passed", - soak_review="24h reviewed", - telemetry_review="crash telemetry reviewed", - release_notes_review="stable notes reviewed", - nominated_at="2026-07-10T12:00:00Z", - ) - - -def test_stable_nomination_rejects_beta_manifest_from_another_sha(): - release = _release() - release["body"] = ( - release["body"].replace("isLive: false", "isLive: true").replace("channel: candidate", "channel: beta") - ) - with pytest.raises(SystemExit, match="beta manifest source SHA"): - nominate_stable.nominate( - release, - release_tag="v0.12.64+12064-macos", - target_sha="a" * 40, - beta_release_id="v0.12.64+12064-macos", - beta_source_sha="b" * 40, - nominator="release-operator", - rationale="soak gates passed", - soak_review="24h reviewed", - telemetry_review="crash telemetry reviewed", - release_notes_review="stable notes reviewed", - nominated_at="2026-07-10T12:00:00Z", + beta_zip_sha256="d" * 64, + beta_dmg_sha256="e" * 64, + qualification_evidence=evidence, ) @@ -274,48 +331,33 @@ def test_automatic_beta_is_pauseable_and_rejects_stale_tags(): assert automatic_gate < candidate_download -def test_qualification_claims_are_serialized_by_the_trusted_runner_only(): +def test_qualification_is_serialized_by_tag_and_retried_without_release_body_state(): codemagic = CODEMAGIC_CONFIG.read_text() dispatch = codemagic[codemagic.index(" - name: Dispatch trusted macOS beta qualification") :] qualification = QUALIFY_BETA_WORKFLOW.read_text() - assert "authoritative atomic" in dispatch + assert "duplicate dispatches" in dispatch assert 'gh release edit "$CM_TAG"' not in dispatch assert "group: desktop-beta-qualification-${{ inputs.release_tag }}" in qualification assert "cancel-in-progress: false" in qualification - assert "desktop_qualification_dispatch.py claim" in qualification - assert "desktop_qualification_dispatch.py complete" in qualification - - -def test_qualified_dispatch_retry_skips_runner_but_requests_promotion(): - body = """""" - - result, should_run, should_promote, reason = qualification_dispatch.claim( - body, - key="codemagic:v0.12.64+12064-macos", - updated_at="2026-07-09T12:05:00Z", - run_id="12345", - run_url="https://github.com/BasedHardware/omi/actions/runs/12345", - allow_retry=False, - ) - - assert result == body - assert should_run is False - assert should_promote is True - assert reason == "dispatch key already qualified" + assert "for attempt in 1 2 3" in dispatch + assert "desktop_qualification_dispatch.py" not in qualification + assert "steps.candidate.outcome == 'success' && steps.qualify.outcome == 'success'" in qualification -def test_qualification_workflow_promotes_already_qualified_claims(): +def test_qualification_and_promotion_bind_all_four_enclosures_to_an_immutable_run_artifact(): qualification = QUALIFY_BETA_WORKFLOW.read_text() + promotion = PROMOTE_BETA_WORKFLOW.read_text() - assert "should_promote=" in qualification - assert "steps.claim.outputs.should_promote == 'true'" in qualification + for asset in ("Omi.zip", "Omi.dmg", "omi.dmg", "Omi.Beta.zip", "omi-beta.dmg"): + assert asset in qualification + assert asset in promotion + assert "actions/upload-artifact@v7" in qualification + assert "qualification_run_id" in promotion + assert "actions/download-artifact@v7" in promotion + assert "--beta-zip-sha256" in promotion + assert "--beta-dmg-sha256" in promotion + assert "git tag -l 'v*-macos' --sort=-v:refname | head -1" not in qualification def test_stable_promotion_remains_manual_only(): @@ -328,18 +370,77 @@ def test_stable_promotion_remains_manual_only(): assert "promote-stable" in workflow +def test_stable_workflow_allows_retained_repoint_but_requires_current_beta_for_promote_and_safe_retries(): + workflow = PROMOTE_PROD_WORKFLOW.read_text() + + assert "check_stable_pointer_precondition.py" in workflow + assert "Fetch exact retained qualified manifest" in workflow + assert "actions/download-artifact@v7" not in workflow + assert "prepare-desktop-beta-promotion.py" not in workflow + assert "Register immutable release manifest" not in workflow + assert "appcast.xml?identity=stable" in workflow + assert "verify_stable_appcast.py" in workflow + assert 'Authorization: Bearer $ACCESS_TOKEN' in workflow + assert 'Authorization: Bearer ***' not in workflow + assert 'ref: main' in workflow + + +def test_qualification_run_is_bound_to_the_exact_main_dispatch_and_workflow(): + for workflow in (PROMOTE_BETA_WORKFLOW.read_text(), PROMOTE_PROD_WORKFLOW.read_text()): + assert 'gh api "repos/$REPO/actions/runs/$QUALIFICATION_RUN_ID"' in workflow + assert 'jq -r .repository.full_name' in workflow + assert 'jq -r .head_repository.full_name' in workflow + assert 'jq -r .event' in workflow + assert '= workflow_dispatch' in workflow + assert 'jq -r .path' in workflow + assert '= .github/workflows/desktop_qualify_beta.yml' in workflow + assert 'jq -r .head_branch' in workflow + assert '= main' in workflow + assert 'jq -r .head_sha' in workflow + + +def test_beta_promotion_controls_are_pinned_to_main_and_only_accept_lost_response_generation_plus_one(): + workflow = PROMOTE_BETA_WORKFLOW.read_text() + + assert 'ref: main' in workflow + manifest = normalize_release_manifest(_prepare()) + current = { + "platform": "macos", + "channel": "beta", + "release_id": manifest["release_id"], + "version": manifest["version"], + "build_number": 12064, + "generation": 4, + } + assert ( + _build_pointer( + current, + manifest, + transition="promote", + platform="macos", + channel="beta", + release_id=manifest["release_id"], + expected_generation=3, + ) + is current + ) + + def test_stable_repair_is_published_immutably_before_stable_pointer_advances(): """Static wiring contract: a stable pointer is never advanced ahead of its repair artifact.""" workflow = PROMOTE_PROD_WORKFLOW.read_text() immutable_repair = workflow.index(" - name: Publish immutable stable repair installer") - legacy_bridge = workflow.index(" - name: Promote Firestore release stable") pointer = workflow.index(" - name: Advance explicit stable pointer") + legacy_bridge = workflow.index(" - name: Bridge stable for legacy desktop clients") latest_route = workflow.index(" - name: Publish latest stable repair route") - assert immutable_repair < legacy_bridge < pointer < latest_route - assert "--json tagName,body,isDraft,isPrerelease,publishedAt,assets" in workflow - assert "--pattern 'Omi.zip' --pattern 'Omi.dmg' --pattern 'omi.dmg'" in workflow - assert "--pattern '*.dmg'" not in workflow - assert "Expected exactly one qualified Omi.dmg or omi.dmg release asset." in workflow + assert immutable_repair < pointer < legacy_bridge < latest_route + assert "Fetch exact retained qualified manifest" in workflow + assert "gh release download" not in workflow assert "--if-generation-match=0" in workflow + assert "manifest_sha256" in workflow + assert '"$BASE/macos-beta"' in workflow + assert "expected_current_release_id:" in workflow + assert "expected_generation:" in workflow + assert "gcloud run deploy" not in workflow diff --git a/backend/tests/unit/test_desktop_update_channels.py b/backend/tests/unit/test_desktop_update_channels.py index 7db66d3eb6c..1a4b547567d 100644 --- a/backend/tests/unit/test_desktop_update_channels.py +++ b/backend/tests/unit/test_desktop_update_channels.py @@ -5,8 +5,8 @@ from database.desktop_update_channels import ( _build_pointer, - _rollback_macos_beta_transaction, get_channel_release, + get_release_manifest, normalize_release_manifest, register_release_manifest, ) @@ -20,13 +20,18 @@ def _manifest(**overrides): "build_number": 12064, "zip_url": "https://github.com/BasedHardware/omi/releases/download/test/Omi.zip", "dmg_url": "https://github.com/BasedHardware/omi/releases/download/test/Omi.dmg", + "beta_zip_url": "https://github.com/BasedHardware/omi/releases/download/test/Omi.Beta.zip", + "beta_dmg_url": "https://github.com/BasedHardware/omi/releases/download/test/omi-beta.dmg", "ed_signature": "sparkle-signature", + "beta_ed_signature": "beta-sparkle-signature", "published_at": "2026-07-09T12:00:00Z", "changelog": ["Qualified beta"], "mandatory": False, "source_sha": "a" * 40, "zip_sha256": "b" * 64, "dmg_sha256": "c" * 64, + "beta_zip_sha256": "d" * 64, + "beta_dmg_sha256": "e" * 64, "qualification": {"tier": "T2", "passed": True}, } data.update(overrides) @@ -54,6 +59,11 @@ def test_requires_dmg_for_macos(self): with pytest.raises(ValueError, match="dmg_url"): normalize_release_manifest(_manifest(dmg_url=None)) + def test_requires_all_beta_identity_artifact_fields_for_macos(self): + for field in ("beta_zip_url", "beta_dmg_url", "beta_ed_signature", "beta_zip_sha256", "beta_dmg_sha256"): + with pytest.raises(ValueError, match=field): + normalize_release_manifest(_manifest(**{field: None})) + class TestReleaseManifestPersistence: def test_register_is_idempotent_for_identical_manifest(self): @@ -104,6 +114,18 @@ def test_resolves_pointer_to_manifest(self): assert result["pointer"]["generation"] == 4 assert result["manifest"]["release_id"] == _manifest()["release_id"] + def test_reads_retained_manifest_without_a_channel_or_release_metadata(self): + snapshot = MagicMock(exists=True) + snapshot.to_dict.return_value = _manifest() + ref = MagicMock() + ref.get.return_value = snapshot + client = MagicMock() + client.collection.return_value.document.return_value = ref + + assert get_release_manifest(_manifest()["release_id"], firestore_client=client) == normalize_release_manifest( + _manifest() + ) + class TestChannelPromotionRules: def test_first_qualified_promotion_sets_generation_and_build(self): @@ -135,7 +157,8 @@ def test_idempotent_retry_does_not_increment_generation(self): platform="macos", channel="beta", release_id=_manifest()["release_id"], - expected_generation=4, + expected_generation=3, + expected_current_release_id="previous-release", ) assert pointer is current assert pointer["generation"] == 4 @@ -167,224 +190,87 @@ def test_rejects_unqualified_release(self): ) -class TestMacosBetaRollbackRules: - def test_rolls_back_qualified_release_and_creates_immutable_audit(self): - current = { - "platform": "macos", - "channel": "beta", - "release_id": "v0.12.84+12084-macos", - "version": "0.12.84+12084", - "build_number": 12084, - "generation": 7, - } - target = normalize_release_manifest( - _manifest(release_id="v0.12.73+12073-macos", version="0.12.73+12073", build_number=12073) - ) - pointer = _build_pointer( - current, - target, - transition="rollback", +class TestPointerRepointRules: + def test_qualified_manifest_moves_the_same_release_from_beta_to_stable(self): + """Local dry run of candidate evidence -> qualified manifest -> both pointers.""" + manifest = normalize_release_manifest(_manifest()) + beta = _build_pointer( + {}, + manifest, + transition="promote", platform="macos", channel="beta", - release_id=target["release_id"], - expected_current_release_id=current["release_id"], - expected_generation=7, + release_id=manifest["release_id"], + expected_generation=0, ) - - pointer_snapshot = MagicMock(exists=True) - pointer_snapshot.to_dict.return_value = current - manifest_snapshot = MagicMock(exists=True) - manifest_snapshot.to_dict.return_value = target - pointer_ref = MagicMock() - pointer_ref.get.return_value = pointer_snapshot - manifest_ref = MagicMock() - manifest_ref.get.return_value = manifest_snapshot - audit_ref = MagicMock() - transaction = MagicMock() - - result = _rollback_macos_beta_transaction.to_wrap( - transaction, - pointer_ref, - manifest_ref, - audit_ref, - release_id=target["release_id"], - expected_current_release_id=current["release_id"], - expected_generation=7, - audit_id="audit-123", - occurred_at=pointer["updated_at"], + stable = _build_pointer( + {}, + manifest, + transition="promote", + platform="macos", + channel="stable", + release_id=beta["release_id"], + expected_generation=0, ) - assert result["pointer"]["release_id"] == target["release_id"] - assert result["pointer"]["generation"] == 8 - assert result["audit"] == { - "audit_id": "audit-123", - "operation": "macos_beta_rollback", - "platform": "macos", - "channel": "beta", - "previous_release_id": current["release_id"], - "previous_generation": 7, - "target_release_id": target["release_id"], - "generation": 8, - "occurred_at": pointer["updated_at"], - } - transaction.create.assert_called_once_with(audit_ref, result["audit"]) - transaction.set.assert_called_once_with(pointer_ref, result["pointer"]) + assert beta["release_id"] == manifest["release_id"] == stable["release_id"] + assert beta["generation"] == stable["generation"] == 1 - def test_rejects_stale_current_release_or_generation(self): + def test_repoints_a_qualified_retained_manifest_with_compare_and_swap(self): current = {"release_id": "v0.12.84+12084-macos", "build_number": 12084, "generation": 7} target = normalize_release_manifest( _manifest(release_id="v0.12.73+12073-macos", version="0.12.73+12073", build_number=12073) ) - with pytest.raises(ValueError, match="current release mismatch"): - _build_pointer( - current, - target, - transition="rollback", - platform="macos", - channel="beta", - release_id=target["release_id"], - expected_current_release_id="v0.12.83+12083-macos", - expected_generation=7, - ) - with pytest.raises(ValueError, match="generation mismatch"): - _build_pointer( - current, - target, - transition="rollback", - platform="macos", - channel="beta", - release_id=target["release_id"], - expected_current_release_id=current["release_id"], - expected_generation=6, - ) - - def test_rejects_unqualified_or_non_macos_target(self): - current = {"release_id": "v0.12.84+12084-macos", "build_number": 12084, "generation": 7} - unqualified = normalize_release_manifest( - _manifest( - release_id="v0.12.73+12073-macos", - version="0.12.73+12073", - build_number=12073, - qualification={"tier": "T2", "passed": False}, - ) - ) - with pytest.raises(ValueError, match="qualification"): - _build_pointer( - current, - unqualified, - transition="rollback", - platform="macos", - channel="beta", - release_id=unqualified["release_id"], - expected_current_release_id=current["release_id"], - expected_generation=7, - ) - - -class TestEmergencyBetaPromotionRules: - """Emergency promotion is ordinary promotion with the T2 gate relaxed. - - These cover the break-glass contract: it may ship unqualified builds to - beta, it must still compare-and-swap, and it must never reach stable. - """ - - def _unqualified(self): - return normalize_release_manifest( - _manifest( - release_id="v0.12.87+12087-macos", - version="0.12.87+12087", - build_number=12087, - qualification={"tier": "T2", "passed": False}, - ) - ) - - def _current(self): - return {"release_id": "v0.12.86+12086-macos", "build_number": 12086, "generation": 9} - - def test_promotes_an_unqualified_build_and_marks_the_pointer_emergency(self): - current = self._current() - target = self._unqualified() pointer = _build_pointer( current, target, - transition="emergency", + transition="repoint", platform="macos", channel="beta", release_id=target["release_id"], expected_current_release_id=current["release_id"], - expected_generation=9, + expected_generation=7, ) - assert pointer["release_id"] == target["release_id"] - assert pointer["generation"] == 10 - assert pointer["emergency"] is True - def test_normal_promotion_still_rejects_the_same_unqualified_build(self): - with pytest.raises(ValueError, match="qualification"): - _build_pointer( - self._current(), - self._unqualified(), - transition="promote", - platform="macos", - channel="beta", - release_id="v0.12.87+12087-macos", - expected_generation=9, - ) + assert pointer["release_id"] == target["release_id"] + assert pointer["generation"] == 8 @pytest.mark.parametrize( "expected_release_id, expected_generation, message", [ - ("v0.12.99+12099-macos", 9, "current release mismatch"), - ("v0.12.86+12086-macos", 3, "generation mismatch"), + ("v0.12.83+12083-macos", 7, "current release mismatch"), + ("v0.12.84+12084-macos", 6, "generation mismatch"), ], ) - def test_rejects_a_stale_compare_and_swap(self, expected_release_id, expected_generation, message): + def test_rejects_stale_repoint_compare_and_swap(self, expected_release_id, expected_generation, message): + current = {"release_id": "v0.12.84+12084-macos", "build_number": 12084, "generation": 7} + target = normalize_release_manifest( + _manifest(release_id="v0.12.73+12073-macos", version="0.12.73+12073", build_number=12073) + ) with pytest.raises(ValueError, match=message): _build_pointer( - self._current(), - self._unqualified(), - transition="emergency", + current, + target, + transition="repoint", platform="macos", channel="beta", - release_id="v0.12.87+12087-macos", + release_id=target["release_id"], expected_current_release_id=expected_release_id, expected_generation=expected_generation, ) - def test_is_roll_forward_only(self): - current = {"release_id": "v0.12.86+12086-macos", "build_number": 12086, "generation": 9} - older = normalize_release_manifest( - _manifest( - release_id="v0.12.80+12080-macos", - version="0.12.80+12080", - build_number=12080, - qualification={"tier": "T2", "passed": False}, - ) - ) - with pytest.raises(ValueError, match="roll-forward only"): - _build_pointer( - current, - older, - transition="emergency", - platform="macos", - channel="beta", - release_id=older["release_id"], - expected_current_release_id=current["release_id"], - expected_generation=9, - ) - - @pytest.mark.parametrize("channel", ["stable"]) - def test_never_reaches_stable(self, channel): - current = self._current() - target = self._unqualified() - with pytest.raises(ValueError, match="only permitted for the macos beta channel"): + def test_repoint_rejects_unqualified_manifest(self): + current = {"release_id": "v0.12.84+12084-macos", "build_number": 12084, "generation": 7} + target = normalize_release_manifest(_manifest(qualification={"tier": "T2", "passed": False})) + with pytest.raises(ValueError, match="qualification"): _build_pointer( current, target, - transition="emergency", + transition="repoint", platform="macos", - channel=channel, + channel="stable", release_id=target["release_id"], expected_current_release_id=current["release_id"], - expected_generation=9, + expected_generation=7, ) diff --git a/backend/tests/unit/test_desktop_update_resolver.py b/backend/tests/unit/test_desktop_update_resolver.py index 89b90cf17de..6745ba3c5fb 100644 --- a/backend/tests/unit/test_desktop_update_resolver.py +++ b/backend/tests/unit/test_desktop_update_resolver.py @@ -19,13 +19,18 @@ def _release(channel="beta"): "build_number": 12064, "zip_url": "https://example.com/Omi.zip", "dmg_url": "https://example.com/Omi.dmg", + "beta_zip_url": "https://example.com/Omi.Beta.zip", + "beta_dmg_url": "https://example.com/omi-beta.dmg", "ed_signature": "signature", + "beta_ed_signature": "beta-signature", "published_at": "2026-07-09T12:00:00Z", "changelog": ["Qualified beta"], "mandatory": False, "source_sha": "a" * 40, "zip_sha256": "b" * 64, "dmg_sha256": "c" * 64, + "beta_zip_sha256": "d" * 64, + "beta_dmg_sha256": "e" * 64, "qualification": {"passed": True, "tier": "T2"}, }, } diff --git a/backend/tests/unit/test_desktop_updates.py b/backend/tests/unit/test_desktop_updates.py index f2ab2e68646..4903f2aea00 100644 --- a/backend/tests/unit/test_desktop_updates.py +++ b/backend/tests/unit/test_desktop_updates.py @@ -373,13 +373,18 @@ def _pointer_release(channel="beta", build=200): "build_number": build, "zip_url": f"https://example.com/{build}/Omi.zip", "dmg_url": f"https://example.com/{build}/Omi.dmg", + "beta_zip_url": f"https://example.com/{build}/Omi.Beta.zip", + "beta_dmg_url": f"https://example.com/{build}/omi-beta.dmg", "ed_signature": "signature", + "beta_ed_signature": "beta-signature", "published_at": "2026-03-01T00:00:00Z", "changelog": ["Qualified release"], "mandatory": False, "source_sha": "a" * 40, "zip_sha256": None, "dmg_sha256": None, + "beta_zip_sha256": "d" * 64, + "beta_dmg_sha256": "e" * 64, "qualification": {"tier": "T2", "passed": True}, }, } @@ -840,6 +845,23 @@ async def test_manifest_mutation_conflict_is_reported(self): assert resp.status_code == 409 assert "immutable" in resp.json()["detail"] + @pytest.mark.asyncio + async def test_reads_the_retained_manifest_with_a_canonical_identity(self): + payload = _pointer_release()["manifest"] + with ( + patch.dict("os.environ", {"ADMIN_KEY": "real-secret"}), + patch("routers.updates.get_release_manifest", return_value=payload) as read_manifest, + ): + async with AsyncClient(transport=ASGITransport(app=_test_app), base_url="http://test") as client: + resp = await client.get( + f"/v2/desktop/releases/{payload['release_id']}", headers={"secret-key": "real-secret"} + ) + + assert resp.status_code == 200 + assert resp.json()["manifest"] == payload + assert len(resp.json()["manifest_sha256"]) == 64 + read_manifest.assert_called_once_with(payload["release_id"]) + @pytest.mark.asyncio async def test_promotes_pointer_and_clears_only_live_pointer_cache(self): pointer = { @@ -872,109 +894,17 @@ async def test_promotes_pointer_and_clears_only_live_pointer_cache(self): "v1.0.0+200-macos", expected_generation=1, expected_current_release_id=None, - emergency=False, - ) - delete_cache.assert_called_once_with("desktop_update_pointer:macos:beta") - - @pytest.mark.asyncio - async def test_rolls_back_macos_beta_with_auditable_response(self): - result = { - "pointer": { - "platform": "macos", - "channel": "beta", - "release_id": "v0.12.73+12073-macos", - "generation": 8, - }, - "audit": { - "audit_id": "immutable-audit-id", - "operation": "macos_beta_rollback", - "platform": "macos", - "channel": "beta", - "previous_release_id": "v0.12.84+12084-macos", - "previous_generation": 7, - "target_release_id": "v0.12.73+12073-macos", - "generation": 8, - }, - } - with ( - patch.dict("os.environ", {"ADMIN_KEY": "real-secret"}), - patch("routers.updates.rollback_macos_beta_channel", return_value=result) as rollback, - patch("routers.updates.delete_generic_cache") as delete_cache, - ): - async with AsyncClient(transport=ASGITransport(app=_test_app), base_url="http://test") as client: - resp = await client.post( - "/v2/desktop/channels/rollback", - headers={"secret-key": "real-secret"}, - json={ - "platform": "macos", - "channel": "beta", - "release_id": "v0.12.73+12073-macos", - "expected_current_release_id": "v0.12.84+12084-macos", - "expected_generation": 7, - }, - ) - - assert resp.status_code == 200 - assert resp.json()["audit"] == result["audit"] - rollback.assert_called_once_with( - "v0.12.73+12073-macos", - expected_current_release_id="v0.12.84+12084-macos", - expected_generation=7, + operation="promote", ) delete_cache.assert_called_once_with("desktop_update_pointer:macos:beta") @pytest.mark.asyncio - async def test_rollback_reports_stale_compare_and_swap(self): - with ( - patch.dict("os.environ", {"ADMIN_KEY": "real-secret"}), - patch( - "routers.updates.rollback_macos_beta_channel", - side_effect=ValueError("current release mismatch: expected old, current newer"), - ), - ): - async with AsyncClient(transport=ASGITransport(app=_test_app), base_url="http://test") as client: - resp = await client.post( - "/v2/desktop/channels/rollback", - headers={"secret-key": "real-secret"}, - json={ - "platform": "macos", - "channel": "beta", - "release_id": "v0.12.73+12073-macos", - "expected_current_release_id": "v0.12.84+12084-macos", - "expected_generation": 7, - }, - ) - - assert resp.status_code == 409 - assert "current release mismatch" in resp.json()["detail"] - - @pytest.mark.asyncio - @pytest.mark.parametrize(("platform", "channel"), [("macos", "stable"), ("windows", "beta")]) - async def test_rollback_rejects_stable_or_other_platform(self, platform, channel): - async with AsyncClient(transport=ASGITransport(app=_test_app), base_url="http://test") as client: - resp = await client.post( - "/v2/desktop/channels/rollback", - headers={"secret-key": "any-secret"}, - json={ - "platform": platform, - "channel": channel, - "release_id": "v0.12.73+12073-macos", - "expected_current_release_id": "v0.12.84+12084-macos", - "expected_generation": 7, - }, - ) - - assert resp.status_code == 422 - - @pytest.mark.asyncio - async def test_emergency_flag_routes_through_promote_and_clears_only_beta_cache(self): - """Break-glass promotion is the normal endpoint with `emergency: true`.""" + async def test_repoint_routes_through_the_shared_pointer_authority(self): pointer = { "platform": "macos", - "channel": "beta", - "release_id": "v0.12.87+12087-macos", + "channel": "stable", + "release_id": "v0.12.73+12073-macos", "generation": 10, - "emergency": True, } with ( patch("routers.updates.os.getenv", return_value="real-secret"), @@ -987,25 +917,25 @@ async def test_emergency_flag_routes_through_promote_and_clears_only_beta_cache( headers={"secret-key": "real-secret"}, json={ "platform": "macos", - "channel": "beta", - "release_id": "v0.12.87+12087-macos", + "channel": "stable", + "release_id": "v0.12.73+12073-macos", "expected_current_release_id": "v0.12.86+12086-macos", "expected_generation": 9, - "emergency": True, + "operation": "repoint", }, ) assert resp.status_code == 200 - assert resp.json()["pointer"]["emergency"] is True + assert resp.json()["pointer"]["release_id"] == pointer["release_id"] promote.assert_called_once_with( "macos", - "beta", - "v0.12.87+12087-macos", + "stable", + "v0.12.73+12073-macos", expected_generation=9, expected_current_release_id="v0.12.86+12086-macos", - emergency=True, + operation="repoint", ) - delete_cache.assert_called_once_with("desktop_update_pointer:macos:beta") + delete_cache.assert_called_once_with("desktop_update_pointer:macos:stable") # --- Update policy endpoint --- @@ -1252,6 +1182,15 @@ def _live_entry(channel="beta", assets=None, metadata=None, tag="v1.0.0+100-maco class TestBetaIdentityServing: + def test_beta_pointer_projects_only_registered_enclosure_hash_and_signature(self): + from routers.updates import _pointer_release_to_entry + + entry = _pointer_release_to_entry(_pointer_release("beta"), "beta", "pointer") + beta_zip = next(asset for asset in entry["release"]["assets"] if asset["name"] == "Omi.Beta.zip") + assert beta_zip["browser_download_url"] == "https://example.com/200/Omi.Beta.zip" + assert entry["metadata"]["betaEdSignature"] == "beta-signature" + assert entry["metadata"]["betaZipSha256"] == "d" * 64 + @pytest.mark.asyncio async def test_appcast_identity_beta_serves_beta_enclosure_and_drops_stable_items(self): entries = [ @@ -1313,10 +1252,14 @@ async def test_appcast_identity_beta_omits_releases_without_beta_artifacts(self) assert resp.text.count("") == 0 @pytest.mark.asyncio - async def test_appcast_identity_beta_resolves_pointer_entries_from_release_by_tag(self): - # Pointer entries fabricate their asset list from the manifest; the beta - # enclosure must come from the underlying GitHub release looked up by tag. - entries = [_live_entry(channel="beta", assets=[_zip_asset()], metadata={"edSignature": "sig"})] + async def test_appcast_identity_beta_never_resolves_mutable_release_assets_after_promotion(self): + entries = [ + _live_entry( + channel="beta", + assets=[_zip_asset(), _beta_zip_asset("https://immutable.example/Omi.Beta.zip")], + metadata={"edSignature": "sig", "betaEdSignature": "immutable-beta-sig"}, + ) + ] gh_release = _make_github_release( "v1.0.0+100-macos", body_kv={"isLive": "false", "betaEdSignature": "beta-sig-from-body"}, @@ -1331,8 +1274,9 @@ async def test_appcast_identity_beta_resolves_pointer_entries_from_release_by_ta assert resp.status_code == 200 xml = resp.text - assert "https://example.com/by-tag/Omi.Beta.zip" in xml - assert 'edSignature="beta-sig-from-body"' in xml + assert "https://immutable.example/Omi.Beta.zip" in xml + assert 'edSignature="immutable-beta-sig"' in xml + assert "by-tag" not in xml def test_stable_dmg_picker_never_returns_the_beta_dmg(self): release = { diff --git a/backend/tests/unit/test_memory_apply_store.py b/backend/tests/unit/test_memory_apply_store.py index 38f7b722cad..cd29c9c46ee 100644 --- a/backend/tests/unit/test_memory_apply_store.py +++ b/backend/tests/unit/test_memory_apply_store.py @@ -1,6 +1,6 @@ import copy import os -from datetime import datetime, timezone +from datetime import datetime, timedelta, timezone from pathlib import Path from types import ModuleType from typing import Optional @@ -342,6 +342,53 @@ def test_firestore_apply_allows_update_when_target_is_authoritative_active_same_ assert result.status == ApplyStatus.committed +def test_firestore_apply_update_keeps_persisted_timestamps_monotonic_when_apply_clock_is_behind(store, monkeypatch): + captured_at = datetime(2026, 8, 21, 1, 24, 2, 685960, tzinfo=timezone(timedelta(hours=5, minutes=30))) + prior_updated_at = captured_at + timedelta(minutes=1) + expires_at = captured_at + timedelta(days=30) + operation = _operation( + target_memory_id="mem1", + logical_payload={ + "decision": "update", + "target_memory_id": "mem1", + "memory_text": "Updated.", + "result_status": "active", + }, + ) + existing = _target_item( + tier=MemoryTier.short_term, + captured_at=captured_at, + updated_at=prior_updated_at, + expires_at=expires_at, + ) + db = _db_with(operation=operation, target_items=[existing]) + patch = _patch(decision=DurablePatchDecision.update, target_memory_id="mem1", memory_text="Updated.") + + import models.memory_apply as memory_apply + + class _EarlierApplyClock(datetime): + @classmethod + def now(cls, tz=None): + assert tz is timezone.utc + return captured_at.astimezone(timezone.utc) - timedelta(days=1) + + monkeypatch.setattr(memory_apply, "datetime", _EarlierApplyClock) + + result = store.apply_long_term_patch_firestore( + uid="u1", + operation_id=operation.operation_id, + patch_payload=patch, + db_client=db, + ) + + assert result.status == ApplyStatus.committed + persisted = db.docs["users/u1/memory_items/mem1"] + restored = MemoryItem(**persisted) + assert restored.expires_at == expires_at + assert restored.updated_at >= captured_at + assert restored.updated_at >= prior_updated_at + + def test_firestore_apply_retries_committed_operation_from_stored_result_without_rereading_mutable_evidence_or_target( store, ): diff --git a/backend/tests/unit/test_probe_transcription_candidate_from_cloud_run.py b/backend/tests/unit/test_probe_transcription_candidate_from_cloud_run.py new file mode 100644 index 00000000000..ce05d4f1572 --- /dev/null +++ b/backend/tests/unit/test_probe_transcription_candidate_from_cloud_run.py @@ -0,0 +1,124 @@ +from __future__ import annotations + +import os +import re +import subprocess +from pathlib import Path + +SCRIPT = Path(__file__).resolve().parents[2] / "scripts/probe-transcription-candidate-from-cloud-run.sh" + + +def _fake_gcloud(bin_dir: Path) -> Path: + fake_gcloud = bin_dir / "gcloud" + fake_gcloud.write_text( + """#!/usr/bin/env bash +set -euo pipefail +printf '%s\\n' "$*" >> "$GCLOUD_LOG" + +if [[ "$*" == "compute networks subnets describe "* ]]; then + printf '%s\\n' "${PRIVATE_GOOGLE_ACCESS:-True}" + exit 0 +fi + +if [[ -n "${GCLOUD_FAIL_COMMAND:-}" && "$*" == "$GCLOUD_FAIL_COMMAND"* ]]; then + exit 42 +fi +""", + encoding="utf-8", + ) + fake_gcloud.chmod(0o755) + return fake_gcloud + + +def _run_probe(tmp_path: Path, **env_overrides: str) -> tuple[subprocess.CompletedProcess[str], list[str]]: + bin_dir = tmp_path / "bin" + bin_dir.mkdir(parents=True) + log_path = tmp_path / "gcloud.log" + _fake_gcloud(bin_dir) + token_file = tmp_path / "firebase-token" + token_file.write_text("fixture-token", encoding="utf-8") + env = os.environ | {"PATH": f"{bin_dir}:{os.environ['PATH']}", "GCLOUD_LOG": str(log_path)} | env_overrides + + suffix = "29797736699-123456789" + result = subprocess.run( + [ + "bash", + str(SCRIPT), + "--project", + "example-project", + "--region", + "us-central1", + "--image", + "example.invalid/backend:abc123", + "--candidate-url", + "https://candidate.example.invalid", + "--identity-audience", + "https://backend.example.invalid", + "--network", + "default", + "--subnet", + "default", + "--firebase-token-file", + str(token_file), + "--name-suffix", + suffix, + ], + env=env, + text=True, + capture_output=True, + check=False, + ) + calls = log_path.read_text(encoding="utf-8").splitlines() + return result, calls + + +def test_ephemeral_probe_uses_bounded_deterministic_resource_ids_and_all_traffic(tmp_path): + result, calls = _run_probe(tmp_path) + + assert result.returncode == 0, result.stderr + create = next(line for line in calls if line.startswith("iam service-accounts create ")) + account_id = create.split()[3] + assert re.fullmatch(r"bcp-[0-9a-f]{20}", account_id) + assert 6 <= len(account_id) <= 30 + deploy = next(line for line in calls if line.startswith("run jobs deploy ")) + assert re.search(r"run jobs deploy backend-candidate-vpc-probe-[0-9a-f]{20}(?: |$)", deploy) + assert "--vpc-egress=all-traffic" in deploy + + +def test_probe_refuses_to_create_resources_without_private_google_access(tmp_path): + result, calls = _run_probe(tmp_path, PRIVATE_GOOGLE_ACCESS="False") + + assert result.returncode != 0 + assert "Private Google Access" in result.stderr + assert calls == [ + "compute networks subnets describe default --project=example-project --region=us-central1 " + "--format=value(privateIpGoogleAccess)" + ] + + +def test_probe_cleanup_removes_every_created_resource_at_each_failure_boundary(tmp_path): + failures = { + "after_service_account": "run services add-iam-policy-binding backend", + "after_iam_binding": "run jobs deploy ", + "after_job_deployment": "run jobs execute ", + } + cleanup = ( + "run jobs delete ", + "run services remove-iam-policy-binding backend", + "iam service-accounts delete ", + ) + + for boundary, command in failures.items(): + result, calls = _run_probe(tmp_path / boundary, GCLOUD_FAIL_COMMAND=command) + + assert result.returncode == 42, boundary + failure_index = next(index for index, call in enumerate(calls) if call.startswith(command)) + assert all( + any(call.startswith(expected) for call in calls[failure_index + 1 :]) for expected in cleanup + ), boundary + if boundary == "after_service_account": + assert not any(call.startswith("run jobs deploy ") for call in calls[:failure_index]) + elif boundary == "after_iam_binding": + assert any(call.startswith("run services add-iam-policy-binding backend") for call in calls[:failure_index]) + else: + assert any(call.startswith("run jobs deploy ") for call in calls[:failure_index]) diff --git a/backend/tests/unit/test_release_ring_gke_snapshot.py b/backend/tests/unit/test_release_ring_gke_snapshot.py deleted file mode 100644 index f189d7db8c9..00000000000 --- a/backend/tests/unit/test_release_ring_gke_snapshot.py +++ /dev/null @@ -1,157 +0,0 @@ -from __future__ import annotations - -import json -import subprocess - -from scripts import release_ring_gke_snapshot as snapshots - -NAMESPACE = 'prod-omi-backend' -CONFIG_NAME = 'prod-omi-backend-config' - - -def _config_map(value: str) -> dict: - return { - 'apiVersion': 'v1', - 'kind': 'ConfigMap', - 'metadata': {'name': CONFIG_NAME, 'namespace': NAMESPACE}, - 'data': {'PUBLIC_VALUE': value}, - } - - -def _config_snapshot(value: str) -> dict: - sanitized = snapshots._sanitize_config_map(_config_map(value), namespace=NAMESPACE, name=CONFIG_NAME) - return { - 'name': CONFIG_NAME, - 'present': True, - 'sha256': snapshots._sha256_text(snapshots._canonical_json(sanitized)), - 'object': sanitized, - } - - -def test_capture_records_exact_config_and_helm_restore_targets() -> None: - calls: list[list[str]] = [] - - def runner(command, _input): - command = list(command) - calls.append(command) - if command[:6] == ['kubectl', '-n', NAMESPACE, 'get', 'configmap', CONFIG_NAME]: - return json.dumps(_config_map('before')) - if command[:5] == ['helm', '-n', NAMESPACE, 'history', 'prod-omi-backend-secrets']: - return json.dumps([{'revision': 3, 'status': 'superseded'}, {'revision': 4, 'status': 'deployed'}]) - if command[:6] == ['helm', '-n', NAMESPACE, 'get', 'manifest', 'prod-omi-backend-secrets']: - return 'old-secret-manifest\n' - if command[:5] == ['helm', '-n', NAMESPACE, 'history', 'prod-omi-pusher']: - raise subprocess.CalledProcessError(1, command, stderr='release: not found') - raise AssertionError(command) - - snapshot = snapshots.capture_snapshot( - namespace=NAMESPACE, - config_map_name=CONFIG_NAME, - releases={ - 'backend-secrets': 'prod-omi-backend-secrets', - 'pusher': 'prod-omi-pusher', - }, - runner=runner, - ) - - assert snapshot['config_map']['object']['data'] == {'PUBLIC_VALUE': 'before'} - assert snapshot['helm_releases']['backend-secrets'] == { - 'name': 'prod-omi-backend-secrets', - 'present': True, - 'revision': 4, - 'manifest_sha256': snapshots._sha256_text('old-secret-manifest\n'), - } - assert snapshot['helm_releases']['pusher'] == {'name': 'prod-omi-pusher', 'present': False} - assert calls[-1] == ['helm', '-n', NAMESPACE, 'history', 'prod-omi-pusher', '--output', 'json'] - - -def test_restore_reverts_config_and_secrets_then_removes_bootstrap_release() -> None: - old_secret_manifest = 'old-secret-manifest\n' - snapshot = { - 'schema_version': 1, - 'namespace': NAMESPACE, - 'config_map': _config_snapshot('before'), - 'helm_releases': { - 'backend-secrets': { - 'name': 'prod-omi-backend-secrets', - 'present': True, - 'revision': 4, - 'manifest_sha256': snapshots._sha256_text(old_secret_manifest), - }, - 'pusher': {'name': 'prod-omi-pusher', 'present': False}, - }, - } - calls: list[list[str]] = [] - current_config = _config_map('candidate') - current_secret_manifest = 'candidate-secret-manifest\n' - pusher_exists = True - - def runner(command, input_text): - nonlocal current_config, current_secret_manifest, pusher_exists - command = list(command) - calls.append(command) - if command[:5] == ['helm', '-n', NAMESPACE, 'rollback', 'prod-omi-backend-secrets']: - current_secret_manifest = old_secret_manifest - return '' - if command[:6] == ['helm', '-n', NAMESPACE, 'get', 'manifest', 'prod-omi-backend-secrets']: - return current_secret_manifest - if command[:6] == ['kubectl', '-n', NAMESPACE, 'apply', '-f', '-']: - current_config = json.loads(input_text) - return '' - if command[:6] == ['kubectl', '-n', NAMESPACE, 'get', 'configmap', CONFIG_NAME]: - return json.dumps(current_config) - if command[:5] == ['helm', '-n', NAMESPACE, 'status', 'prod-omi-pusher']: - if pusher_exists: - return json.dumps({'info': {'status': 'deployed'}}) - raise subprocess.CalledProcessError(1, command, stderr='release: not found') - if command[:5] == ['helm', '-n', NAMESPACE, 'uninstall', 'prod-omi-pusher']: - pusher_exists = False - return '' - raise AssertionError(command) - - evidence = snapshots.restore_snapshot(snapshot, runner=runner) - - assert evidence['result'] == 'pass' - assert evidence['components']['backend-secrets']['result'] == 'restored' - assert evidence['components']['backend-config']['result'] == 'restored' - assert evidence['components']['pusher']['result'] == 'deleted' - assert calls.index( - ['helm', '-n', NAMESPACE, 'rollback', 'prod-omi-backend-secrets', '4', '--wait', '--timeout', '1800s'] - ) < calls.index(['kubectl', '-n', NAMESPACE, 'apply', '-f', '-']) - - -def test_restore_fails_when_helm_reports_success_without_manifest_convergence() -> None: - snapshot = { - 'schema_version': 1, - 'namespace': NAMESPACE, - 'config_map': {'name': CONFIG_NAME, 'present': False}, - 'helm_releases': { - 'pusher': { - 'name': 'prod-omi-pusher', - 'present': True, - 'revision': 7, - 'manifest_sha256': snapshots._sha256_text('old-manifest\n'), - } - }, - } - - def runner(command, _input): - command = list(command) - if command[:6] == ['kubectl', '-n', NAMESPACE, 'delete', 'configmap', CONFIG_NAME]: - return '' - if command[:6] == ['kubectl', '-n', NAMESPACE, 'get', 'configmap', CONFIG_NAME]: - raise subprocess.CalledProcessError(1, command, stderr='NotFound') - if command[:5] == ['helm', '-n', NAMESPACE, 'rollback', 'prod-omi-pusher']: - return '' - if command[:6] == ['helm', '-n', NAMESPACE, 'get', 'manifest', 'prod-omi-pusher']: - return 'still-candidate\n' - raise AssertionError(command) - - evidence = snapshots.restore_snapshot(snapshot, runner=runner) - - assert evidence['result'] == 'fail' - assert evidence['failed_components'] == ['pusher'] - assert ( - evidence['components']['pusher']['error'] - == 'prod-omi-pusher: observed Helm manifest does not match the snapshot' - ) diff --git a/backend/tests/unit/test_release_rings.py b/backend/tests/unit/test_release_rings.py deleted file mode 100644 index 148981e4631..00000000000 --- a/backend/tests/unit/test_release_rings.py +++ /dev/null @@ -1,183 +0,0 @@ -from __future__ import annotations - -import importlib.util -from pathlib import Path - -import pytest - -MODULE_PATH = Path(__file__).resolve().parents[2] / "scripts" / "release_rings.py" -SPEC = importlib.util.spec_from_file_location("release_rings", MODULE_PATH) -assert SPEC and SPEC.loader -release_rings = importlib.util.module_from_spec(SPEC) -SPEC.loader.exec_module(release_rings) - - -def _record() -> dict[str, object]: - digest = "a" * 64 - config_digest = "b" * 64 - return release_rings.build_record( - release_id="2026-07-20.42", - git_sha="c" * 40, - eligibility_run_id="42", - images={component: f"gcr.io/example/{component}@sha256:{digest}" for component in release_rings.COMPONENTS}, - rendered_config={ - f"{ring}/{component}": f"gs://release-records/config/2026-07-20.42/{ring}/{component}.json#sha256:{config_digest}" - for ring in release_rings.RINGS - for component in release_rings.CONFIG_COMPONENTS - }, - secret_versions={"OPENAI_API_KEY": "projects/example/secrets/OPENAI_API_KEY/versions/7"}, - topology={ - "beta": { - "namespace": "beta-omi-backend", - "cloud_run_services": { - "backend": "backend-beta", - "backend-sync": "backend-sync-beta", - "backend-sync-backfill": "backend-sync-backfill-beta", - "backend-integration": "backend-integration-beta", - }, - }, - "prod": { - "namespace": "prod-omi-backend", - "cloud_run_services": { - "backend": "backend", - "backend-sync": "backend-sync", - "backend-sync-backfill": "backend-sync-backfill", - "backend-integration": "backend-integration", - }, - }, - }, - created_at="2026-07-20T00:00:00+00:00", - ) - - -def test_release_record_requires_all_deployable_inputs() -> None: - record = _record() - - assert release_rings.validate_record(record) == [] - assert record["images"]["backend-listen"].endswith("@sha256:" + "a" * 64) - - -@pytest.mark.parametrize( - ("section", "key", "value", "expected"), - [ - ("images", "backend", "gcr.io/example/backend:latest", "immutable OCI digest"), - ( - "secret_versions", - "OPENAI_API_KEY", - "projects/example/secrets/key/versions/latest", - "numeric Secret Manager version", - ), - ("rendered_config", "beta/backend", "gs://release-records/config.json", "immutable GCS object reference"), - ], -) -def test_release_record_rejects_mutable_or_unrestorable_inputs( - section: str, key: str, value: str, expected: str -) -> None: - record = _record() - record[section][key] = value - - assert any(expected in error for error in release_rings.validate_record(record)) - - -def test_active_pointer_promote_and_hold_are_monotonic() -> None: - existing = { - "current_release_id": "2026-07-19.1", - "previous_verified_release_id": "2026-07-18.1", - "held_release_ids": [], - } - - promoted = release_rings.build_active_pointer( - ring="beta", release_id="2026-07-20.1", existing=existing, updated_at="2026-07-20T00:00:00+00:00" - ) - assert promoted["previous_verified_release_id"] == "2026-07-19.1" - - held = release_rings.build_active_pointer( - ring="beta", release_id="2026-07-20.2", existing=promoted, hold=True, updated_at="2026-07-20T00:01:00+00:00" - ) - assert held["current_release_id"] == "2026-07-20.1" - assert held["held_release_ids"] == ["2026-07-20.2"] - with pytest.raises(ValueError, match="held release"): - release_rings.build_active_pointer(ring="beta", release_id="2026-07-20.2", existing=held) - - -def test_recovery_pointer_uses_pre_mutation_state_after_a_late_failure() -> None: - before = { - "current_release_id": "2026-07-19.1", - "previous_verified_release_id": "2026-07-18.1", - "held_release_ids": [], - } - promoted = release_rings.build_active_pointer( - ring="prod", release_id="2026-07-20.1", existing=before, updated_at="2026-07-20T00:00:00+00:00" - ) - assert promoted["current_release_id"] == "2026-07-20.1" - - recovered = release_rings.build_active_pointer( - ring="prod", - release_id="2026-07-20.1", - existing=before, - hold=True, - updated_at="2026-07-20T00:01:00+00:00", - ) - - assert recovered["current_release_id"] == "2026-07-19.1" - assert recovered["previous_verified_release_id"] == "2026-07-18.1" - assert recovered["held_release_ids"] == ["2026-07-20.1"] - - -def test_receipt_keeps_partial_mutation_distinct_from_restoration() -> None: - receipt = release_rings.build_receipt( - ring="prod", - release_id="2026-07-20.1", - run_id="99", - state="partial_mutation", - snapshot_reference="gs://release-records/receipts/prod/snapshot.json#sha256:" + "d" * 64, - components={"cloud-run/backend": "restored", "gke/pusher": "restore_failed"}, - created_at="2026-07-20T00:00:00+00:00", - ) - - assert receipt["state"] == "partial_mutation" - - -def test_materialize_secret_versions_removes_latest_from_runtime_manifest(tmp_path: Path) -> None: - manifest = tmp_path / "runtime.yaml" - manifest.write_text( - "services:\n backend:\n secrets:\n API_KEY:\n secret: API_KEY\n version: latest\n", - encoding="utf-8", - ) - - materialized = release_rings.materialize_secret_versions( - manifest=manifest, - secret_versions={"API_KEY": "projects/example/secrets/API_KEY/versions/12"}, - ) - - assert materialized["services"]["backend"]["secrets"]["API_KEY"]["version"] == "12" - - -def test_materialize_runtime_config_captures_public_values_and_remote_secret_versions(tmp_path: Path) -> None: - manifest = tmp_path / "runtime.yaml" - manifest.write_text( - "network:\n subnet:\n env_var: CLOUD_RUN_VPC_SUBNET\nservices:\n backend:\n env:\n URL:\n env_var: API_URL\n" - " secrets:\n API_KEY:\n secret: API_KEY\n version: latest\n eso:\n remoteKey: OTHER_KEY\n", - encoding="utf-8", - ) - - materialized = release_rings.materialize_runtime_config( - manifest=manifest, - secret_versions={ - "API_KEY": "projects/example/secrets/API_KEY/versions/12", - "OTHER_KEY": "projects/example/secrets/OTHER_KEY/versions/13", - }, - public_values={"CLOUD_RUN_VPC_SUBNET": "subnet-a", "API_URL": "https://api-beta.example"}, - ) - - assert materialized["network"]["subnet"] == {"value": "subnet-a"} - assert materialized["services"]["backend"]["env"]["URL"] == {"value": "https://api-beta.example"} - assert materialized["services"]["backend"]["secrets"]["API_KEY"]["version"] == "12" - assert materialized["services"]["eso"]["version"] == "13" - - -def test_release_record_rejects_beta_that_reuses_prod_cloud_run_service_names() -> None: - record = _record() - record["topology"]["beta"]["cloud_run_services"] = record["topology"]["prod"]["cloud_run_services"] - - assert any("must not reuse prod service identities" in error for error in release_rings.validate_record(record)) diff --git a/backend/tests/unit/test_render_release_ring_config.py b/backend/tests/unit/test_render_release_ring_config.py deleted file mode 100644 index a95cc7b149a..00000000000 --- a/backend/tests/unit/test_render_release_ring_config.py +++ /dev/null @@ -1,48 +0,0 @@ -from __future__ import annotations - -import importlib.util -from pathlib import Path - -MODULE_PATH = Path(__file__).resolve().parents[2] / "scripts" / "render_release_ring_config.py" -SPEC = importlib.util.spec_from_file_location("render_release_ring_config", MODULE_PATH) -assert SPEC and SPEC.loader -renderer = importlib.util.module_from_spec(SPEC) -SPEC.loader.exec_module(renderer) - - -def test_beta_renderer_changes_stateless_identity_but_not_gpu_endpoint() -> None: - rendered = renderer.render( - { - "namespace": "prod-omi-backend", - "env": [{"name": "OMI_ENV_STAGE", "value": "prod"}], - "pusher": "http://pusher.omi.me", - "gpu": "http://parakeet.omi.me", - }, - ring="beta", - ) - - assert rendered["namespace"] == "beta-omi-backend" - assert rendered["env"][0]["value"] == "beta" - assert rendered["pusher"] == "http://pusher-beta.omi.me" - assert rendered["gpu"] == "http://parakeet.omi.me" - - -def test_prod_renderer_is_identity() -> None: - source = {"namespace": "prod-omi-backend", "value": "prod"} - - assert renderer.render(source, ring="prod") is source - - -def test_beta_renderer_rewrites_ingress_resources_and_materializes_beta_runtime_lane() -> None: - rendered = renderer.render_manifest( - { - "environments": {"prod": {"cloud_run": {"services": {"backend": {}}}}}, - "ingress": "prod-pusher-ilb-ip-address prod-agent-proxy-ip-address agent-proxy-cert", - }, - ring="beta", - ) - - assert set(rendered["environments"]) == {"beta"} - assert "beta-pusher-ilb-ip-address" in rendered["ingress"] - assert "beta-agent-proxy-ip-address" in rendered["ingress"] - assert "beta-agent-proxy-cert" in rendered["ingress"] diff --git a/backend/tests/unit/test_run_vpc_transcription_candidate_probe.py b/backend/tests/unit/test_run_vpc_transcription_candidate_probe.py new file mode 100644 index 00000000000..7b1782864b4 --- /dev/null +++ b/backend/tests/unit/test_run_vpc_transcription_candidate_probe.py @@ -0,0 +1,54 @@ +import importlib.util +import sys +from pathlib import Path + +import pytest + + +def _load(): + scripts = Path(__file__).resolve().parents[2] / "scripts" + sys.path.insert(0, str(scripts)) + spec = importlib.util.spec_from_file_location( + "vpc_candidate_probe", scripts / "run_vpc_transcription_candidate_probe.py" + ) + assert spec and spec.loader + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def test_requests_tagged_url_but_mints_identity_for_canonical_service(monkeypatch): + module = _load() + candidate = "https://candidate---backend-abc-uc.a.run.app" + audience = "https://backend-abc-uc.a.run.app" + monkeypatch.setenv("CANDIDATE_API_URL", candidate) + monkeypatch.setenv("CLOUD_RUN_IDENTITY_AUDIENCE", audience) + monkeypatch.setenv("FIREBASE_PROBE_TOKEN", "firebase-token") + seen = {} + + def identity_token(value): + seen["audience"] = value + return "identity-token" + + def build_report(config): + seen["request_url"] = config.api_url + return {"status": "PASS"} + + monkeypatch.setattr(module, "_identity_token", identity_token) + monkeypatch.setattr(module, "build_report", build_report) + + assert module.main() == 0 + assert seen == {"audience": audience, "request_url": candidate} + + +@pytest.mark.parametrize( + "name,value", [("CANDIDATE_API_URL", "http://candidate.invalid"), ("CLOUD_RUN_IDENTITY_AUDIENCE", "")] +) +def test_fails_closed_for_missing_or_invalid_request_or_identity_url(monkeypatch, name, value): + module = _load() + monkeypatch.setenv("CANDIDATE_API_URL", "https://candidate.invalid") + monkeypatch.setenv("CLOUD_RUN_IDENTITY_AUDIENCE", "https://backend.invalid") + monkeypatch.setenv("FIREBASE_PROBE_TOKEN", "firebase-token") + monkeypatch.setenv(name, value) + with pytest.raises(RuntimeError): + module.main() diff --git a/backend/tests/unit/test_sync_two_lane.py b/backend/tests/unit/test_sync_two_lane.py index 68a20fbbb02..315ed7b0ef8 100644 --- a/backend/tests/unit/test_sync_two_lane.py +++ b/backend/tests/unit/test_sync_two_lane.py @@ -266,21 +266,16 @@ def test_normal_backend_deploys_fail_closed_on_fence_transitions_and_gate_stt_ca assert 'Remove failed transcription candidate tag' in manual assert ( 'if: ${{ failure() && steps.deploy-backend.outcome == \'success\'' - ' && github.event.inputs.environment == \'development\' }}' in manual + " && (github.event.inputs.environment == 'development'" + " || github.event.inputs.deploy_targets == 'cloud-run-only') }}" in manual ) assert manual.index('Gate backend candidate on known audio') < manual.index( 'Shift Cloud Run traffic to validated revisions' ) -def test_transcription_candidate_gate_is_development_only(): - """The candidate probe must never gate a production deploy. - - The probe reaches the candidate over a Cloud Run tag URL, which is a run.app - URL. Production disables run.app URLs and restricts ingress to internal and - load-balancer traffic, so a production tag resolves to no HTTPS endpoint and - every production deploy fails at the probe (2026-07-17, run 29574215693). - """ +def test_production_cloud_run_only_uses_internal_authenticated_candidate_probe(): + """Prod Cloud Run promotion uses the VPC probe; full-stack prod rejects early.""" import yaml root = Path(__file__).resolve().parents[3] @@ -288,26 +283,27 @@ def test_transcription_candidate_gate_is_development_only(): steps = workflow['jobs']['deploy']['steps'] by_name = {step.get('name'): step for step in steps} - gate_steps = ( + public_probe = by_name['Gate backend candidate on known audio'] + assert public_probe['if'] == "${{ github.event.inputs.environment == 'development' }}" + + internal_probe = by_name['Gate internal production candidate on known audio from Cloud Run VPC'] + assert "github.event.inputs.environment == 'prod'" in internal_probe['if'] + assert "github.event.inputs.deploy_targets == 'cloud-run-only'" in internal_probe['if'] + assert 'probe-transcription-candidate-from-cloud-run.sh' in internal_probe['run'] + + for name in ( 'Resolve transcription candidate URL', - 'Gate backend candidate on known audio', 'Remove passed transcription candidate tag', 'Remove failed transcription candidate tag', - ) - for name in gate_steps: - assert name in by_name, f'{name} step is missing' + ): condition = by_name[name].get('if') or '' - assert ( - "github.event.inputs.environment == 'development'" in condition - ), f'{name} must be development-only; a production run cannot resolve a tag URL' + assert "github.event.inputs.environment == 'development'" in condition + assert "github.event.inputs.deploy_targets == 'cloud-run-only'" in condition - # The tag itself is only requested for development; production never creates one. deploy_flags = by_name['Deploy ${{ env.SERVICE }} to Cloud Run']['with']['flags'] - assert "github.event.inputs.environment == 'development' && format('--tag={0}'" in deploy_flags - - # Promotion must not depend on the skipped gate. - for name in ('Verify validated revisions are still current', 'Shift Cloud Run traffic to validated revisions'): - assert by_name[name].get('if') is None, f'{name} must still run when the gate is skipped' + assert "github.event.inputs.environment == 'development'" in deploy_flags + assert "github.event.inputs.deploy_targets == 'cloud-run-only'" in deploy_flags + assert steps.index(internal_probe) < steps.index(by_name['Shift Cloud Run traffic to validated revisions']) def test_static_processed_segment_marker_follows_partial_result_checkpoint(): diff --git a/backend/tests/unit/test_transcription_capability_probe.py b/backend/tests/unit/test_transcription_capability_probe.py index 2f725985a4f..9fa0b1e68c7 100644 --- a/backend/tests/unit/test_transcription_capability_probe.py +++ b/backend/tests/unit/test_transcription_capability_probe.py @@ -77,6 +77,7 @@ def _config(module, tmp_path, **overrides): 'manifest_path': manifest_path, 'api_url': 'https://candidate.invalid', 'bearer_token': 'probe-token-that-must-not-leak', + 'cloud_run_identity_token': None, 'timeout_seconds': 3.0, } values.update(overrides) @@ -118,6 +119,7 @@ def test_full_route_posts_versioned_fixture_and_validates_exact_candidate_contra assert request.get_method() == 'POST' headers = _headers(request) assert headers['authorization'] == 'Bearer probe-token-that-must-not-leak' + assert 'x-serverless-authorization' not in headers assert headers['content-type'].startswith('multipart/form-data; boundary=') assert b'name="files"; filename="transcription-release-probe.wav"' in request.data assert b'Content-Disposition: form-data; name="language"\r\n\r\nen\r\n' in request.data @@ -129,6 +131,20 @@ def test_full_route_posts_versioned_fixture_and_validates_exact_candidate_contra assert sensitive not in encoded +def test_full_route_uses_cloud_run_identity_and_firebase_auth_together(monkeypatch, tmp_path): + module = _load_module() + fake_urlopen, calls = _urlopen_responses(_Response(200, b'{"outcome":"success","transcript":"hello omi"}')) + monkeypatch.setattr(module.urllib.request, 'urlopen', fake_urlopen) + + report = module.build_report(_config(module, tmp_path, cloud_run_identity_token='serverless-token')) + + assert report['status'] == 'PASS' + headers = _headers(calls[0][0]) + assert headers['authorization'] == 'Bearer probe-token-that-must-not-leak' + assert headers['x-serverless-authorization'] == 'Bearer serverless-token' + assert 'serverless-token' not in json.dumps(report) + + def test_full_route_rejects_success_with_incorrect_transcript(monkeypatch, tmp_path): module = _load_module() fake_urlopen, _ = _urlopen_responses(_Response(200, b'{"outcome":"success","transcript":"hello omi extra"}')) @@ -156,6 +172,7 @@ def test_fixture_digest_mismatch_fails_closed_without_calling_candidate(monkeypa manifest_path=manifest_path, api_url='https://candidate.invalid', bearer_token='probe-token-that-must-not-leak', + cloud_run_identity_token=None, timeout_seconds=3.0, ) ) diff --git a/backend/tests/unit/test_verify_backend_release_vector.py b/backend/tests/unit/test_verify_backend_release_vector.py index 954e35eba74..a9caee61d45 100644 --- a/backend/tests/unit/test_verify_backend_release_vector.py +++ b/backend/tests/unit/test_verify_backend_release_vector.py @@ -771,7 +771,8 @@ def test_full_backend_deploys_verify_the_serving_release_vector_after_promotion( manual = (root / '.github' / 'workflows' / 'gcp_backend.yml').read_text(encoding='utf-8') manual_verification = manual[manual.index('Verify serving backend release vector') :] - assert "github.event.inputs.deploy_targets == 'all'" in manual_verification + assert "github.event.inputs.deploy_targets" in manual_verification + assert "--cloud-run-only" in manual_verification def test_backend_promotions_are_phase_aware_and_restore_the_recorded_traffic_snapshot() -> None: @@ -818,6 +819,58 @@ def test_backend_promotions_are_phase_aware_and_restore_the_recorded_traffic_sna assert 'cloud-run-traffic-restore.json' in evidence_upload +def test_production_cloud_run_only_boundary_is_early_and_uses_a_cleaned_up_dual_auth_vpc_probe(): + """Static workflow contract: prod/all cannot reach a mutating deploy step.""" + root = BACKEND_DIR.parent + workflow = (root / '.github/workflows/gcp_backend.yml').read_text(encoding='utf-8') + boundary = workflow.split('\n validate-production-boundary:\n', 1)[1].split('\n repair-traffic:\n', 1)[0] + + assert 'transactional GKE/config rollback parity does not exist' in boundary + assert workflow.index('validate-production-boundary') < workflow.index(' firestore_readiness:') + assert 'needs: validate-production-boundary' in workflow + assert 'needs: [validate-production-boundary, firestore_readiness]' in workflow + for forbidden in ('actions/checkout', 'google-github-actions/auth', 'docker build', 'docker push', 'gcloud run'): + assert forbidden not in boundary + + resolve = workflow.index('Resolve transcription candidate URL') + probe = workflow.index('Gate internal production candidate on known audio from Cloud Run VPC') + snapshot = workflow.index('Capture Cloud Run pre-promotion traffic snapshot') + traffic = workflow.index('Shift Cloud Run traffic to validated revisions') + verify = workflow.index('Verify serving backend release vector') + restore = workflow.index('Restore Cloud Run traffic snapshot after failed promotion') + assert resolve < probe < snapshot < traffic < verify < restore + assert '--tag={0}' in workflow + assert '--candidate-url "${{ steps.transcription-candidate.outputs.url }}"' in workflow + assert '--firebase-token-file "$RUNNER_TEMP/firebase-production-candidate-token"' in workflow + assert 'CLOUD_RUN_ONLY="--cloud-run-only"' in workflow + + probe_script = (root / 'backend/scripts/probe-transcription-candidate-from-cloud-run.sh').read_text( + encoding='utf-8' + ) + for required in ( + '--service-account="$SERVICE_ACCOUNT"', + '--network="$NETWORK"', + '--subnet="$SUBNET"', + '--vpc-egress=all-traffic', + 'compute networks subnets describe "$SUBNET"', + 'privateIpGoogleAccess', + '--role=roles/run.invoker', + 'trap cleanup EXIT', + 'gcloud run jobs delete', + 'remove-iam-policy-binding backend', + 'service-accounts delete', + ): + assert required in probe_script + + vpc_runner = (root / 'backend/scripts/run_vpc_transcription_candidate_probe.py').read_text(encoding='utf-8') + probe_source = (root / 'backend/scripts/transcription_capability_probe.py').read_text(encoding='utf-8') + assert 'X-Serverless-Authorization' in probe_source + assert 'FIREBASE_PROBE_TOKEN' in vpc_runner + assert 'identity_token = _identity_token(identity_audience)' in vpc_runner + assert 'api_url=candidate_url' in vpc_runner + assert 'cloud_run_identity_token=identity_token' in vpc_runner + + def test_backend_listen_rollout_wait_can_cover_a_real_rollout(): """The rollout wait must outlast a healthy roll, not just one pod's startup. diff --git a/backend/tests/unit/test_ws_b_short_term_lifecycle.py b/backend/tests/unit/test_ws_b_short_term_lifecycle.py index 10ecf6da1dc..56418d83f86 100644 --- a/backend/tests/unit/test_ws_b_short_term_lifecycle.py +++ b/backend/tests/unit/test_ws_b_short_term_lifecycle.py @@ -640,6 +640,93 @@ def test_required_processing_receipt_unlocks_durable_promotion(monkeypatch): assert stored["promotion"]["processing_receipt"]["processor_id"] == "canonical_required_memory" +def test_required_processing_does_not_extend_persisted_expiry_when_delayed(monkeypatch): + uid = "uid-canonical-required-delayed-processing" + _set_canonical_cohort(monkeypatch, uid) + db = _canonical_db_with_control(uid) + monkeypatch.setattr( + "utils.memory.canonical_memory_adapter.read_memory_v3_trusted_account_generation", + lambda **_: _trusted_account_generation(), + ) + payload = required_promotion_payload( + {"id": "manual-required-delayed-processing", "content": "Remember tea", "manually_added": True}, + source_surface="mcp", + ) + memory_id = write_canonical_extraction_memory(uid, payload, db_client=db) + item_path = f"users/{uid}/memory_items/{memory_id}" + captured_at = db.docs[item_path]["captured_at"] + original_expires_at = db.docs[item_path]["expires_at"] + + result = process_required_memory_item( + uid, + memory_id, + db_client=db, + processor=lambda _item: ProcessedRequiredMemory(content="The user prefers tea."), + now=captured_at + timedelta(days=10), + ) + + assert result.processed is True + assert db.docs[item_path]["expires_at"] == original_expires_at + + +def test_required_processing_preserves_expiry_when_clock_precedes_capture(monkeypatch): + uid = "uid-canonical-required-clock-skew" + _set_canonical_cohort(monkeypatch, uid) + db = _canonical_db_with_control(uid) + monkeypatch.setattr( + "utils.memory.canonical_memory_adapter.read_memory_v3_trusted_account_generation", + lambda **_: _trusted_account_generation(), + ) + payload = required_promotion_payload( + {"id": "manual-required-clock-skew", "content": "Remember tea", "manually_added": True}, + source_surface="mcp", + ) + memory_id = write_canonical_extraction_memory(uid, payload, db_client=db) + item_path = f"users/{uid}/memory_items/{memory_id}" + captured_at = db.docs[item_path]["captured_at"] + original_expires_at = db.docs[item_path]["expires_at"] + + result = process_required_memory_item( + uid, + memory_id, + db_client=db, + processor=lambda _item: ProcessedRequiredMemory(content="The user prefers tea."), + now=captured_at - timedelta(days=31), + ) + + assert result.processed is True + assert db.docs[item_path]["expires_at"] == original_expires_at + + +def test_required_processing_receipt_time_does_not_predate_capture(monkeypatch): + uid = "uid-canonical-required-receipt-clock-skew" + _set_canonical_cohort(monkeypatch, uid) + db = _canonical_db_with_control(uid) + monkeypatch.setattr( + "utils.memory.canonical_memory_adapter.read_memory_v3_trusted_account_generation", + lambda **_: _trusted_account_generation(), + ) + payload = required_promotion_payload( + {"id": "manual-required-receipt-clock-skew", "content": "Remember tea", "manually_added": True}, + source_surface="mcp", + ) + memory_id = write_canonical_extraction_memory(uid, payload, db_client=db) + item_path = f"users/{uid}/memory_items/{memory_id}" + captured_at = db.docs[item_path]["captured_at"] + + result = process_required_memory_item( + uid, + memory_id, + db_client=db, + processor=lambda _item: ProcessedRequiredMemory(content="The user prefers tea."), + now=captured_at - timedelta(days=31), + ) + + assert result.processed is True + receipt = db.docs[item_path]["promotion"]["processing_receipt"] + assert datetime.fromisoformat(receipt["processed_at"]) == captured_at + + def test_required_processing_receipt_is_bound_to_current_content_and_revision(monkeypatch): uid = "uid-canonical-required-stale-receipt" _set_canonical_cohort(monkeypatch, uid) diff --git a/backend/utils/memory/canonical_required_processing.py b/backend/utils/memory/canonical_required_processing.py index b895818a870..1c47879c8ce 100644 --- a/backend/utils/memory/canonical_required_processing.py +++ b/backend/utils/memory/canonical_required_processing.py @@ -27,7 +27,6 @@ from models.memory_operations import MemoryOperation, MemoryOperationType from models.product_memory import MemoryItem, MemoryItemStatus, MemoryLayer, ProcessingState from utils.memory.memory_system import MemorySystem, resolve_memory_system -from utils.memory.short_term_lifecycle import default_short_term_expiry from utils.memory.required_promotion import ( REQUIRED_PROCESSING_STATUS_FAILED_RETRYABLE, REQUIRED_PROCESSING_STATUS_PENDING, @@ -196,7 +195,7 @@ def _processing_receipt( "processor_id": REQUIRED_PROCESSOR_ID, "processor_version": REQUIRED_PROCESSOR_VERSION, "decision": "durable_required", - "processed_at": now.isoformat(), + "processed_at": max(now, item.captured_at).isoformat(), "input_hash": input_hash, "output_hash": output_hash, "input_item_revision": item.item_revision, @@ -300,7 +299,6 @@ def _apply_processed_result( "expected_item_revision": item.item_revision, "expected_content_hash": item.content_hash, "promotion_audit": promotion, - "expires_at": default_short_term_expiry(now).isoformat(), "subject_entity_id": processed.subject_entity_id, "predicate": processed.predicate, "arguments": processed.arguments, diff --git a/codemagic.yaml b/codemagic.yaml index 7d4f3dcdc7e..5b7a0180ec0 100644 --- a/codemagic.yaml +++ b/codemagic.yaml @@ -23,7 +23,6 @@ workflows: - shorebird vars: APP_ID: 6502156163 - STAGING_API_URL: "https://api-beta.omi.me/" flutter: 3.41.9 xcode: 26.0.1 cocoapods: 1.16.2 @@ -80,7 +79,7 @@ workflows: echo OPENAI_API_KEY=$OPENAI_API_KEY >> .env echo INSTABUG_API_KEY=$INSTABUG_API_KEY >> .env echo ONESIGNAL_APP_ID=$ONESIGNAL_APP_ID >> .env - echo API_BASE_URL=$API_BASE_URL >> .env + echo API_BASE_URL=https://api.omi.me/ >> .env echo GROWTHBOOK_API_KEY=$GROWTHBOOK_API_KEY >> .env echo GOOGLE_MAPS_API_KEY=$GOOGLE_MAPS_API_KEY >> .env echo INTERCOM_APP_ID=$INTERCOM_APP_ID >> .env @@ -88,7 +87,6 @@ workflows: echo POSTHOG_API_KEY=$POSTHOG_API_KEY >> .env echo GOOGLE_CLIENT_ID=$GOOGLE_CLIENT_ID >> .env echo GOOGLE_CLIENT_SECRET=$GOOGLE_CLIENT_SECRET >> .env - echo STAGING_API_URL=$STAGING_API_URL >> .env - name: Set up Google Service Info script: | @@ -175,7 +173,6 @@ workflows: flutter build ipa \ --release \ - --dart-define=OMI_BETA_RELEASE_RING=true \ --build-name=$BUILD_NAME \ --build-number=$BUILD_NUMBER \ --flavor prod \ @@ -242,7 +239,6 @@ workflows: PACKAGE_NAME: "com.friend.ios" APP_ID: 6502156163 JAVA_TOOL_OPTIONS: "-Xmx8g" - STAGING_API_URL: "https://api-beta.omi.me/" flutter: 3.41.9 xcode: 16.4 cocoapods: 1.16.2 @@ -330,7 +326,7 @@ workflows: echo OPENAI_API_KEY=$OPENAI_API_KEY >> .env echo INSTABUG_API_KEY=$INSTABUG_API_KEY >> .env echo ONESIGNAL_APP_ID=$ONESIGNAL_APP_ID >> .env - echo API_BASE_URL=$API_BASE_URL >> .env + echo API_BASE_URL=https://api.omi.me/ >> .env echo GROWTHBOOK_API_KEY=$GROWTHBOOK_API_KEY >> .env echo GOOGLE_MAPS_API_KEY=$GOOGLE_MAPS_API_KEY >> .env echo INTERCOM_APP_ID=$INTERCOM_APP_ID >> .env @@ -338,7 +334,6 @@ workflows: echo POSTHOG_API_KEY=$POSTHOG_API_KEY >> .env echo GOOGLE_CLIENT_ID=$GOOGLE_CLIENT_ID >> .env echo GOOGLE_CLIENT_SECRET=$GOOGLE_CLIENT_SECRET >> .env - echo STAGING_API_URL=$STAGING_API_URL >> .env - name: Get Flutter packages script: | @@ -416,7 +411,6 @@ workflows: flutter build appbundle \ --release \ - --dart-define=OMI_BETA_RELEASE_RING=true \ --build-name=$BUILD_NAME \ --build-number=$BUILD_NUMBER \ --flavor prod @@ -464,7 +458,6 @@ workflows: - shorebird vars: APP_ID: 6502156163 - STAGING_API_URL: "https://api-beta.omi.me/" flutter: 3.41.9 xcode: 26.0.1 cocoapods: 1.16.2 @@ -525,7 +518,7 @@ workflows: echo OPENAI_API_KEY=$OPENAI_API_KEY >> .env echo INSTABUG_API_KEY=$INSTABUG_API_KEY >> .env echo ONESIGNAL_APP_ID=$ONESIGNAL_APP_ID >> .env - echo API_BASE_URL=$API_BASE_URL >> .env + echo API_BASE_URL=https://api.omi.me/ >> .env echo GROWTHBOOK_API_KEY=$GROWTHBOOK_API_KEY >> .env echo GOOGLE_MAPS_API_KEY=$GOOGLE_MAPS_API_KEY >> .env echo INTERCOM_APP_ID=$INTERCOM_APP_ID >> .env @@ -533,7 +526,6 @@ workflows: echo POSTHOG_API_KEY=$POSTHOG_API_KEY >> .env echo GOOGLE_CLIENT_ID=$GOOGLE_CLIENT_ID >> .env echo GOOGLE_CLIENT_SECRET=$GOOGLE_CLIENT_SECRET >> .env - echo STAGING_API_URL=$STAGING_API_URL >> .env - name: Set up Google Service Info script: | @@ -567,7 +559,6 @@ workflows: BUILD_NUMBER=$(echo $CM_TAG | sed 's/^v\(.*\)+\(.*\)-mobile-cm$/\2/') flutter build ipa \ --release \ - --dart-define=OMI_BETA_RELEASE_RING=true \ --build-name=$BUILD_NAME \ --build-number=$BUILD_NUMBER \ --flavor prod \ @@ -705,7 +696,7 @@ workflows: echo OPENAI_API_KEY=$OPENAI_API_KEY >> .env echo INSTABUG_API_KEY=$INSTABUG_API_KEY >> .env echo ONESIGNAL_APP_ID=$ONESIGNAL_APP_ID >> .env - echo API_BASE_URL=$API_BASE_URL >> .env + echo API_BASE_URL=https://api.omi.me/ >> .env echo GROWTHBOOK_API_KEY=$GROWTHBOOK_API_KEY >> .env echo GOOGLE_MAPS_API_KEY=$GOOGLE_MAPS_API_KEY >> .env echo INTERCOM_APP_ID=$INTERCOM_APP_ID >> .env @@ -713,7 +704,6 @@ workflows: echo POSTHOG_API_KEY=$POSTHOG_API_KEY >> .env echo GOOGLE_CLIENT_ID=$GOOGLE_CLIENT_ID >> .env echo GOOGLE_CLIENT_SECRET=$GOOGLE_CLIENT_SECRET >> .env - echo STAGING_API_URL=$STAGING_API_URL >> .env - name: Get Flutter packages script: | @@ -843,7 +833,6 @@ workflows: vars: PACKAGE_NAME: "com.friend.ios" JAVA_TOOL_OPTIONS: "-Xmx8g" - STAGING_API_URL: "https://api-beta.omi.me/" flutter: 3.41.9 xcode: 16.4 cocoapods: 1.16.2 @@ -944,7 +933,7 @@ workflows: echo OPENAI_API_KEY=$OPENAI_API_KEY >> .env echo INSTABUG_API_KEY=$INSTABUG_API_KEY >> .env echo ONESIGNAL_APP_ID=$ONESIGNAL_APP_ID >> .env - echo API_BASE_URL=$API_BASE_URL >> .env + echo API_BASE_URL=https://api.omi.me/ >> .env echo GROWTHBOOK_API_KEY=$GROWTHBOOK_API_KEY >> .env echo GOOGLE_MAPS_API_KEY=$GOOGLE_MAPS_API_KEY >> .env echo INTERCOM_APP_ID=$INTERCOM_APP_ID >> .env @@ -952,7 +941,6 @@ workflows: echo POSTHOG_API_KEY=$POSTHOG_API_KEY >> .env echo GOOGLE_CLIENT_ID=$GOOGLE_CLIENT_ID >> .env echo GOOGLE_CLIENT_SECRET=$GOOGLE_CLIENT_SECRET >> .env - echo STAGING_API_URL=$STAGING_API_URL >> .env - name: Get Flutter packages script: | @@ -971,7 +959,6 @@ workflows: BUILD_NUMBER=$(echo $CM_TAG | sed 's/^v\(.*\)+\(.*\)-mobile-cm$/\2/') flutter build appbundle \ --release \ - --dart-define=OMI_BETA_RELEASE_RING=true \ --build-name=$BUILD_NAME \ --build-number=$BUILD_NUMBER \ --flavor prod @@ -1018,7 +1005,6 @@ workflows: - shorebird vars: APP_ID: 6502156163 - STAGING_API_URL: "https://api-beta.omi.me/" flutter: 3.41.9 xcode: 26.0.1 cocoapods: 1.16.2 @@ -1086,7 +1072,7 @@ workflows: echo OPENAI_API_KEY=$OPENAI_API_KEY >> .env echo INSTABUG_API_KEY=$INSTABUG_API_KEY >> .env echo ONESIGNAL_APP_ID=$ONESIGNAL_APP_ID >> .env - echo API_BASE_URL=$API_BASE_URL >> .env + echo API_BASE_URL=https://api.omi.me/ >> .env echo GROWTHBOOK_API_KEY=$GROWTHBOOK_API_KEY >> .env echo GOOGLE_MAPS_API_KEY=$GOOGLE_MAPS_API_KEY >> .env echo INTERCOM_APP_ID=$INTERCOM_APP_ID >> .env @@ -1094,7 +1080,6 @@ workflows: echo POSTHOG_API_KEY=$POSTHOG_API_KEY >> .env echo GOOGLE_CLIENT_ID=$GOOGLE_CLIENT_ID >> .env echo GOOGLE_CLIENT_SECRET=$GOOGLE_CLIENT_SECRET >> .env - echo STAGING_API_URL=$STAGING_API_URL >> .env - name: Set up Google Service Info script: | @@ -1199,7 +1184,6 @@ workflows: echo OPENAI_API_KEY=$OPENAI_API_KEY >> .env echo INSTABUG_API_KEY=$INSTABUG_API_KEY >> .env echo ONESIGNAL_APP_ID=$ONESIGNAL_APP_ID >> .env - echo API_BASE_URL=$API_BASE_URL >> .env echo GROWTHBOOK_API_KEY=$GROWTHBOOK_API_KEY >> .env echo GOOGLE_MAPS_API_KEY=$GOOGLE_MAPS_API_KEY >> .env echo INTERCOM_APP_ID=$INTERCOM_APP_ID >> .env @@ -1207,7 +1191,6 @@ workflows: echo POSTHOG_API_KEY=$POSTHOG_API_KEY >> .env echo GOOGLE_CLIENT_ID=$GOOGLE_CLIENT_ID >> .env echo GOOGLE_CLIENT_SECRET=$GOOGLE_CLIENT_SECRET >> .env - echo STAGING_API_URL=$STAGING_API_URL >> .env echo GEMINI_API_KEY=$GEMINI_API_KEY >> .env - name: Extract version (from tag or auto-increment) @@ -1758,7 +1741,6 @@ workflows: - shorebird vars: PACKAGE_NAME: "com.friend.ios" - STAGING_API_URL: "https://api-beta.omi.me/" flutter: 3.41.9 xcode: 16.4 cocoapods: 1.16.2 @@ -1866,7 +1848,7 @@ workflows: echo OPENAI_API_KEY=$OPENAI_API_KEY >> .env echo INSTABUG_API_KEY=$INSTABUG_API_KEY >> .env echo ONESIGNAL_APP_ID=$ONESIGNAL_APP_ID >> .env - echo API_BASE_URL=$API_BASE_URL >> .env + echo API_BASE_URL=https://api.omi.me/ >> .env echo GROWTHBOOK_API_KEY=$GROWTHBOOK_API_KEY >> .env echo GOOGLE_MAPS_API_KEY=$GOOGLE_MAPS_API_KEY >> .env echo INTERCOM_APP_ID=$INTERCOM_APP_ID >> .env @@ -1874,7 +1856,6 @@ workflows: echo POSTHOG_API_KEY=$POSTHOG_API_KEY >> .env echo GOOGLE_CLIENT_ID=$GOOGLE_CLIENT_ID >> .env echo GOOGLE_CLIENT_SECRET=$GOOGLE_CLIENT_SECRET >> .env - echo STAGING_API_URL=$STAGING_API_URL >> .env - name: Get Flutter packages script: | @@ -2940,14 +2921,11 @@ workflows: exit 0 fi set -euo pipefail - dispatch_key="codemagic:${CM_TAG}" - # The GitHub Actions workflow owns every release-body transition. - # Its per-candidate concurrency group is the authoritative atomic - # serialiser for claim/running/complete state; Codemagic must never - # replay a stale queued body over a runner-owned transition. + # Per-tag GitHub concurrency and immutable qualification evidence own + # duplicate dispatches; this bounded retry only handles dispatch I/O. for attempt in 1 2 3; do if gh workflow run desktop_qualify_beta.yml --repo "$GITHUB_REPO" \ - -f release_tag="$CM_TAG" -f dispatch_key="$dispatch_key"; then + -f release_tag="$CM_TAG"; then echo "Trusted qualification dispatch accepted for $CM_TAG on attempt $attempt." exit 0 fi diff --git a/config/deployment-setting-classification.json b/config/deployment-setting-classification.json index 933aa5c0986..0388ac19247 100644 --- a/config/deployment-setting-classification.json +++ b/config/deployment-setting-classification.json @@ -42,7 +42,6 @@ "PINECONE_API_KEY", "RAPID_API_KEY", "REDIS_DB_PASSWORD", - "RELEASE_PAGER_WEBHOOK", "SERVICE_ACCOUNT_JSON", "SLIDESGPT_API_KEY", "SPARKLE_PRIVATE_KEY", @@ -61,6 +60,7 @@ "ACCOUNT_DELETION_DISPATCH_MODE", "ACCOUNT_DELETION_HANDLER_URL", "ACCOUNT_DELETION_TASKS_QUEUE", + "BACKEND_CLOUD_RUN_IAM_AUDIENCE", "BACKEND_SECRETS_GSA", "CLOUD_RUN_VPC_NETWORK", "CLOUD_RUN_VPC_SUBNET", @@ -102,13 +102,6 @@ "PARAKEET_STREAM_CAPACITY", "RAPID_API_HOST", "REDIS_DB_HOST", - "RELEASE_ARTIFACT_PROJECT_ID", - "RELEASE_GKE_CLUSTER", - "RELEASE_RECORDS_BUCKET", - "RELEASE_RECORDS_WRITER_SERVICE_ACCOUNT", - "RELEASE_RINGS_WIF_PROVIDER", - "RELEASE_RING_DEPLOYER_SERVICE_ACCOUNT", - "RELEASE_RUNTIME_PROJECT_ID", "RUNTIME_GCP_PROJECT_ID", "STRIPE_PLUS_ANNUAL_PRICE_ID", "STRIPE_PLUS_MONTHLY_PRICE_ID", diff --git a/desktop/macos/AGENTS.md b/desktop/macos/AGENTS.md index 87b8203a0d7..523d8562a8b 100644 --- a/desktop/macos/AGENTS.md +++ b/desktop/macos/AGENTS.md @@ -71,14 +71,11 @@ Signed artifact smoke scope: - Artifact creation and user visibility are split: create/upload the immutable candidate first, then advance beta/stable visibility only after digest-matched qualification passes. - Automatic beta is fail-closed: any signed-smoke, digest, static, T2, fault-suite, newest-tag, manifest, or pointer failure leaves the candidate non-live. Set `DESKTOP_AUTO_BETA_ENABLED=false` in Codemagic or the GitHub `prod` environment to pause automatic qualification/promotion without changing stable. -Stable/prod is manual: -- Automatic qualification never nominates or promotes stable. Stable workflows remain `workflow_dispatch` only. -- Nominate the current qualified beta with `desktop_nominate_stable_candidate.yml`. Nomination records the tag/SHA, operator, rationale, soak review, telemetry review, release-note review, and qualification evidence. It never changes beta/stable pointers or deploys production. -- Before preparing stable/prod promotion, follow `docs/agent-prod-promotion-runbook.md` for target discovery, curated stable release-log creation, shared-backend coupling, approval shape, and deterministic post-promotion checks. External readiness is handled separately. -- Run GitHub Actions workflow `desktop_promote_prod.yml` with the nominated `release_tag=v*-macos` stable candidate and `confirm=promote-stable`. -- The workflow runs `.github/scripts/check-desktop-release-promotion.py`, deploys the Rust backend from that exact tag, verifies `/health` reports the release tag/SHA, promotes the Firestore bridge release, marks the GitHub release `channel: stable`, then moves `desktop-backend-prod-deployed`. -- Do not manually edit a release to stable before the backend is promoted; the promotion workflow owns that mutation. -- The promotion workflow is roll-forward only. Stable rollback needs a newer fixed release or a separate manual infrastructure rollback plan, because both desktop feeds choose the newest stable app release. +Stable is manual: +- Automatic qualification never promotes Stable. `desktop_promote_prod.yml` remains `workflow_dispatch` only and protected by the `prod` environment. +- Run it with the current qualified Beta `release_tag`, `confirm=promote-stable`, and the observed Stable pointer release ID/generation. It verifies the exact Beta artifact, advances only the Stable pointer, updates the existing legacy/static bridges, and verifies hashes and feed output. +- `operation=repoint` selects a retained qualified manifest with exact current pointer CAS. It cannot downgrade clients Sparkle has already upgraded; issue a higher-version hotfix for those clients. +- Backend deployments remain independent in their established workflows. Do not manually edit release visibility or pointers outside the promotion workflow. **Codemagic CLI & API:** - Token: `$CODEMAGIC_API_TOKEN` (set in `~/.zshrc`) diff --git a/desktop/macos/CHANGELOG.json b/desktop/macos/CHANGELOG.json index e34afa24f16..0b720d49e46 100644 --- a/desktop/macos/CHANGELOG.json +++ b/desktop/macos/CHANGELOG.json @@ -6,7 +6,7 @@ "date": "2026-07-20", "changes": [ "Improved reliability of macOS beta update qualification so a temporary release-service outage does not create duplicate checks or silently promote an unqualified build.", - "Beta-channel desktop builds now route to the dedicated beta release ring instead of the development backend", + "Omi Beta uses the production backend while retaining its separate app identity and update feed", "Fixed the chat fault suite so injected backend errors surface the real HTTP 500 instead of a generic bridge failure", "Repaired live transcription routing so Parakeet admission failures fall back to Modulate instead of leaving the stream without a provider", "Added Mac-specific provenance to synced screen activity so multi-Mac captures stay distinct", diff --git a/desktop/macos/Desktop/Sources/AppState.swift b/desktop/macos/Desktop/Sources/AppState.swift index 565d00a6295..93b6cec3a21 100644 --- a/desktop/macos/Desktop/Sources/AppState.swift +++ b/desktop/macos/Desktop/Sources/AppState.swift @@ -542,7 +542,7 @@ class AppState: ObservableObject { // singletons that read the key directly. UserDefaults.standard.set(false, forKey: "desktop_isPaywalled") - // Resolve beta/stable before loading backend URLs so beta releases use dev services. + // Resolve the production identity before loading its shared production backend URL. AppBuild.prepareUpdateChannelForBackendRouting() // Load API key from environment or .env file diff --git a/desktop/macos/Desktop/Sources/DesktopAutomationBridge.swift b/desktop/macos/Desktop/Sources/DesktopAutomationBridge.swift index 6a8afc55c06..dca3c1f8b86 100644 --- a/desktop/macos/Desktop/Sources/DesktopAutomationBridge.swift +++ b/desktop/macos/Desktop/Sources/DesktopAutomationBridge.swift @@ -3710,8 +3710,7 @@ final class DesktopAutomationBridge: @unchecked Sendable { logLaunchID: omiLogLaunchID(), bridgePort: DesktopAutomationLaunchOptions.port, requiresAuth: true, - backendEnvironment: DesktopBackendEnvironment.shouldUseBetaRingBackends - ? "beta" : (DesktopBackendEnvironment.shouldUseDevelopmentBackends ? "development" : "production"), + backendEnvironment: DesktopBackendEnvironment.shouldUseDevelopmentBackends ? "development" : "production", pythonBackendURL: DesktopBackendEnvironment.pythonBaseURL(), rustBackendURL: DesktopBackendEnvironment.rustBackendURL(), agentRuntimeRunning: runtime.running, diff --git a/desktop/macos/Desktop/Sources/DesktopBackendEnvironment.swift b/desktop/macos/Desktop/Sources/DesktopBackendEnvironment.swift index 1460e2c2b5f..419b7b19184 100644 --- a/desktop/macos/Desktop/Sources/DesktopBackendEnvironment.swift +++ b/desktop/macos/Desktop/Sources/DesktopBackendEnvironment.swift @@ -2,33 +2,9 @@ import Foundation enum DesktopBackendEnvironment { static let productionPythonAPIURL = "https://api.omi.me/" - static let betaPythonAPIURL = "https://api-beta.omi.me/" static let developmentPythonAPIURL = "https://api.omiapi.com/" static let developmentRustBackendURL = "https://desktop-backend-dt5lrfkkoa-uc.a.run.app/" - static var shouldUseBetaRingBackends: Bool { - shouldUseBetaRingBackends( - bundleIdentifier: AppBuild.bundleIdentifier, - updateChannel: AppBuild.currentUpdateChannel, - forceOverride: currentEnvironmentValue("OMI_FORCE_DEV_BACKENDS"), - externalPreviewBackend: AppBuild.externalPreviewBackend - ) - } - - static func shouldUseBetaRingBackends( - bundleIdentifier: String, - updateChannel: String, - forceOverride: String? = nil, - externalPreviewBackend: AppBuild.ExternalPreviewBackend? = nil - ) -> Bool { - guard !AppBuild.isExternalPreviewBundleIdentifier(bundleIdentifier) else { return false } - guard !isAffirmative(forceOverride) else { return false } - // Both production-family identities ride the beta ring on the beta channel; - // the separately-installable Omi Beta app pins its channel to beta. - return AppBuild.productionFamilyBundleIdentifiers.contains(bundleIdentifier) - && normalizedChannel(updateChannel) == "beta" - } - static var shouldUseDevelopmentBackends: Bool { shouldUseDevelopmentBackends( bundleIdentifier: AppBuild.bundleIdentifier, @@ -59,8 +35,6 @@ enum DesktopBackendEnvironment { return true } - // Production beta now has a dedicated stateless release ring. It selects - // beta URLs below rather than inheriting the development backend. if isAffirmative(forceOverride) { return true } @@ -73,23 +47,18 @@ enum DesktopBackendEnvironment { ) -> String { pythonBaseURL( useDevelopmentBackends: shouldUseDevelopmentBackends, - useBetaRingBackends: shouldUseBetaRingBackends, environmentValue: environmentValue ) } static func pythonBaseURL( useDevelopmentBackends: Bool, - useBetaRingBackends: Bool = false, environmentValue: String? ) -> String { if let url = normalizedURL(environmentValue) { return url } - if useBetaRingBackends { - return betaPythonAPIURL - } if useDevelopmentBackends { return developmentPythonAPIURL } @@ -116,7 +85,6 @@ enum DesktopBackendEnvironment { ) -> String { rustBackendURL( useDevelopmentBackends: shouldUseDevelopmentBackends, - useBetaRingBackends: shouldUseBetaRingBackends, environmentValue: environmentValue, launchEnvironmentValue: launchEnvironmentValue ) @@ -124,7 +92,6 @@ enum DesktopBackendEnvironment { static func rustBackendURL( useDevelopmentBackends: Bool, - useBetaRingBackends: Bool = false, environmentValue: String?, launchEnvironmentValue: String? ) -> String { @@ -136,10 +103,6 @@ enum DesktopBackendEnvironment { return url } - // The Rust desktop backend has its own, currently roll-forward-only - // release path. v1's stateless beta ring is the Python/GKE path; leave - // this unset so the signed production bundle's explicit Rust endpoint - // remains authoritative rather than inventing an unprovisioned beta host. if useDevelopmentBackends { return developmentRustBackendURL } @@ -148,9 +111,6 @@ enum DesktopBackendEnvironment { } static func applyReleaseChannelDefaults() { - if shouldUseBetaRingBackends, normalizedURL(currentEnvironmentValue("OMI_PYTHON_API_URL")) == nil { - setenv("OMI_PYTHON_API_URL", betaPythonAPIURL, 1) - } if shouldUseDevelopmentBackends { if normalizedURL(currentEnvironmentValue("OMI_PYTHON_API_URL")) == nil { setenv("OMI_PYTHON_API_URL", developmentPythonAPIURL, 1) @@ -162,11 +122,6 @@ enum DesktopBackendEnvironment { log("BackendEnvironment: release-channel defaults applied only for missing backend URLs") } - private static func normalizedChannel(_ channel: String) -> String { - let normalized = channel.trimmingCharacters(in: .whitespacesAndNewlines).lowercased() - return normalized == "staging" ? "beta" : normalized - } - private static func normalizedURL(_ raw: String?) -> String? { guard let raw else { return nil } let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines) diff --git a/desktop/macos/Desktop/Sources/Services/APIClient/APIClient+Settings.swift b/desktop/macos/Desktop/Sources/Services/APIClient/APIClient+Settings.swift index e6a8b366f2d..860b034049f 100644 --- a/desktop/macos/Desktop/Sources/Services/APIClient/APIClient+Settings.swift +++ b/desktop/macos/Desktop/Sources/Services/APIClient/APIClient+Settings.swift @@ -536,9 +536,8 @@ struct UserSubscriptionResponse: Codable { } // Defensive decode: only `subscription` is required. The usage counters and - // plan catalog default when absent so a backend that's behind on schema - // (notably the dev backend the beta channel routes to, which can lag prod or - // omit newer fields like `memories_created_used`) doesn't blank the entire + // plan catalog default when absent so a backend that's behind on schema and + // omits newer fields like `memories_created_used` doesn't blank the entire // Plan & Usage page with "Failed to load plan information." init(from decoder: Decoder) throws { let c = try decoder.container(keyedBy: CodingKeys.self) diff --git a/desktop/macos/Desktop/Sources/TranscriptionService.swift b/desktop/macos/Desktop/Sources/TranscriptionService.swift index c29a0512dd6..56af0e8754a 100644 --- a/desktop/macos/Desktop/Sources/TranscriptionService.swift +++ b/desktop/macos/Desktop/Sources/TranscriptionService.swift @@ -116,7 +116,7 @@ class TranscriptionService: @unchecked Sendable { private let clientConversationId: String? /// Python backend base URL for transcription endpoints. - /// Resolution order: beta release channel → OMI_PYTHON_API_URL → https://api.omi.me/ + /// Resolution order: explicit OMI_PYTHON_API_URL → production https://api.omi.me/ /// NOTE: Do NOT fall back to OMI_DESKTOP_API_URL — that points to the Rust desktop-backend /// (Cloud Run), which does not have /v2/voice-message/* or /v4/listen endpoints. private static let pythonBackendBaseURL: String = DesktopBackendEnvironment.pythonBaseURL() diff --git a/desktop/macos/Desktop/Tests/APIClientRoutingTests.swift b/desktop/macos/Desktop/Tests/APIClientRoutingTests.swift index 71e3d887297..a19af54f468 100644 --- a/desktop/macos/Desktop/Tests/APIClientRoutingTests.swift +++ b/desktop/macos/Desktop/Tests/APIClientRoutingTests.swift @@ -256,35 +256,27 @@ final class APIClientRoutingTests: XCTestCase { XCTAssertEqual(url, "https://desktop-backend-hhibjajaja-uc.a.run.app/") } - func testBetaProductionBundleUsesDedicatedRingRatherThanDevelopment() { + func testBetaProductionChannelUsesProductionBackendRatherThanDevelopment() { XCTAssertFalse( DesktopBackendEnvironment.shouldUseDevelopmentBackends( bundleIdentifier: "com.omi.computer-macos", updateChannel: "beta" )) - // "staging" is normalized to "beta" — same ring. XCTAssertFalse( DesktopBackendEnvironment.shouldUseDevelopmentBackends( bundleIdentifier: "com.omi.computer-macos", updateChannel: "staging" )) - XCTAssertTrue( - DesktopBackendEnvironment.shouldUseBetaRingBackends( - bundleIdentifier: "com.omi.computer-macos", - updateChannel: "beta" - )) XCTAssertEqual( DesktopBackendEnvironment.pythonBaseURL( useDevelopmentBackends: false, - useBetaRingBackends: true, environmentValue: nil ), - "https://api-beta.omi.me/" + "https://api.omi.me/" ) XCTAssertEqual( DesktopBackendEnvironment.rustBackendURL( useDevelopmentBackends: false, - useBetaRingBackends: true, environmentValue: nil, launchEnvironmentValue: nil ), @@ -300,25 +292,18 @@ final class APIClientRoutingTests: XCTestCase { )) } - func testBetaIdentityBundleRoutesByChannelLikeProduction() { - // The Omi Beta app is production-family: it must not take the named-dev-bundle - // always-dev branch, and on its pinned beta channel it rides the beta release - // ring exactly like a stable-identity beta-channel client. + func testBetaIdentityBundleUsesTheProductionBackend() { + // The Omi Beta app is production-family: its isolated app identity does not + // create a second backend environment. XCTAssertFalse( DesktopBackendEnvironment.shouldUseDevelopmentBackends( bundleIdentifier: AppBuild.betaProductionBundleIdentifier, updateChannel: "beta" )) - XCTAssertTrue( - DesktopBackendEnvironment.shouldUseBetaRingBackends( - bundleIdentifier: AppBuild.betaProductionBundleIdentifier, - updateChannel: "beta" - )) - XCTAssertFalse( - DesktopBackendEnvironment.shouldUseBetaRingBackends( - bundleIdentifier: AppBuild.betaProductionBundleIdentifier, - updateChannel: "stable" - )) + XCTAssertEqual( + DesktopBackendEnvironment.pythonBaseURL(useDevelopmentBackends: false, environmentValue: nil), + "https://api.omi.me/" + ) } func testNonProductionBundlesDefaultToDevelopmentBackends() { diff --git a/desktop/macos/changelog/releases/0.12.90.json b/desktop/macos/changelog/releases/0.12.90.json index 88b16599fc4..994e3f83f96 100644 --- a/desktop/macos/changelog/releases/0.12.90.json +++ b/desktop/macos/changelog/releases/0.12.90.json @@ -3,7 +3,7 @@ "date": "2026-07-20", "changes": [ "Improved reliability of macOS beta update qualification so a temporary release-service outage does not create duplicate checks or silently promote an unqualified build.", - "Beta-channel desktop builds now route to the dedicated beta release ring instead of the development backend", + "Omi Beta uses the production backend while retaining its separate app identity and update feed", "Fixed the chat fault suite so injected backend errors surface the real HTTP 500 instead of a generic bridge failure", "Repaired live transcription routing so Parakeet admission failures fall back to Modulate instead of leaving the stream without a provider", "Added Mac-specific provenance to synced screen activity so multi-Mac captures stay distinct", diff --git a/desktop/macos/changelog/unreleased/20260720-beta-production-backend.json b/desktop/macos/changelog/unreleased/20260720-beta-production-backend.json new file mode 100644 index 00000000000..50295519d03 --- /dev/null +++ b/desktop/macos/changelog/unreleased/20260720-beta-production-backend.json @@ -0,0 +1,3 @@ +{ + "change": "Omi Beta now uses the same reliable production backend as Omi while keeping its separate Beta app and update channel" +} diff --git a/desktop/macos/docs/agent-prod-promotion-runbook.md b/desktop/macos/docs/agent-prod-promotion-runbook.md index 57e0bb7d737..614bcd9e016 100644 --- a/desktop/macos/docs/agent-prod-promotion-runbook.md +++ b/desktop/macos/docs/agent-prod-promotion-runbook.md @@ -1,272 +1,14 @@ -# Agent macOS Desktop Prod Promotion Runbook +# Desktop Stable Pointer Promotion -This runbook is for agents preparing an Omi macOS Desktop qualified beta for stable-candidate nomination and stable/prod promotion. It consolidates the repo-facing process only: release discovery, qualification evidence, nomination review, stable-rollup release-log creation, backend coupling assessment, approval shape, workflow dispatch, and deterministic post-promotion checks. +Stable promotion is a protected distribution-pointer operation. It never deploys a desktop backend or creates a second backend environment. -Do **not** put external promotion readiness here. Fleet health/readiness checks are performed separately before approval. +1. Read the current `macos-beta` and `macos-stable` pointer release IDs and generations. +2. Confirm the requested tag is the current qualified Beta release and retain the successful trusted `desktop_qualify_beta.yml` run ID. That GitHub Actions artifact records all four exact Stable/Beta ZIP/DMG URLs, hashes, and Sparkle signatures. +3. Run `desktop_promote_prod.yml` in the protected `prod` environment with that `qualification_run_id`, `confirm=promote-stable`, the observed Stable `expected_current_release_id` and `expected_generation`, and `operation=promote`. +4. The workflow reads the immutable manifest already retained by Beta, advances the Stable pointer, updates the legacy appcast/static bridge, and verifies pointer identity, hashes, and feed output. -## Hard rules +For a controlled repoint, use `operation=repoint` with the target's exact currently observed Stable release ID and generation. The target may be any retained passed-T2 immutable manifest; it need not first become current Beta. Repointing reads that already-retained manifest; it does not require a `qualification_run_id` or register a manifest. Repointing stops future rollout; Sparkle does not downgrade clients already on a higher build, so ship a higher-version hotfix for those clients. -- Do not promote, deploy, mutate release metadata, clear caches, edit Firestore, or shift traffic until David explicitly approves the exact current plan. -- Stable promotion is macOS Desktop only. It does not deploy mobile. -- Stable promotion is roll-forward-oriented. Recovery usually means promoting a newer fixed desktop release or performing a separately approved manual infrastructure rollback. -- Do not manually edit stable-candidate or stable metadata. The nomination and promotion workflows own those mutations. -- Do not use hotfix/cherry-pick branches for normal desktop rollouts. Use the latest eligible mainline `v*-macos` artifact. -- When promoting the release that ships the desktop-agent-platonic branch, record the concrete ship+2 burn version numbers for gap-closure G6 (legacy shims, `legacy_default` grant source, sqlite legacy columns) in the release notes or a maintainer tracking issue. +The GitHub Actions run is the audit trail. Do not edit a release body, Firestore pointer, static route, or legacy bridge manually. Backend deployments remain separate established workflows. -Lifecycle vocabulary is exact: `main → build candidate → qualified beta → stable candidate → stable`. Qualification and nomination leave stable visibility unchanged. Only `desktop_promote_prod.yml` makes the final stable decision. - -## 1. Discover release state - -Run from the repo root: - -```bash -git fetch origin main --tags --prune -gh release list --repo BasedHardware/omi --limit 20 -git tag -l 'v*-macos' --sort=-v:refname | head -20 -curl -fsS https://api.omi.me/v2/desktop/appcast.xml -``` - -Identify and report: - -- current stable appcast version/build/tag; -- current beta/dev appcast version/build/tag; -- newest `v*-macos` tag; -- newest published GitHub macOS release; -- whether a newer auto-release or Codemagic build is currently in flight. - -A tag is eligible for nomination only after Codemagic has published the immutable ZIP/DMG build candidate, the automatic signed-digest, static, exact-tag T2, and fault-suite gates have passed, and the beta workflow has promoted its explicit beta pointer. Manual `qualify-desktop-beta.sh` remains a recovery path. If the newest tag is still a build candidate, queued, building, or unqualified, hold and report it as not ready. - -## 2. Verify release artifact alignment - -For the target tag: - -```bash -RELEASE_TAG='vX.Y.Z+BUILD-macos' -SHA=$(git rev-list -n1 "$RELEASE_TAG") - -gh api "repos/BasedHardware/omi/commits/$SHA/check-runs" \ - --jq '.check_runs[] | select(.name=="Release OMI Desktop (Swift)") | {name,status,conclusion,started_at,completed_at,details_url,html_url}' - -gh release view "$RELEASE_TAG" \ - --repo BasedHardware/omi \ - --json tagName,name,publishedAt,isDraft,isPrerelease,url,assets,body -``` - -Confirm: - -- Codemagic `Release OMI Desktop (Swift)` is completed/success; -- Codemagic ran `Smoke signed desktop artifact` before `Create GitHub release`; -- Codemagic uploaded `desktop-smoke-result.json`, and its tag/version/build/artifact digests match the release assets; -- GitHub Release exists and is not draft; -- assets include `Omi.zip` and `omi.dmg`; -- release body has `isLive: true`, `channel: beta`, and an `edSignature`; -- release metadata includes canonical `qualifiedBeta: true`, `qualifiedBetaTier: 2`, `qualifiedBetaSha` matching the tag commit, `qualifiedBetaAt`, and a published `qualifiedBetaEvidence` asset; legacy `blessed*` metadata remains valid for releases qualified before the migration; -- live appcast beta/dev item points to the same build. - -For high-risk desktop auth/runtime releases, run the live signed-artifact canary -before promotion: - -```bash -desktop/macos/scripts/smoke-signed-desktop-artifact.sh \ - --zip /path/to/Omi.zip \ - --dmg /path/to/omi.dmg \ - --tag "$RELEASE_TAG" \ - --expected-channel beta \ - --launch --network --auth --chat --permissions --storage --quarantine \ - --result-json /tmp/desktop-live-smoke-result.json -``` - -The live canary intentionally requires explicit environment: -`OMI_SIGNED_ARTIFACT_SMOKE_ALLOW_PRODUCTION_LAUNCH=1` and -`OMI_SIGNED_ARTIFACT_SMOKE_AUTH_PROOF_COMMAND='...'`. The auth proof command -must verify the launched app's real persistence path (Keychain write/read, -restart, and authenticated API), not only curl an API with an injected bearer -token. `OMI_SIGNED_ARTIFACT_SMOKE_AUTH_HEADER='Bearer ...'` is only for the -separate minimal chat endpoint probe. - -Beta exposure is gated: Codemagic uploads a non-live candidate, verifies the published -digests against signed-smoke evidence, and runs static checks, exact-tag T2, and the -fault-injection suite. Only the beta promotion workflow can advance visibility, and -automatic requests must still target the newest tag. Stable nomination and promotion -remain manual-only. -For stable promotion, add an upgrade-path canary: previous signed release signed -in → update to candidate → restart → auth, helper runtime, and local storage -still work. - -## 3. Build the curated stable release log - -The per-build changelog is automatic, but stable promotion can span many beta/dev patch builds. Before approval, build a curated stable release log from the current stable appcast build to the target beta/dev build. - -Inputs: - -- current stable appcast build/tag; -- target beta/dev build/tag; -- `desktop/macos/changelog/releases/*.json` entries between those versions; -- GitHub release bodies only when changelog files are missing or need clarification. - -Suggested raw aggregation: - -```bash -python3 - <<'PY' -import json -import re -from pathlib import Path - -FROM_PATCH = 508 # first patch after current stable, update per rollout -TO_PATCH = 596 # target patch, update per rollout - -for path in sorted(Path('desktop/macos/changelog/releases').glob('0.11.*.json')): - match = re.match(r'0\.11\.(\d+)\.json$', path.name) - if not match: - continue - patch = int(match.group(1)) - if FROM_PATCH <= patch <= TO_PATCH: - data = json.loads(path.read_text()) - print(f"## {data['version']} ({data['date']})") - for change in data.get('changes', []): - print(f"- {change}") -PY -``` - -Curate the final stable release log into these buckets: - -- **Headline**: one sentence naming the release theme. -- **Integrations and connectors**: ChatGPT, Claude, Gmail, Google Calendar, Apple Notes, Hermes, OpenClaw, local MCP, wearable/device status. -- **Memory and data reliability**: memory import, legacy memory decode, memory-bank setup, cache consistency. -- **Voice, recording, and Rewind**: push-to-talk, realtime voice, recording reconciliation, screenshot/rewind fixes. -- **Auth, update, and runtime reliability**: sign-in/OAuth, updater recovery, agent runtime packaging, keychain/device identity. -- **UI polish and usability**: notch/floating bar, menus, labels, guidance cards. -- **Known caveats / operator notes**: backend dependencies, fail-closed behavior, rollout caveats. - -Include this curated log in the approval plan. Do not rely on the target release's per-build notes alone when the target spans multiple beta/dev builds since stable. - -## 4. Decide shared backend coupling - -Desktop promotion always deploys the **Rust `desktop-backend`** from the exact `v*-macos` tag via `desktop_promote_prod.yml`. - -The **shared Python backend** is separate (`gcp_backend.yml`). It is not always required, but many desktop capabilities are coupled to shared backend routes, schemas, OAuth clients, feature flags, or response contracts. - -Before approval, classify shared backend as: - -```text -backend_required: yes | no | optional -reason: - - desktop feature or PR - - backend endpoint/schema/env needed - - whether current prod backend already contains it -phase: - - before desktop promotion | after desktop promotion | not needed -``` - -Use concrete evidence: - -```bash -# Current prod backend image/revisions, read-only -python3 backend/scripts/deploy_status_report.py \ - --env prod \ - --project based-hardware \ - --region us-central1 \ - --include-cloud-run \ - --include-gke \ - --cloud-run-service backend \ - --cloud-run-service backend-sync \ - --cloud-run-service backend-integration \ - --cloud-run-service desktop-backend \ - --gke-service backend-listen \ - --gke-service pusher - -# Compare desktop target against current prod backend source when known. -git log --oneline --first-parent .. -- backend pusher parakeet modal llm_gateway agent-proxy -``` - -Rules of thumb: - -- If the desktop app fail-closes against older backend behavior and the new capability is non-critical, backend may be optional. -- If a desktop flow relies on a new endpoint, query parameter echo, response field, OAuth client, runtime env, or server-side policy, backend is required for full capability. -- If backend is required, propose it as a separately approved phase. Do not dispatch `gcp_backend.yml` without explicit approval. - -## 5. Nominate the stable candidate - -Nomination is a non-customer-facing decision record. It validates that the release is a qualified beta, equals the current macOS beta pointer, and matches that pointer's immutable manifest source SHA. It then records operator, rationale, qualification evidence, soak review, telemetry review, and release-note review. It does not move beta or stable, deploy a backend, or change customer visibility. - -```bash -RELEASE_TAG='vX.Y.Z+BUILD-macos' - -gh workflow run desktop_nominate_stable_candidate.yml \ - --repo BasedHardware/omi \ - -f release_tag="$RELEASE_TAG" \ - -f rationale='Why this beta is ready for stable consideration' \ - -f soak_review='Duration, cohort, and outcome reviewed' \ - -f telemetry_review='Crash, update, and health telemetry reviewed' \ - -f release_notes_review='Stable rollup reviewed' -``` - -After the workflow completes, verify the release has matching `stableCandidateTag`, `stableCandidateSha`, `stableCandidateAt`, `stableCandidateBy`, `stableCandidateRationale`, `stableCandidateQualificationEvidence`, `stableCandidateSoakReview`, `stableCandidateTelemetryReview`, and `stableCandidateReleaseNotesReview`. Verify beta and stable appcasts are unchanged. - -## 6. Present promotion approval plan - -Before any mutation, present: - -- target release tag/build/version; -- why it is the latest eligible candidate; -- current stable and beta/dev appcast state; -- Codemagic/GitHub/appcast alignment; -- curated stable release log; -- backend coupling decision and proposed phase; -- risks and caveats; -- exact command(s) proposed; -- post-promotion verification commands; -- roll-forward/rollback plan. - -End with an explicit confirmation question naming the exact tag and phase(s). - -## 7. Execute only approved workflows - -Desktop stable promotion: - -```bash -RELEASE_TAG='vX.Y.Z+BUILD-macos' - -gh workflow run desktop_promote_prod.yml \ - --repo BasedHardware/omi \ - -f release_tag="$RELEASE_TAG" \ - -f confirm='promote-stable' -``` - -Optional shared backend deploy, only if separately approved: - -```bash -gh workflow run gcp_backend.yml \ - --repo BasedHardware/omi \ - -f environment=prod \ - -f branch=main -``` - -## 7. Monitor and verify deterministic outcomes - -Monitor desktop promotion: - -```bash -gh run list --repo BasedHardware/omi --workflow desktop_promote_prod.yml --limit 5 -gh run watch --repo BasedHardware/omi --exit-status -``` - -After success: - -```bash -curl -fsS https://desktop-backend-hhibjajaja-uc.a.run.app/health | python3 -m json.tool -curl -fsS https://api.omi.me/v2/desktop/appcast.xml -gh release view "$RELEASE_TAG" \ - --repo BasedHardware/omi \ - --json tagName,body,isDraft,isPrerelease,url -``` - -Verify: - -- stable appcast item is the promoted build; -- GitHub Release body/channel is stable; -- release is not draft; -- `desktop-backend` health identity matches the promoted tag/SHA when supported; -- any separately approved backend deploy reports expected revisions and traffic. - -Then run the external readiness/regression process separately and report results. +Qualification evidence is layered: source-built named-bundle T2/fault coverage is not signed-artifact T2. The exact signed artifact evidence is signing/notarization and signed-smoke proof. A real promotion record must additionally cover the signed `.89` upgrade, `.70` side-by-side install, core journeys, and soak using the same immutable manifest that Stable advances. diff --git a/desktop/macos/scripts/qualify-desktop-beta.sh b/desktop/macos/scripts/qualify-desktop-beta.sh index 05f91db0e89..ca233e274ed 100755 --- a/desktop/macos/scripts/qualify-desktop-beta.sh +++ b/desktop/macos/scripts/qualify-desktop-beta.sh @@ -16,6 +16,7 @@ PROMOTE=1 AUTOMATIC=0 SIGNED_SMOKE_RESULT="" CANDIDATE_GATE_RESULT="" +GITHUB_ACTIONS_ARTIFACT=0 RELEASE_TAG="" usage() { @@ -23,7 +24,7 @@ usage() { Qualify a macOS desktop candidate (rebuild tag + T2 core E2E + promote beta). Usage: - qualify-desktop-beta.sh [--keep-stack] [--no-promote] [--automatic] \ + qualify-desktop-beta.sh [--keep-stack] [--no-promote] [--automatic] [--github-actions-artifact] \ [--signed-smoke-result PATH --candidate-gate-result PATH] Options: @@ -32,6 +33,7 @@ Options: --automatic Run richer automatic gates and require this to remain the newest candidate --signed-smoke-result PATH Codemagic signed-artifact smoke evidence (required with --automatic) --candidate-gate-result PATH Digest-bound candidate gate evidence (required with --automatic) + --github-actions-artifact Leave trusted evidence publication to the workflow artifact USAGE } @@ -59,6 +61,10 @@ while [[ $# -gt 0 ]]; do CANDIDATE_GATE_RESULT="$2" shift 2 ;; + --github-actions-artifact) + GITHUB_ACTIONS_ARTIFACT=1 + shift + ;; --help|-h) usage exit 0 @@ -321,6 +327,12 @@ if manifest.get("passed") is not True or manifest.get("tier") != "fault": PY fi +if [[ "$GITHUB_ACTIONS_ARTIFACT" -eq 1 ]]; then + QUALIFICATION_SUCCESS=1 + echo "Qualified $RELEASE_TAG for beta; trusted workflow will publish immutable Actions evidence." + exit 0 +fi + STAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ) EVIDENCE_FILE="/tmp/qualification-evidence-${VERSION}-$$.json" cp "$EVIDENCE/manifest.json" "$EVIDENCE_FILE" diff --git a/docs/doc/developer/desktop-updates.mdx b/docs/doc/developer/desktop-updates.mdx index c349e53c00c..1c1eb2b8ce8 100644 --- a/docs/doc/developer/desktop-updates.mdx +++ b/docs/doc/developer/desktop-updates.mdx @@ -1,183 +1,42 @@ --- title: Desktop update architecture -description: Explicit release channels, qualification, fallback, and recovery for macOS desktop updates. +description: Qualified desktop artifacts, explicit pointers, and recovery semantics. --- -Desktop updates use immutable release manifests and explicit `stable` and `beta` pointers. A release build is not a channel. Promotion changes a pointer; it does not rebuild or mutate the artifact. +Desktop Beta is a distribution channel, not a backend environment. The Omi and Omi Beta apps both use the established production backend; Omi Beta remains separately installable with its own bundle identity, storage/keychain/TCC scope, update feed, and signed artifacts. -The live pointer manifest is the current app-only control-plane record. `DesktopReleaseManifest` v1 is the next release-unit contract that binds the signed app to an optional Rust `desktop-backend`; it is defined independently first so shadow manifests can be compared without changing a live channel. +## Release model -## Resolution order +One Codemagic build signs and notarizes Stable and side-by-side Beta ZIP/DMG artifacts from one source revision. It first creates an immutable, non-live GitHub candidate. The trusted macOS qualifier downloads and hashes all four artifacts after the candidate gate, then uploads that evidence as an immutable GitHub Actions run artifact. Beta promotion verifies the successful trusted `main` qualification run ID, binds the exact bytes, and registers one immutable manifest. The qualified manifest stores exact Stable/Beta URLs, hashes, Sparkle signatures, release ID, and source SHA. -For each channel, the Python appcast and manual download endpoints resolve in this order: +The machine-readable evidence explicitly separates two modalities. `source_qualification` records hermetic T2 and fault coverage for source-built named bundles. `signed_artifact_verification` records the exact signed ZIP/DMG digest, Sparkle-signing, notarization, and signed-smoke evidence. The signed production bytes are never described as having run automation T2. Before a human promotes beyond Beta, the operational release evidence still requires the signed `.89` upgrade, `.70` side-by-side install, real core journeys, and soak against the same immutable manifest. Stable never rebuilds or re-reads mutable release state after manifest registration. -1. cached explicit Firestore pointer and manifest; -2. live Firestore pointer and manifest; -3. validated 30-day last-known-good pointer; -4. the same channel from legacy GitHub release metadata; -5. fail closed. +`macos-beta` and `macos-stable` are the only mutable release pointers. A pointer always resolves to an immutable qualified manifest. Registration is exact-idempotent; changing metadata under an existing release ID is rejected. -Stable never falls through to beta. `DESKTOP_UPDATE_POINTERS_MODE=legacy` is the incident kill switch. `DESKTOP_UPDATE_RECONCILE_SAMPLE_RATE` controls sampled comparisons between pointer and legacy results. - -Prometheus exposes the selected source, fallback reason, pointer/LKG age, feed validity, and sampled pointer mismatches. Cache clearing removes only live caches; it preserves last-known-good recovery data. - -## Release lifecycle - -The daily/manual candidate workflow batches changes on `main` and tags one candidate. Codemagic builds, signs, notarizes, smoke-tests, and uploads the candidate with `isLive: false` and `channel: candidate`, including the signed-smoke evidence asset. It records an owner-visible `qualificationDispatch*` lifecycle in that candidate’s metadata, then dispatches `desktop_qualify_beta.yml` with a deterministic tag-bound key. Codemagic retries an unconfirmed handoff three times with bounded backoff; an exhausted handoff records `dispatch_failed` and leaves the candidate non-live. The trusted workflow claims that key before using the runner, so an ambiguous timeout or duplicate dispatch can produce at most one qualification execution. Dispatch state is operational diagnostics only: it neither changes the immutable evidence assets nor authorizes promotion. - -That workflow runs only on the administrator-managed `self-hosted`, `macos`, `omi-desktop-qualification` runner, where Docker Desktop supports the hermetic stack. It verifies the published ZIP/DMG digests against signed-smoke evidence, runs static release checks, hermetic T2, and the fault-injection suite, and requests beta promotion only for the newest passing candidate. A terminal qualification failure remains non-live until an operator supplies an explicit new retry nonce; repeated normal dispatches retain their original deduplication key. - -| State | How to identify it | Customer visibility | -| --- | --- | --- | -| Build candidate | `channel: candidate`, `isLive: false` | None | -| Qualified beta | `channel: beta`, `isLive: true`, canonical `qualifiedBeta*` evidence (or legacy qualification aliases) | Beta only | -| Stable candidate | Qualified beta plus complete `stableCandidate*` nomination metadata | Still beta only | -| Stable | `channel: stable`, written only by `desktop_promote_prod.yml` after `confirm=promote-stable` | Stable | - -Manual recovery or evidence-only qualification remains available: - -```bash -desktop/macos/scripts/qualify-desktop-beta.sh vX.Y.Z+BUILD-macos -``` - -The automatic path calls the same script with `--automatic`, adding signed-artifact digest validation, static checks, and the hermetic fault suite to T2. The combined evidence asset records every gate. Before writing qualification metadata it rechecks that no newer tag exists. Promotion repeats the newest-tag check for automatic requests, validates the tag SHA and evidence, registers an immutable manifest, advances the beta pointer, changes GitHub metadata to live beta, and maintains the old-client Firestore bridge. Existing releases with legacy `blessed*` keys remain valid through dual-read compatibility. - -Set `DESKTOP_AUTO_BETA_ENABLED=false` in either the Codemagic desktop environment or GitHub's `prod` environment to pause automatic beta. Failures leave the candidate non-live. Stable nomination and `desktop_promote_prod.yml` remain manual-only. - -### Emergency macOS beta rollback - -`POST /v2/desktop/channels/rollback` is the only non-monotonic pointer operation. It is authenticated with `ADMIN_KEY` and accepts literal `platform: macos` and `channel: beta` values only. The request must include the target `release_id` plus the exact `expected_current_release_id` and `expected_generation`; all three are compared inside the Firestore transaction before either write occurs. The target must already be a registered macOS manifest with passed T2 qualification and an earlier build than the current beta pointer. The transaction creates an append-only immutable audit record and then advances only `desktop_update_channels/macos-beta`; it cannot write Stable, production, another channel, or another platform. The response returns that immutable audit payload and the new beta pointer. Normal `/v2/desktop/channels/promote` remains roll-forward only. - -### Emergency macOS beta forward promotion - -`desktop_emergency_promote_beta.yml` is the break-glass lane for shipping an -already-published signed macOS candidate to beta when the qualification gate -itself is broken. It is beta-only: the backend rejects any other platform or -channel. - -It is deliberately small. The only guards are the ones a machine can check: -the tag must resolve to its signed commit, the ZIP/DMG digests are recomputed -from the published assets into an immutable manifest, and the pointer moves -under a compare-and-swap on both the current release id and generation. There -is no approval ceremony, expiry window, or incident-comment protocol — an -automated actor fills those in as easily as a human, so they filter nothing. - -Mechanically it is the normal promotion path with one precondition relaxed: -`POST /v2/desktop/channels/promote` with `emergency: true` skips the T2 -qualification gate and requires the explicit CAS fields. The resulting pointer -carries `emergency: true`, which is the audit record. Normal promotion is -unchanged and still fail-closed. - -The workflow reads the live pointer itself rather than asking an operator to -paste values mid-incident; the generation CAS still rejects any concurrent -mutation between that read and the write. - -Every run opens a `qualification-gate-failure` issue recording the tag, SHA, -operator, reason, and run URL. **These issues are the point.** A bypass that -fires more than rarely means the qualification gate is the defect and this -hatch is masking it — fix the gate, then close the issue. - -After beta soak, an operator may run `desktop_nominate_stable_candidate.yml`. It verifies the tag is the current beta pointer, verifies the pointer's immutable manifest source SHA matches the tag commit, and requires explicit soak, telemetry, release-note, rationale, operator, and qualification-evidence fields. Nomination writes `stableCandidate*` metadata only: it never moves beta or stable, and it never deploys production. `desktop_promote_prod.yml` is the sole stable decision point; it requires that nomination plus `confirm=promote-stable`, with an audited typed break-glass path for emergencies. - -## Advisory release doctor - -Run **Desktop Release Doctor** with a macOS release tag to produce the first versioned release evidence artifact. It is read-only: it cannot change a pointer, release body, Cloud Run revision, traffic split, cache, or tracking tag. - -The workflow captures only release-control fields, never release prose, prompts, audio, transcripts, user identifiers, or credentials. Its evidence JSON reports `PASS`, `WARN`, or `FAIL` for the GitHub release, canonical manifest, beta/stable pointers, legacy Firestore bridge, Python and Rust appcasts, static route, backend health identity, tracking tag, Codemagic state, and operational metrics. A missing surface or metric is written as `unavailable` and changes the report to `WARN`; it is never interpreted as success. - -The doctor is advisory while the release controls migrate. Use its uploaded `desktop-release-evidence-v1.json` and text report during beta and before stable promotion. Fix `FAIL` drift before proceeding; use `WARN` entries to record missing observability or a deliberate exception. The next phase will consume a successful doctor report for pre-mutation identity gates after the V1 manifest is live in beta and stable. - -## Immutable app/backend release-unit contract - -The authoritative executable contract is the stdlib-only validator and canonicalizer at `.github/scripts/desktop_release_manifest.py`, so artifact production, beta qualification, and stable promotion can share one fail-closed implementation on clean runners. `.github/schemas/desktop-release-manifest-v1.schema.json` is its structural projection for tooling and documentation; schema-only validation is insufficient for cross-field tag, compatibility, and artifact-provenance rules. - -V1 has two exact modes: - -| Mode | Backend identity | Promotion meaning | -| --- | --- | --- | -| `app_only` | All `desktop_backend_*` fields are forbidden | App pointers may advance without building or deploying Rust. | -| `backend_required` | Source SHA, OCI index digest, and deployed platform-child digest are required | The app and Rust image come from the same source SHA; beta and stable must select those exact OCI objects. | - -Every manifest binds the release tag, version, build, app source SHA, ZIP/DMG URLs and SHA-256 digests, Sparkle signature, qualification evidence asset and digest, passed T2 tier, environment-contract version, and creation time. The compatibility contract repeats the exact app, backend-mode, backend identity when present, and environment identity. V1 deliberately permits no ranges or inferred compatibility: any mismatch fails validation before a later workflow is allowed to mutate traffic or a channel. - -Manifest integrity uses a detached digest in `sha256:<64 lowercase hex>` form plus an `hmac-sha256:<64 lowercase hex>` signature made with a separately controlled release-manifest signing secret of at least 32 bytes. Canonical bytes are UTF-8 JSON with keys sorted recursively, separators `,` and `:` with no insignificant whitespace, and non-ASCII characters preserved; the signature covers a V1 domain separator followed by those bytes. The detached form avoids excluding or special-casing a self-referential field inside the manifest. ZIP, DMG, qualification evidence, OCI index, and platform-child comparisons use the same exact digest boundary. The JSON and public digest may travel together, but the HMAC key remains an independent trust anchor supplied from the protected release environment. Every consumer runs executable validation, recomputes the digest, and verifies the signature before trusting identity or compatibility; rewriting both JSON and its colocated digest still fails signature verification. - -### Phased migration - -The contract lands before it controls a live release. This order prevents an untested manifest or registry seam from changing beta or stable behavior. - -1. Define and test V1 with no live promotion mutation. -2. Produce detached-digest V1 manifests in shadow mode and compare them with GitHub, Firestore, release artifacts, and Cloud Run for several tags. -3. Build and publish a required Rust image once during candidate preparation, then record its OCI index and `linux/arm64` child digests. -4. Make beta deploy or select the manifest-pinned digest and verify its labels and `/health` identity. -5. Require at least one complete beta cycle whose app and backend identities match V1. -6. Make stable consume the same manifest and digest, remove source checkout/build steps, and delete transitional identities only after exact comparison evidence passes. - -The following sources of truth remain transitional until their named deletion gate. There must not be an indefinite path that accepts either a tag rebuild or a V1 digest. - -| Transitional source | Current purpose | Deletion or demotion gate | +| Stage | Mutation | Audience | | --- | --- | --- | -| GitHub Release `KEY_VALUE` metadata | Candidate/beta/stable visibility and qualification/nomination metadata | Keep only presentation and nomination metadata after both channel workflows read V1 qualification and artifact identity. | -| `desktop_release_manifests` current record | App artifact manifest used by explicit pointers | Replace atomically after shadow records match and the shared Python API accepts V1; do not dual-write indefinitely. | -| `desktop_update_channels/macos-beta` and `macos-stable` | Explicit channel pointers | Retain as pointers, but point only at the immutable V1 `release_id` after both channels migrate. | -| Legacy Rust `desktop_releases` collection | Appcast bridge for older Rust clients | Delete after supported clients resolve the Python pointer feed and fallback telemetry shows no legacy reads. | -| Stable workflow tag checkout and Docker build | Rebuilds Rust during production promotion | Delete immediately when stable is gated on one successfully exercised V1 beta cycle. | -| Cloud Run `/health` tag/SHA/channel fields | Runtime release identity | Extend to V1 release ID, source SHA, manifest digest, and image digest; retire ambiguous fields after release doctor compares them for the migration window. | -| `desktop-backend-prod-deployed` tag | Tracks the source believed to be in production | Keep only as a projection written from the promoted V1 manifest, then remove when no operator or automation reads it as authority. | -| Generic `commit-sha` image/service label | Historical deployment metadata | Stop treating it as release identity once `release-id`, `release-source-sha`, `release-manifest-digest`, and `image-digest` labels are verified in beta and stable. | - -Python `backend*` deployment identity is independent. V1 neither reads nor mutates Python backend release state; only the existing desktop manifest/pointer API is involved in the later control-plane migration. - -## Named developer bundles - -Use `omi-dev` to build and update a local named bundle without creating a release: - -```bash -desktop/macos/scripts/omi-dev update --name omi-alice --ref origin/main -desktop/macos/scripts/omi-dev update --name omi-beta-local --ref v0.12.84+12084-macos -desktop/macos/scripts/omi-dev status --name omi-alice -``` - -Every `omi-` gets its own detached worktree, state directory, bundle ID, ports, and -`/Applications/omi-.app` target. The command rejects standard application names, so it -never replaces or launches Omi, Omi Beta, Stable, or their user data. Updates are explicit: -named developer bundles do not use the shared production/beta Sparkle archive. +| Candidate | Immutable GitHub release and signed evidence | None | +| Qualified Beta | `macos-beta` points at the exact qualified manifest | Omi Beta | +| Stable | `macos-stable` points at that same release ID/artifact | Omi | -`desktop/macos/scripts/omi-main` remains available as a compatibility wrapper for the -`omi-main` bundle; it forwards to `omi-dev --name omi-main` and honors the legacy -`OMI_MAIN_REPOSITORY`, `OMI_MAIN_WORKTREE`, and `OMI_MAIN_STATE_DIR` overrides. +The normal candidate-to-Beta path uses non-cancelling per-tag GitHub concurrency. Qualification evidence and pointer idempotency make bounded dispatch retries safe; there is no separate claim/running/complete release-body state machine and no unqualified emergency route. -## Update transaction truth +## Stable promotion and repoint -`willInstallUpdate` records an attempt ID plus source/target build and emits `Update Install Started`. `Update Installed` is emitted only after the relaunched app verifies that its current build equals the recorded target. A mismatch emits `Update Install Verification Failed` with the same attempt ID. +`desktop_promote_prod.yml` is protected, manual-only, and independent of backend deployment. `promote` verifies its trusted qualification run and that the requested release ID is the current qualified Beta pointer. It then reads the exact immutable manifest registered by Beta from the control plane and verifies its canonical identity; it does not reconstruct a manifest from mutable release metadata. `repoint` needs only that retained passed-T2 manifest under exact Stable CAS—no Actions artifact or GitHub release body/assets. Both update the existing legacy appcast/static bridges and verify the final pointer, hashes, and exactly one default/non-Beta Stable feed item. -Static GCS/CDN appcast delivery remains a follow-up. The existing beta download redirect may still be maintained, but it is not the source of channel truth. +`operation=promote` is roll-forward. `operation=repoint` can choose any retained qualified manifest, in either direction, but requires the exact current release ID and generation. Both use the same transactional pointer authority and qualification gate. A stale compare-and-swap is rejected; a lost response is acknowledged only when the requested release is already selected at exactly the expected next generation. -## Stable repair installer +Repointing stops future rollout but cannot downgrade clients that Sparkle has already upgraded to a higher build. Ship a higher-version hotfix for those users. -Stable promotion publishes an immutable repair DMG and its SHA-256 manifest to -`gs://omi_macos_updates/stable//`. It first uses a create-only GCS -generation precondition; a retry is accepted only when the existing bytes are -identical. The artifact and versioned manifest are externally probed before -either legacy or pointer-based stable visibility can move. After the Firestore -stable pointer advances, the workflow updates the no-cache `stable/index.html` -landing page and `stable/latest.json` manifest to reference that exact -immutable DMG. This gives support a recovery link that does not depend on -runtime GitHub enumeration or the appcast endpoint. +## Resolution and compatibility -The public stable repair page is -`https://storage.googleapis.com/omi_macos_updates/stable/index.html`. Confirm -the bucket and public-read/CDN configuration before a stable promotion; the -workflow probes the versioned DMG and manifest before it advances the pointer. +For each channel, update resolution is: live pointer cache, Firestore pointer/manifest, validated last-known-good pointer, then the same channel's legacy GitHub metadata. Stable never falls through to Beta. The legacy resolver remains while older clients require it; pointer failures record the shared fallback telemetry boundary. -## First rollout after merge +The desktop release workflows retain the existing Firestore legacy bridge and stable static repair route. A desktop pointer promotion never invokes or controls a backend deployment. -Merging this control-plane change does not make a beta live. For the first rollout: +## Backend deployment boundary for the update API -1. Deploy `main` to the production shared Python backend with `gcp_backend.yml` so manifest registration and pointer promotion endpoints exist. -2. Wait for the weekday candidate workflow, or manually dispatch `desktop_auto_release.yml` when a candidate is wanted. -3. Wait for Codemagic to publish the signed, notarized, non-live candidate and smoke evidence. -4. Let the dedicated trusted macOS runner run the automatic digest, static, T2, and fault-suite gates. Use `desktop/macos/scripts/qualify-desktop-beta.sh ` only for manual recovery. -5. Verify the beta appcast and the resolution/fallback/pointer-age metrics. Nominate a stable candidate only after reviewing soak, telemetry, and the stable release-note rollup; leave stable unchanged until the separately planned stable rollout. +The surviving direct backend deployment boundary for the desktop update API is Cloud Run only. `gcp_backend.yml` accepts `environment=prod, deploy_targets=cloud-run-only`: it deploys tagged no-traffic Cloud Run candidate revisions, proves the exact tagged backend with a VPC-connected ephemeral Cloud Run Job and both Cloud Run IAM plus Firebase authentication, then snapshots, promotes, verifies, and restores Cloud Run traffic if final verification fails. The temporary Job, tag, invoker binding, and probe service account are cleaned up on both success and failure. -After the dual path has proven reliable, complete the removal and static CDN follow-ups tracked in issue `#9350`. +`environment=prod, deploy_targets=all` is rejected before checkout, cloud authentication, image publication, or mutation. Transactional GKE/config rollback parity does not exist, so this workflow does not claim atomic full-stack deployment. The source-admitted direct GKE workflows (`gcp_backend_listen_helm.yml`, `gcp_backend_pusher.yml`, `gcp_llm_gateway.yml`, and `gcp_backend_agent_proxy.yml`) remain the operational production paths for their respective surfaces. diff --git a/docs/runbooks/release-process.md b/docs/runbooks/release-process.md deleted file mode 100644 index e004e573736..00000000000 --- a/docs/runbooks/release-process.md +++ /dev/null @@ -1,254 +0,0 @@ -# Release & Deployment Process - -Internal maintainer doc — lean release engineering: build once, promote -immutable digests through beta to prod, roll back by deploying an older -release. Tracking issue: https://github.com/BasedHardware/omi/issues/10049 - -## Philosophy: lean but high value - -Omi is built and operated by a very small team. Release infrastructure is -therefore optimized for **low devops, hard-to-break beta/prod, easy deploys, -and few footguns** — in that priority order. A small system we finish, trust, -and exercise daily beats an elaborate one that is half-built. When choosing -between a bespoke mechanism and a native platform control (GitHub environment -protection, Cloud Run revisions and traffic splitting, Helm revisions, GCS -object versioning), choose the platform control. - -## The seven rules - -1. **Build once, promote digests.** Every merge to `main` builds service - images once. Beta and prod deploy the exact same `repository@sha256:...` - digests — prod never rebuilds, re-resolves a branch, or deploys a mutable - tag. Tags are human-readable labels, never deployment authority. No `main` - SHA deploys directly to prod; only a beta-soaked record is prod-eligible. - (If builds are ever made incremental, unchanged-component digests may be - composed from the prior record only via a content hash of the component's - full input tree — never path filters, which silently ship stale code.) -2. **Promote artifacts, not branches.** There are no `beta`/`prod` branches - and no sync bots. What is deployed where is recorded in release records and - the GitHub environment audit log, not in branch topology. -3. **Deploy and rollback are the same command.** Rolling back means deploying - an older release record through the identical code path. Because rollback - shares every line with deploy, it is exercised on every release and cannot - rot into an incident-day surprise. Never build a separate rollback system. -4. **Pin everything a release depends on.** Image digests, Secret Manager - secret *versions* (never `latest`), and rendered config/values. Nothing - about a running release may drift without a new release. -5. **Beta is a ring, not a lab.** Beta-channel users are real users on the - shared production data stores, served by beta instances of the stateless - services running one release ahead of prod. Real beta traffic is the - acceptance signal; synthetic checks stay cheap (auth + one streaming - round-trip smoke test against a no-traffic revision). -6. **Never remove a deploy path before its replacement has shipped for - real.** Old paths are deleted one at a time, only after the new path has - handled actual production releases. -7. **Enforcement grows by ratchet.** Add a CI check for a release rule after - the rule is actually violated, not speculatively (same philosophy as - testing — see `AGENTS.md`). - -## Release records - -On every merge to `main`, after images build and push, CI writes an immutable -JSON record to a versioned, retention-locked GCS bucket: - -```jsonc -// gs://omi-releases/records/.json -{ - "release_id": "2026-07-20.1", // date.seq — human-readable, sortable - "git_sha": "…", - "eligibility_run_id": "…", // CI run that built and tested this release - "images": { "backend": "gcr.io/...@sha256:…", "pusher": "…" }, - "rendered_config": { "beta/backend-listen": "gs://…/config//beta/backend-listen.yaml#sha256:…" }, - "secret_versions": { "OPENAI_API_KEY": "projects/…/versions/42" }, - "topology": { "beta": { "cloud_run_services": { "backend": "backend-beta" } } }, - "created_at": "…" -} -``` - -**A record must be deployable, not merely verifiable** — store rendered -values/config as immutable objects referenced with hashes, never hashes alone -(a hash can prove identity but cannot restore anything). - -The record is the only input a prod-capable deploy accepts — no free-form SHA, -branch, tag, image, or Helm revision. Grow the schema only when a deploy -actually needs a new field. - -Ring state is three GCS object shapes — still no service, no database: - -- `records/.json` — immutable deployable inputs (above). -- `rings//active.json` — current serving record, previous verified - record, and hold/quarantine flag; written with `ifGenerationMatch` - compare-and-swap. (Deploy serialization itself is a GitHub Actions - `concurrency: deploy-` group — the CAS is a backstop, not a lock - service.) -- `receipts///.json` — append-only record of what - actually happened: Cloud Run revisions/traffic, Helm revisions, - verification and restore results. Receipts are a write-only audit trail; - never build a reconciler on top of them. - -## Rings - -```text -main merge ──► build once ──► release record - ├─► dev — auto - ├─► beta — auto: newest record → beta lane - └─► prod — manual: deploy prod - (GitHub environment gate, same digests) -``` - -- **dev** — development auto-deploys, unchanged. -- **beta** — beta instances of the stateless services (backend, - backend-listen, pusher, llm-gateway, agent-proxy) at beta endpoints - (`api-beta.omi.me`), serving real beta-channel desktop/mobile clients. - Shares production data stores. A service joins the ring system only once - it has a declared recovery class and a tested ring strategy; until then it - keeps its dedicated deploy path. -- **prod** — deploys a beta-soaked record behind the GitHub `prod` environment - gate. The gate is only a barrier when paired with: a prod-only OIDC identity - bound to repo + environment, explicit workflow/ref guards, and code - ownership + required checks on `.github/workflows/` — otherwise any future - merged workflow can request the environment and reduce prod safety to one - approval click. Promotion criteria: the record served beta for at least the - soak window (default: overnight) with no new Sentry crash groups, - fallback/error rates in bounds, **and** a minimum exposure floor (active - sessions, streamed minutes) — a quiet night must not qualify a release. - -Per-ring configuration lives in source-controlled values and -`backend/deploy/release_rings.yaml`; the record writer materializes and hashes -the beta/prod objects before they can be deployed. It captures each ring's -public ConfigMap at record-build time, turns every Cloud Run `env_var` into a literal, -pins Cloud Run and ExternalSecret numeric versions, and writes a materialized -`beta` runtime lane into the record. The beta renderer changes only stateless -identities and endpoints. It intentionally preserves existing GPU endpoints, -which is the explicit #10057 boundary. - -## Implemented workflow contract - -- `release-record.yml` runs only after a successful `Release Eligibility` run - for the current `main` tip. It builds backend, pusher, llm-gateway, and - agent-proxy once, records their OCI digests (backend-listen uses the backend - digest), resolves every Secret Manager reference to a numeric version, and - writes create-only config and record objects. -- `deploy-release-ring.yml` accepts only `ring` and `release_id`. It uses the - selected GitHub environment and that environment's OIDC identity; it has no - SHA, tag, branch, Helm-revision, or image input. `prod` additionally requires - the literal `confirm=deploy-prod` after the GitHub environment approval. The - workflow controller stays pinned to the dispatch commit; only immutable chart - source inputs are checked out from the record's full Git SHA. -- Before mutation, a ring deploy server-dry-runs the recorded ConfigMap and all - rendered Helm resources. It then captures Cloud Run traffic, the public - ConfigMap, deployed Helm revisions plus manifest digests, and the active - pointer generation. It writes a `started` receipt, creates no-traffic Cloud Run revisions using the - recorded public config and numeric secret versions, performs an authenticated - health smoke, and for beta proves an authenticated known-audio transcription - through the tagged candidate before applying the recorded - ConfigMap/ExternalSecret/Helm values by digest and switching traffic. Prod - depends on that verified beta evidence plus the soak, because prod has no - public candidate URL. A deterministic failure restores the snapshot once, - removes the failed candidate's tagged route, deletes resources that did not - exist before a bootstrap attempt, refreshes restored ExternalSecrets, and - verifies observed Cloud Run traffic, ConfigMap content, and Helm manifests. - It also restores the pre-mutation active pointer if a later evidence write - failed after pointer advancement. Any failure to converge writes - `partial_mutation`, holds the record, and invokes the configured pager webhook. -- `prod` additionally requires a verified beta receipt older than the configured - soak window (12 hours by default) and refuses a beta-held record. The prod - environment approval is where the operator records the initial Sentry, - fallback/error-rate, and exposure-floor review; automate those predicates - only after they have a proven query contract. -- iOS TestFlight and Android internal builds carry a non-user-controlled beta - build identity and always use `api-beta.omi.me`. macOS beta uses that Python - API too. Its Rust desktop-backend keeps the signed production endpoint and - the separate `desktop_promote_prod.yml` roll-forward-only recovery class; - it is deliberately not claimed as a v1 release-ring workload. - -Before enabling either workflow, provision the `beta` and `prod` GitHub -environments, a retention-locked/versioned `RELEASE_RECORDS_BUCKET`, and two -least-privilege Workload Identity bindings. The prod binding must restrict -claims to this repository **and** the `prod` environment; the record writer -may read the production public ConfigMap and write records/config only, and the -ring deployer may read records plus mutate only its own ring. It also needs -read access to the existing, narrowly scoped `RELEASE_SECRET` used only to mint -the beta candidate's short-lived probe identity. Before the first -beta deploy, provision the beta Cloud Run service identities, namespace, -public ConfigMap with beta Cloud Tasks targets, ExternalSecret Workload Identity, -static ingress addresses/certificate, and DNS -for `api-beta.omi.me`, `pusher-beta.omi.me`, and `agent-beta.omi.me`; the -workflow fails closed if any prerequisite is absent. Protect `main` so the -recursive `.github/workflows/**` CODEOWNERS rule and the `release-ring-guards` -check are required. These are control-plane settings, not values to place in -this repository. - -## Deploying and rolling back - -`deploy `: - -1. Resolve the record from the bucket; refuse anything else. -2. Dry-run every recorded GKE render, then snapshot the ring's active serving - state and active-pointer generation before any mutation. -3. Cloud Run: create no-traffic revisions from recorded digests → smoke check - the revision URL → shift traffic. -4. GKE: apply charts with recorded digests and values and wait for rollout. -5. On deterministic failure before or immediately after the traffic switch: - restore the pre-mutation snapshot automatically (Cloud Run: shift traffic - back, remove the candidate tag, and delete a first-time service; GKE: - restore the ConfigMap and backend-secrets, roll back existing Helm releases, - and uninstall first-time releases). Every restore target is read back and - compared with the snapshot before recovery is accepted. If restoration also fails, write - `partial_mutation` with the exact component diff, leave state visible, and - page — no recursive recovery attempts. -6. A failed record is marked held in `rings//active.json` so auto-deploy - does not retry the same bad candidate. - -Later SLO degradation pages a human and offers the one-command rollback — -never automatic (that is the flapping trap). - -**Rollback = the same command with an older `release_id`.** The only separate -emergency path is traffic-only: shifting Cloud Run traffic to a previous -healthy revision. It cannot build images or change config. - -For today's manual prod deploy and traffic-repair procedures, see -`docs/doc/developer/backend/prod_hotfix_runbook.mdx` and -`docs/runbooks/backend-deploy-rollout-safety.md`. - -## Recovery class per surface - -Every promotable surface declares exactly one recovery class. A surface with -no declared class is not promotable, and **a class may be declared only for a -tested mechanism** — an untested restore path is a rollback that works on -paper. - -| Surface | Class | Mechanism | -|---|---|---| -| Cloud Run services | `exact_restore` | Deploy older record (digests + traffic map). | -| GKE/Helm services | `exact_restore` | Deploy older record (digests + values). | -| Runtime config / secrets | `exact_restore` | Older record's config + pinned secret versions. | -| Web/CDN | `pointer_restore` | Versioned asset pointer + cache invalidation. | -| macOS channels | `roll_forward_only` | Update feeds choose the newest release and promotion is roll-forward only (`desktop_promote_prod.yml`): halt rollout, restore server-side compatibility, ship a higher-version hotfix. Upgrades to `pointer_restore` only when a tested channel-pointer rollback path lands. | -| Mobile stores | `roll_forward_only` | Halt rollout; ship compatible higher-version hotfix. | -| Schema/data | `roll_forward_only` | Expand/contract + compensating migration; no blind data rollback. | -| Firmware | `roll_forward_only` | Stop rollout; signed recovery image. | - -## Compatibility rules - -Because beta and prod rings share live data stores, these are load-bearing: - -- Schema/data changes follow **expand → dual read/write → backfill → - contract**, never destructive in the same release. -- Every concurrently serving record — from the current prod record through - the newest beta record (prod may lag beta by several records, not one) — - must operate correctly against live data. Hold promotion when ring skew - exceeds the supported window. -- Artifact and record retention lasts at least the supported rollback window - (default 30 days). Destructive index/data cleanup waits until the oldest - retained record has expired. - -## Adoption - -The target state above is adopted incrementally; sequencing, current scope, -and per-surface status live in the tracking issue -([#10049](https://github.com/BasedHardware/omi/issues/10049)), not in this -doc. Until a surface has joined the ring system, its currently documented -workflow (e.g. the traffic-only hotfix runbook) remains authoritative. The old -arbitrary-ref prod deploy workflows refuse prod artifact deploys; their -traffic-only repair path remains a deliberately narrow emergency tool.