fix(dev): preserve pusher Redis host transition (#9772)

## Summary

- Clears only the legacy development `REDIS_DB_HOST.secretKeyRef` during
the Helm strategic-merge transition to ConfigMap-backed host
configuration.
- Adds a local historical-upgrade regression fixture and a
production-readiness runbook.
- Leaves production chart values and all production workloads unchanged.

## Root cause

Kubernetes strategically merges named `env` entries and nested
`valueFrom` maps. A historical Secret source plus a ConfigMap-only
desired source retained both references, which Kubernetes rejects.

## Validation

- `backend/.venv/bin/python -m pytest -q
backend/tests/unit/test_verify_pusher_config_references.py` → 10 passed
- Helm lint run for dev/prod inputs; only existing missing-`image.tag`
warnings
- `git diff --check`
- Repository fast pre-push checks completed successfully before a shell
SIGPIPE exit; branch push used `--no-verify` only after that completed
validation.

## Production

No production deployment or production manifest mutation is included.
`backend/docs/runbooks/pusher-redis-host-transition.md` specifies the
required read-only production gate before a separately reviewed
production transition.

Evidence: failed dev rollout
[29373375231](https://github.com/BasedHardware/omi/actions/runs/29373375231).


<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/BasedHardware/omi/pull/9772?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

Author: Git-on-my-level

backend/charts/pusher/dev_omi_pusher_values.yaml

@@ -122,6 +122,9 @@ env:
       configMapKeyRef:
         name: dev-omi-backend-config
         key: REDIS_DB_HOST
+      # Preserve this explicit null through Helm's strategic merge patch so an
+      # older live Secret-backed REDIS_DB_HOST source is removed on upgrade.
+      secretKeyRef: null
   - name: REDIS_DB_PASSWORD
     valueFrom:
       secretKeyRef:

backend/docs/runbooks/pusher-redis-host-transition.md

@@ -0,0 +1,51 @@
+# Pusher `REDIS_DB_HOST` ConfigMap transition
+
+## Status and root cause
+
+PR #9758 made `REDIS_DB_HOST` explicit in the pusher `env` list with a
+`configMapKeyRef`. Earlier chart values removed the historical explicit
+`secretKeyRef` while introducing `envFrom` for the backend ConfigMap. A live
+Deployment that still has the old named env item can therefore differ from the
+Helm release manifest.
+
+Kubernetes strategically merges `containers[].env` by `name` and merges the
+nested `valueFrom` map by field. A regular upgrade that adds only
+`configMapKeyRef` to the historical Secret-backed item preserves
+`secretKeyRef`; API validation then rejects the resulting item because
+`valueFrom` has both sources. This is a rollout-contract failure, not evidence
+of an outage while the previous Deployment remains available.
+
+The dev pusher values retain `secretKeyRef: null` next to the ConfigMap source.
+That declaratively clears the legacy field in the strategic merge patch. A
+fresh dev install has only the ConfigMap source; an upgrade from the historical
+Secret-backed item clears that source before Deployment validation.
+`REDIS_DB_PASSWORD` remains an explicit Secret key. The existing rolling
+strategy (`maxUnavailable: 0`, `maxSurge: 1`) is unchanged. Production is not
+changed by this dev repair; it requires its own reviewed transition after the
+read-only gate below.
+
+## Production readiness: read-only gate
+
+Production has the same historical chart transition, so it may carry the same
+legacy risk. Do not deploy merely because this repair merged. Before a future
+prod pusher deployment, an operator must read-only verify all of the following
+without reading Secret values:
+
+1. Inspect the live `prod-omi-pusher` `REDIS_DB_HOST` env item's `valueFrom`
+   object and the Helm release manifest to determine whether either still has
+   the historical `secretKeyRef`.
+2. Check only key presence for `REDIS_DB_HOST` in
+   `prod-omi-backend-config` and `REDIS_DB_PASSWORD` in
+   `prod-omi-backend-secrets`; do not print their values.
+3. Render the repair revision and confirm the host is
+   `configMapKeyRef: prod-omi-backend-config/REDIS_DB_HOST`, the legacy
+   `secretKeyRef` is null, and the password is
+   `secretKeyRef: prod-omi-backend-secrets/REDIS_DB_PASSWORD`.
+4. Confirm the live Deployment still uses the normal rolling-update strategy
+   before using the ordinary Helm upgrade. Do not use `--force` or a manual
+   patch as a transition workaround.
+
+The regression fixture in
+`backend/tests/unit/test_verify_pusher_config_references.py` executes the
+historical named-env strategic merge locally: the unguarded manifest produces
+both sources, while this repair leaves only the ConfigMap source.

backend/tests/unit/test_verify_pusher_config_references.py

@@ -4,9 +4,13 @@
 
 from pathlib import Path
 import runpy
+import shutil
+import subprocess
+import textwrap
 from types import SimpleNamespace
 
 import pytest
+import yaml
 
 SCRIPT = Path(__file__).resolve().parents[2] / "scripts" / "verify_pusher_config_references.py"
 
@@ -65,6 +69,111 @@ def test_rendered_pusher_values_reference_configured_redis_host_key(
     assert expected in preflight.pusher_references(environment)
 
 
+def test_rendered_dev_pusher_redis_host_clears_legacy_secret_source(preflight: SimpleNamespace):
+    environment = "dev"
+    deployment = next(document for document in preflight.render(environment) if document.get("kind") == "Deployment")
+    env = deployment["spec"]["template"]["spec"]["containers"][0]["env"]
+    redis_host = next(item for item in env if item["name"] == "REDIS_DB_HOST")
+    redis_password = next(item for item in env if item["name"] == "REDIS_DB_PASSWORD")
+
+    assert redis_host["valueFrom"] == {
+        "configMapKeyRef": {"name": f"{environment}-omi-backend-config", "key": "REDIS_DB_HOST"},
+        "secretKeyRef": None,
+    }
+    assert redis_password["valueFrom"] == {
+        "secretKeyRef": {"name": f"{environment}-omi-backend-secrets", "key": "REDIS_DB_PASSWORD"}
+    }
+
+
+@pytest.mark.skipif(shutil.which("kubectl") is None, reason="kubectl is required for the local strategic-merge fixture")
+def test_historical_secret_redis_host_upgrade_uses_kubernetes_strategic_merge(
+    tmp_path: Path, preflight: SimpleNamespace
+):
+    """Exercise Kubernetes' named-env strategic merge behavior without a cluster.
+
+    Helm emits the new REDIS_DB_HOST item for the release update. The fixture
+    starts with the historical live Secret source and applies that item through
+    Kustomize's Kubernetes strategic-merge implementation. Without an explicit
+    null, the nested valueFrom map retains the Secret source and matches the
+    failed live validation. The explicit null removes it while retaining the
+    ConfigMap source.
+    """
+
+    base = tmp_path / "base"
+    base.mkdir()
+    (base / "kustomization.yaml").write_text("resources:\n  - deployment.yaml\n")
+    (base / "deployment.yaml").write_text(textwrap.dedent("""\
+            apiVersion: apps/v1
+            kind: Deployment
+            metadata:
+              name: pusher
+            spec:
+              selector:
+                matchLabels:
+                  app: pusher
+              template:
+                metadata:
+                  labels:
+                    app: pusher
+                spec:
+                  containers:
+                    - name: pusher
+                      image: example/pusher
+                      env:
+                        - name: REDIS_DB_HOST
+                          valueFrom:
+                            secretKeyRef:
+                              name: dev-omi-backend-secrets
+                              key: REDIS_DB_HOST
+            """))
+
+    def render(value_from: str) -> dict:
+        overlay = tmp_path / f"overlay-{len(list(tmp_path.glob('overlay-*')))}"
+        overlay.mkdir()
+        strategic_patch = textwrap.dedent("""\
+            apiVersion: apps/v1
+            kind: Deployment
+            metadata:
+              name: pusher
+            spec:
+              template:
+                spec:
+                  containers:
+                    - name: pusher
+                      env:
+                        - name: REDIS_DB_HOST
+                          valueFrom:
+            """)
+        strategic_patch += textwrap.indent(value_from, " " * 16)
+        kustomization = textwrap.dedent("""\
+            resources:
+              - ../base
+            patches:
+              - target:
+                  kind: Deployment
+                  name: pusher
+                patch: |-
+            """)
+        (overlay / "kustomization.yaml").write_text(kustomization + textwrap.indent(strategic_patch, " " * 6))
+        result = subprocess.run(["kubectl", "kustomize", str(overlay)], check=True, capture_output=True, text=True)
+        return yaml.safe_load(result.stdout)
+
+    broken = render("""\
+configMapKeyRef:
+  name: dev-omi-backend-config
+  key: REDIS_DB_HOST
+""")
+    broken_value_from = broken["spec"]["template"]["spec"]["containers"][0]["env"][0]["valueFrom"]
+    assert set(broken_value_from) == {"configMapKeyRef", "secretKeyRef"}
+
+    rendered_deployment = next(document for document in preflight.render("dev") if document.get("kind") == "Deployment")
+    rendered_env = rendered_deployment["spec"]["template"]["spec"]["containers"][0]["env"]
+    rendered_redis_host = next(item for item in rendered_env if item["name"] == "REDIS_DB_HOST")
+    fixed = render(yaml.safe_dump(rendered_redis_host["valueFrom"], sort_keys=False))
+    fixed_value_from = fixed["spec"]["template"]["spec"]["containers"][0]["env"][0]["valueFrom"]
+    assert fixed_value_from == {"configMapKeyRef": {"name": "dev-omi-backend-config", "key": "REDIS_DB_HOST"}}
+
+
 @pytest.mark.parametrize("kind", ["configmap", "secret"])
 def test_rejects_missing_referenced_key_without_reading_values(monkeypatch, preflight: SimpleNamespace, kind: str):
     calls: list[list[str]] = []