chore(backend): disable hosted Deepgram serving (#9936)

## Summary
- Disable hosted Deepgram for all real-traffic STT surfaces; deployment
preferences cannot reactivate it.
- Keep the self-hosted Deepgram Helm chart, its production `dg.omi.me`
endpoint, and explicit self-hosted credential configuration.
- Make provider/surface availability code-owned: hosted Deepgram is
disabled, while self-hosted Deepgram is explicit streaming-only; default
streaming, prerecorded, and PTT traffic uses Parakeet or Modulate.
- Preserve the follow-up PTT stereo rejection and its E2E fake
adjustment.

## Verification
- 325 focused backend unit tests passed, covering provider policy,
hosted-Deepgram rejection, prerecorded routing, streaming backoff,
deployment references, runtime-env validation, and the PTT follow-up.
- `validate-backend-runtime-env.py --env dev --check-workflows
--check-rendered-cloud-run` passed.
- `validate-backend-runtime-env.py --env prod --check-workflows
--check-rendered-cloud-run` passed.
- `make preflight` passed (24 deterministic checks).
- Final Luna review approved the combined branch: zero net
self-hosted-chart diff, hosted Deepgram blocked from real traffic, and
the PTT follow-up retained.

## Local environment note
- Attempted `backend/testing/e2e/test_listen_stt.py`; all 11 cases were
blocked before app setup by an installed `google-auth` /
`google_auth_httplib2` API mismatch (`google.auth.transport.Response`
missing), not by an application assertion.

## Scope note
This verifies checked-in deployment source and rendered manifests. No
live Kubernetes or Cloud Run inspection was possible because this
environment has no cluster context or gcloud access.

Failure-Class: none

Author: Git-on-my-level

.github/scripts/deferred-work-marker-count.py

@@ -40,7 +40,6 @@
 NORMALIZED_EXCLUDED_PREFIXES = (
     ".github/workflows/",
     "app/lib/l10n/",
-    "backend/charts/deepgram-self-hosted/nova-3/charts/",
     "omi/firmware/devkit/src/lib/opus-1.2.1/",
     "omi/firmware/omi/src/lib/core/lib/opus-1.2.1/",
 )

.github/scripts/product_file_line_count_ratchet_baseline/backend-other.json

@@ -1,9 +1,9 @@
 {
   "files": {
-    "backend/scripts/validate-backend-runtime-env.py": 1790
+    "backend/scripts/validate-backend-runtime-env.py": 1831
   },
   "raise_justifications": {
-    "backend/scripts/validate-backend-runtime-env.py": "The shared forbidden-env guard keeps Cloud Run runtime-contract validation auditable in its authoritative validator."
+    "backend/scripts/validate-backend-runtime-env.py": "Serving STT retirement and shared forbidden-env guard keep provider/runtime validation in the authoritative deployment contract validator."
   },
   "threshold": 1500
 }

.github/scripts/product_file_line_count_ratchet_baseline/backend-routers.json

@@ -1,13 +1,14 @@
 {
   "files": {
     "backend/routers/apps.py": 2335,
-    "backend/routers/chat.py": 1548,
+    "backend/routers/chat.py": 1580,
     "backend/routers/developer.py": 2186,
     "backend/routers/mcp_sse.py": 1857,
     "backend/routers/sync.py": 1981,
     "backend/routers/users.py": 2046
   },
   "raise_justifications": {
+    "backend/routers/chat.py": "PTT stereo rejection keeps the serving STT provider boundary explicit in the established chat admission owner.",
     "backend/routers/users.py": "GET /v1/users/subscription serializes the new mobile plus/max plans as `unlimited` for clients whose plan enum predates them, so day-one buyers read as paid instead of Free (mirrors the existing operator remap); real limits/grandfather are computed from the true plan first."
   },
   "threshold": 1500

.github/workflows/desktop_backend_auto_dev.yml

@@ -123,7 +123,6 @@ jobs:
             REDIS_DB_HOST=REDIS_DB_HOST:latest
             REDIS_DB_PORT=REDIS_DB_PORT:latest
             PINECONE_HOST=PINECONE_HOST:latest
-            DEEPGRAM_API_KEY=DESKTOP_DEEPGRAM_API_KEY:latest
             ANTHROPIC_API_KEY=DESKTOP_ANTHROPIC_API_KEY:latest
             DESKTOP_LEGACY_ANTHROPIC_KEY=DESKTOP_LEGACY_ANTHROPIC_KEY:latest
             GOOGLE_CALENDAR_API_KEY=DESKTOP_GOOGLE_CALENDAR_API_KEY:latest

.github/workflows/desktop_promote_prod.yml

@@ -369,7 +369,6 @@ jobs:
             FIREBASE_API_KEY=DESKTOP_FIREBASE_API_KEY:latest
             REDIS_DB_HOST=DESKTOP_REDIS_DB_HOST:latest
             REDIS_DB_PORT=DESKTOP_REDIS_DB_PORT:latest
-            DEEPGRAM_API_KEY=DESKTOP_DEEPGRAM_API_KEY:latest
             ANTHROPIC_API_KEY=DESKTOP_ANTHROPIC_API_KEY:latest
             DESKTOP_LEGACY_ANTHROPIC_KEY=DESKTOP_LEGACY_ANTHROPIC_KEY:latest
             GOOGLE_CALENDAR_API_KEY=DESKTOP_GOOGLE_CALENDAR_API_KEY:latest

.github/workflows/gcp_backend.yml

@@ -386,7 +386,6 @@ jobs:
           MCP_OAUTH_CLAUDE_CLIENT_NAME: ${{ vars.MCP_OAUTH_CLAUDE_CLIENT_NAME }}
           MCP_OAUTH_CLAUDE_REDIRECT_URIS: ${{ vars.MCP_OAUTH_CLAUDE_REDIRECT_URIS }}
           OMI_LLM_GATEWAY_URL: ${{ steps.gateway-serving.outputs.gateway_url || 'http://127.0.0.1:9' }}
-          STT_PRERECORDED_MODEL: ${{ vars.STT_PRERECORDED_MODEL }}
           SYNC_LEDGER_FENCE_MODE: ${{ vars.SYNC_LEDGER_FENCE_MODE || 'legacy' }}
           SYNC_TASKS_HANDLER_URL: ${{ steps.account-deletion-target.outputs.sync_tasks_handler_url }}
           SYNC_TASKS_INVOKER_SA: ${{ steps.account-deletion-target.outputs.sync_tasks_invoker_sa }}
@@ -535,8 +534,8 @@ jobs:
           MCP_RESOURCE_URL: ${{ vars.MCP_RESOURCE_URL }}
           RAPID_API_HOST: ${{ vars.RAPID_API_HOST }}
           REDIS_DB_HOST: ${{ vars.REDIS_DB_HOST }}
-          STT_PRERECORDED_MODEL: ${{ vars.STT_PRERECORDED_MODEL }}
-          STT_SERVICE_MODELS: ${{ vars.STT_SERVICE_MODELS }}
+          STT_PRERECORDED_MODEL: parakeet,modulate-velma-2
+          STT_SERVICE_MODELS: parakeet,modulate-velma-2
           SYNC_TASKS_HANDLER_URL: ${{ steps.account-deletion-target.outputs.sync_tasks_handler_url }}
           SYNC_TASKS_INVOKER_SA: ${{ steps.account-deletion-target.outputs.sync_tasks_invoker_sa }}
           TYPESENSE_HOST: ${{ vars.TYPESENSE_HOST }}

.github/workflows/gcp_backend_auto_dev.yml

@@ -212,7 +212,6 @@ jobs:
           CLOUD_RUN_VPC_SUBNET: ${{ vars.CLOUD_RUN_VPC_SUBNET }}
           GOOGLE_CLIENT_ID: ${{ vars.GOOGLE_CLIENT_ID }}
           OMI_LLM_GATEWAY_URL: ${{ steps.gateway-serving.outputs.gateway_url }}
-          STT_PRERECORDED_MODEL: ${{ vars.STT_PRERECORDED_MODEL }}
           SYNC_LEDGER_FENCE_MODE: ${{ vars.SYNC_LEDGER_FENCE_MODE || 'legacy' }}
         run: |
           python3 backend/scripts/render_backend_runtime_env.py --env dev >> "$GITHUB_OUTPUT"
@@ -322,8 +321,8 @@ jobs:
           MCP_RESOURCE_URL: ${{ vars.MCP_RESOURCE_URL }}
           RAPID_API_HOST: ${{ vars.RAPID_API_HOST }}
           REDIS_DB_HOST: ${{ vars.REDIS_DB_HOST }}
-          STT_PRERECORDED_MODEL: ${{ vars.STT_PRERECORDED_MODEL }}
-          STT_SERVICE_MODELS: ${{ vars.STT_SERVICE_MODELS }}
+          STT_PRERECORDED_MODEL: parakeet,modulate-velma-2
+          STT_SERVICE_MODELS: parakeet,modulate-velma-2
           TYPESENSE_HOST: ${{ vars.TYPESENSE_HOST }}
           TWILIO_ACCOUNT_SID: ${{ vars.TWILIO_ACCOUNT_SID }}
           TWILIO_API_KEY_SID: ${{ vars.TWILIO_API_KEY_SID }}

.github/workflows/gcp_backend_listen_helm.yml

@@ -98,8 +98,8 @@ jobs:
           MCP_RESOURCE_URL: ${{ vars.MCP_RESOURCE_URL }}
           RAPID_API_HOST: ${{ vars.RAPID_API_HOST }}
           REDIS_DB_HOST: ${{ vars.REDIS_DB_HOST }}
-          STT_PRERECORDED_MODEL: ${{ vars.STT_PRERECORDED_MODEL }}
-          STT_SERVICE_MODELS: ${{ vars.STT_SERVICE_MODELS }}
+          STT_PRERECORDED_MODEL: parakeet,modulate-velma-2
+          STT_SERVICE_MODELS: parakeet,modulate-velma-2
           TYPESENSE_HOST: ${{ vars.TYPESENSE_HOST }}
           TWILIO_ACCOUNT_SID: ${{ vars.TWILIO_ACCOUNT_SID }}
           TWILIO_API_KEY_SID: ${{ vars.TWILIO_API_KEY_SID }}

.github/workflows/gcp_backend_pusher.yml

@@ -180,8 +180,8 @@ jobs:
           MCP_RESOURCE_URL: ${{ vars.MCP_RESOURCE_URL }}
           RAPID_API_HOST: ${{ vars.RAPID_API_HOST }}
           REDIS_DB_HOST: ${{ vars.REDIS_DB_HOST }}
-          STT_PRERECORDED_MODEL: ${{ vars.STT_PRERECORDED_MODEL }}
-          STT_SERVICE_MODELS: ${{ vars.STT_SERVICE_MODELS }}
+          STT_PRERECORDED_MODEL: parakeet,modulate-velma-2
+          STT_SERVICE_MODELS: parakeet,modulate-velma-2
           SYNC_TASKS_HANDLER_URL: ${{ steps.pusher-runtime-targets.outputs.sync_tasks_handler_url }}
           SYNC_TASKS_INVOKER_SA: ${{ steps.pusher-runtime-targets.outputs.sync_tasks_invoker_sa }}
           TYPESENSE_HOST: ${{ vars.TYPESENSE_HOST }}

backend/AGENTS.md

@@ -23,7 +23,7 @@ When intentionally changing backend Python dependencies, edit the relevant `requ
 
 By default, the lock refresh preserves already-locked package versions so unrelated transitive upgrades do not sneak into infrastructure changes. Set `PYLOCK_UPGRADE=1` only when intentionally refreshing dependency versions.
 
-Key env vars: `OPENAI_API_KEY` (LLM calls — not `OPENAI_ADMIN_KEY` which is billing-only), `DEEPGRAM_API_KEY` (STT), `GEMINI_API_KEY` and `ANTHROPIC_API_KEY` (local harness chat/realtime via Rust desktop backend), `ENCRYPTION_SECRET` (required for tests), `REDIS_DB_HOST` (cache/rate-limiting, fail-open without it), `ADMIN_KEY` (local dev auth bypass via token `ADMIN_KEY<uid>`), `SERVICE_ACCOUNT_JSON` (Firestore/GCS credentials).
+Key env vars: `OPENAI_API_KEY` (LLM calls — not `OPENAI_ADMIN_KEY` which is billing-only), `HOSTED_PARAKEET_API_URL` and `MODULATE_API_KEY` (default serving STT), `DEEPGRAM_API_KEY` with `DEEPGRAM_SELF_HOSTED_ENABLED=true` and a non-cloud `DEEPGRAM_SELF_HOSTED_URL` (explicit self-hosted Deepgram streaming only), `GEMINI_API_KEY` and `ANTHROPIC_API_KEY` (local harness chat/realtime via Rust desktop backend), `ENCRYPTION_SECRET` (required for tests), `REDIS_DB_HOST` (cache/rate-limiting, fail-open without it), `ADMIN_KEY` (local dev auth bypass via token `ADMIN_KEY<uid>`), `SERVICE_ACCOUNT_JSON` (Firestore/GCS credentials).
 
 Chat SSE deadlines: `AGENT_STREAM_FIRST_EVENT_TIMEOUT_SECONDS` (default `25`), `AGENT_STREAM_PROGRESS_HEARTBEAT_SECONDS` (default `20`), `AGENT_STREAM_MAX_DURATION_SECONDS` (default `150`), and `AGENT_STREAM_CANCEL_GRACE_SECONDS` (default `2`) bound silent setup/producer work and keep valid long tool calls observable. Values must be positive.
 
@@ -66,7 +66,7 @@ backend/
                           #   app generation, fair-use classification, usage tracking
       clients.py          #     Model instances: OpenAI (gpt-4.1-mini, o4-mini), Anthropic (claude-sonnet-4-6),
                           #     OpenRouter (gemini-flash), with prompt caching and usage callbacks
-    stt/                  #   Speech-to-text (7 files): Deepgram streaming, VAD gating, speech profiles,
+    stt/                  #   Speech-to-text (7 files): Parakeet/Modulate and explicit self-hosted Deepgram streaming, VAD gating, speech profiles,
                           #   pre-recorded batch transcription, speaker embeddings
     conversations/        #   Conversation lifecycle (6 files): ingestion, memory extraction, action items,
                           #   merge, post-processing, search
@@ -123,14 +123,15 @@ backend (main.py)
   ├── ws ──► pusher (pusher/)
   ├── ──────► diarizer (diarizer/)
   ├── ──────► vad (modal/)
-  ├── ──────► deepgram (self-hosted or cloud)
   ├── ──────► parakeet (parakeet/)
+  ├── ──────► modulate (managed API)
+  ├── ──────► deepgram-self-hosted (explicit streaming policy only)
   ├── ──────► nllb-translation (nllb_translation/)
   └── ──────► llm-gateway (llm_gateway/main.py)
 
 pusher
   ├── ──────► diarizer (diarizer/)
-  └── ──────► deepgram (cloud)
+  └── ──────► parakeet / modulate (STT)
 
 agent-proxy (agent-proxy/main.py)
   └── ws ──► user agent VM (private IP, port 8080)
@@ -147,17 +148,20 @@ memory-maintenance-job (modal/memory_maintenance_job.py)  [cron]
 agent-vm-reaper (backend/charts/agent-vm-reaper)  [cron]
 ```
 
-Helm charts: `backend/charts/{agent-proxy,agent-vm-reaper,backend-listen,backend-secrets,deepgram-self-hosted,diarizer,llm-gateway,monitoring,nllb-translation,parakeet,pusher,vad}/`
+Helm charts: `backend/charts/{agent-proxy,agent-vm-reaper,backend-listen,backend-secrets,deepgram-self-hosted,diarizer,llm-gateway,monitoring,nllb-translation,parakeet,pusher,vad}/`.
 
-- **backend** (`main.py`) — REST API. Streams audio to pusher via WebSocket (`utils/pusher.py`). Calls diarizer for speaker embeddings (`utils/stt/speaker_embedding.py`). Calls vad for voice activity detection and speaker identification (`utils/stt/vad.py`, `utils/stt/speech_profile.py`). Calls deepgram or parakeet for STT (`HOSTED_PARAKEET_API_URL`, `utils/stt/streaming.py`). Calls NLLB translation when `HOSTED_TRANSLATION_API_URL` is set and NLLB is selected (`utils/translation.py`).
+Serving STT provider/surface policy and canonical model order are owned exclusively by `config/stt_provider_policy.py`; deployment values are validated against it.
+
+- **backend** (`main.py`) — REST API. Streams audio to pusher via WebSocket (`utils/pusher.py`). Calls diarizer for speaker embeddings (`utils/stt/speaker_embedding.py`). Calls vad for voice activity detection and speaker identification (`utils/stt/vad.py`, `utils/stt/speech_profile.py`). Default STT is Parakeet or Modulate (`HOSTED_PARAKEET_API_URL`, `MODULATE_API_KEY`); self-hosted Deepgram is a separately gated streaming option (`DEEPGRAM_SELF_HOSTED_*`, `utils/stt/streaming.py`). Calls NLLB translation when `HOSTED_TRANSLATION_API_URL` is set and NLLB is selected (`utils/translation.py`).
 - **hosted MCP OAuth** (`routers/mcp_sse.py`) — Provider-neutral OAuth for `/v1/mcp/sse`. Configure public or confidential clients with `MCP_OAUTH_CLIENTS_JSON`; allowlist the exact connector callback URI from the provider. The temporary `MCP_OAUTH_CHATGPT_*` envs still define the legacy confidential ChatGPT test client, and `MCP_OAUTH_PUBLIC_*` can expose a no-secret PKCE public client. Also set `MCP_AUTHORIZATION_SERVER_URL`, optional `MCP_RESOURCE_URL`, and token TTL env vars.
 - **llm-gateway** (`llm_gateway/main.py`) — Internal FastAPI service for Omi-managed LLM auto lanes. Called by backend with service auth for `omi:auto:*` chat-completions routes; not exposed to clients.
-- **pusher** (`pusher/main.py`) — Receives audio via binary WebSocket protocol. Calls diarizer and deepgram for speaker sample extraction (`utils/speaker_identification.py` → `utils/speaker_sample.py`).
+- **pusher** (`pusher/main.py`) — Receives audio via binary WebSocket protocol. Calls diarizer and the configured Parakeet/Modulate STT provider for speaker sample extraction (`utils/speaker_identification.py` → `utils/speaker_sample.py`).
 - **agent-proxy** (`agent-proxy/main.py`) — GKE. WebSocket proxy at `wss://agent.omi.me/v1/agent/ws`. Validates Firebase ID token, looks up `agentVm` in Firestore, proxies bidirectionally to VM's `ws://<ip>:8080/ws`.
 - **diarizer** (`diarizer/main.py`) — GPU. Speaker embeddings at `/v2/embedding`. Called by backend and pusher (`HOSTED_SPEAKER_EMBEDDING_API_URL`).
 - **vad** (`modal/main.py`) — GPU. `/v1/vad` and `/v1/speaker-identification`. Called by backend only.
-- **deepgram** — STT. Streaming uses self-hosted (`DEEPGRAM_SELF_HOSTED_URL`) or cloud based on `DEEPGRAM_SELF_HOSTED_ENABLED`. Pre-recorded always uses Deepgram cloud. Called by backend and pusher.
-- **parakeet** (`parakeet/`) — GPU STT service for streaming and pre-recorded transcription. Called by backend when `HOSTED_PARAKEET_API_URL` is set and parakeet is selected.
+- **deepgram-self-hosted** — GPU STT deployment behind an explicit non-cloud endpoint. Hosted Deepgram is disabled on every serving surface; do not add a hosted API key or `api.deepgram.com` fallback.
+- **parakeet** (`parakeet/`) — GPU STT service for streaming and pre-recorded transcription. Called by backend when `HOSTED_PARAKEET_API_URL` is set and Parakeet is selected.
+- **modulate** — Managed STT provider for configured languages. Called by backend when `MODULATE_API_KEY` is configured and Modulate is selected.
 - **nllb-translation** (`nllb_translation/`) — GPU translation service. Called by backend when `HOSTED_TRANSLATION_API_URL` is set and NLLB is selected.
 - **backend-sync** (`main.py`, same image as backend) — Cloud Run admission service for `/v2/sync-local-files`. The server classifies whole batches: recordings no more than six hours old enter `sync-jobs` (fresh), while older or untrusted batches enter `sync-backfill` and the scale-to-zero **backend-sync-backfill** worker. Fresh keeps its bounded inline fallback; backfill never falls into fresh/inline capacity. Backfill defaults to one in-flight job per UID, four processed speech hours per UID/day, 555 processed speech hours globally/day, a 30-day lookback, and four queue workers. Live fair-use reads only `realtime + sync_fresh`; `sync_backfill` is separately metered. A 45-day Firestore content ledger protects transcription and usage side effects across job expiry and re-upload. Audio playback merges (`/v1/sync/audio/*`) follow the same pattern via queue `audio-merge` building 30-day MP3 artifacts under `playback/` (`AUDIO_MERGE_DISPATCH_MODE`) — per-part files plus one dense per-conversation `conversation.mp3` whose spans manifest + audio_files fingerprint are stamped on the conversation doc (`conversation_audio`); a fingerprint mismatch after late chunks re-enqueues the build. In production, account deletion requires `ACCOUNT_DELETION_DISPATCH_MODE=cloud_tasks` and complete Cloud Tasks bindings to enqueue opaque job IDs to queue `account-deletion`, which posts `/v1/users/account-deletion-wipes/run`; startup rejects inline or incomplete configuration, reconciliation only re-dispatches tasks so the OIDC handler is the sole wipe executor, and the post-deploy queue-drain window accepts the former sync OIDC audience only for legacy UID payloads. API success is returned only after the deletion marker is persisted and the wipe task is durably enqueued.
 - **notifications-job** (`modal/job.py`) — Cron job, reads Firestore/Redis, sends push notifications and runs X connector sync. It has no canonical maintenance flags or Typesense secrets; its deploy workflow removes only those retired bindings and preserves unrelated notification/X-sync env.