Optimize GPT-5.6 caching and route scheduled memory work through Flex (#11648)

## Summary

- restore explicit GPT-5.6 prompt caching through the LangChain/OpenAI
wire payload, with stable versioned cache keys and a default-off rollout
flag
- put all eligible scheduled Flex work behind one live switch:
Short-term to Long-term promotion, required L2 memory processing, and
scheduled X memory extraction
- preserve the prior Standard path exactly when the shared switch is
off; manual and OAuth-triggered X syncs remain Standard regardless of
the switch
- isolate the three workloads on dedicated gateway lanes with 900-second
request timeouts, zero SDK retries, generation/apply fencing, and
durable deferral rather than silent Standard fallback

## One Flex switch

The deployment capability and live Firestore control are fail-closed
gates:

- capability: `OMI_BACKGROUND_FLEX_CAPABLE=true` only on the scheduled
memory-maintenance and notifications jobs
- live control: `llm_runtime_controls/background_flex`
- accepted document shape: exactly `enabled` (boolean) and `generation`
(positive integer)
- missing, malformed, or disabled control: all three workloads use their
existing Standard logic
- enabled control: all three eligible scheduled workloads use Flex;
capacity/deadline deferrals remain pending for a later scheduled run and
never silently fall back to Standard

Initial disabled control:

```json
{
  "enabled": false,
  "generation": 1
}
```

Enable all scheduled Flex processing without redeploying:

```json
{
  "enabled": true,
  "generation": 2
}
```

Increment `generation` on every change. A generation change fences
in-flight results before durable apply. Turning `enabled` off restores
the old Standard routes for new work without a redeployment.

No live Firestore control or deployment is changed by this PR.

## Safety and latency

- dedicated gateway routes: `memory_conflict_flex`, `memory_l2_flex`,
and `x_memory_extraction_flex`
- 15-minute provider timeout, 20-minute leases, one-hour Cloud Run job
budgets, and a five-minute completion margin
- zero SDK retries on Flex calls, preventing a request from consuming
two full Flex windows
- memory maintenance limits the Flex-enabled run to one user and one
required L2 item so worst-case work fits the job budget
- deferred L2 work releases its claim without consuming the existing
quality-failure retry budget
- scheduled X acknowledges raw posts only after extraction, writes, and
generation fencing succeed; deferred posts stay pending
- the notifications deployment verifies and probes the private LLM
gateway route and carries the required VPC configuration

OpenAI documents Flex as lower-cost processing with slower responses and
occasional resource-unavailable errors, intended for non-production or
lower-priority asynchronous work:
https://developers.openai.com/api/docs/guides/flex-processing

## Product invariants affected

- INV-MEM-1

## Failure class (fixes)

Failure-Class: none

## Prompt-cache rollout

1. Deploy the LLM gateway revision first so generated GPT-5.6 routes no
longer contain the legacy `prompt_cache_retention: 24h` field.
2. Deploy backend-listen with
`OMI_LLM_GPT56_EXPLICIT_CACHE_ENABLED=false`.
3. Enable the prompt-cache flag in a reviewed rollout change and watch
cache writes/hits, provider 4xx errors, finalizer retries, and structure
latency.
4. Roll back by setting the flag to `false`.

## Review

Cursor Grok 4.6 reviewed the implementation four times through agentctl:

- initial promotion review:
`exec-curious-involve-shoulder-tower-panda-attend`
- promotion post-fix review: `exec-viable-tool-profit-polar-small-chef`
- shared-switch review: `exec-chunk-banana-abstract-potato-grab-else`
- final shared-switch post-fix review:
`exec-seven-dragon-enough-recycle-ginger-orphan`

The shared-switch reviews drove zero-retry Flex invocation, whole-job
budgeting, private-gateway routing and deployment verification,
cursor-safe memory limits, pending-until-applied X acknowledgement, and
preservation of Flex deferrals across extraction boundaries.

## Verification

- 304 focused post-rebase tests passed: 209 routing, X, L2, cron,
gateway, runtime, and prompt-cache tests plus 95 QoS tests
- local pre-push gate passed after typecheck completed with 0 errors;
runtime-image closure, workflow/runtime validation, generated manifests,
and selected backend tests passed
- runtime environment validation passed for development and production
- Python compile validation and `git diff --check` passed
- GitHub CI is authoritative for the full suite

Line-Count-Exception: backend/utils/memory/canonical_consolidation.py |
1927 -> 2079 | adds transactional Flex deferral release and apply
fencing beside the consolidation retry state it protects

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/BasedHardware/omi/pull/11648?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->


Line-Count-Exception: backend/utils/llm/conversation_processing.py |
1498 -> 1510 | gateway-mode cache_key split review fix: explicit
formatted-flag docstrings plus the gateway-mode/explicit-cache split in
get_reprocess_transcript_structure and get_app_result

Author: Git-on-my-level

.github/workflows/gcp_notifications_job.yml

@@ -80,13 +80,30 @@ jobs:
       - name: Install Python deps for deploy scripts
         run: python3 -m pip install -q pyyaml
 
+      - name: Get GKE credentials for gateway serving gate
+        uses: google-github-actions/get-gke-credentials@v3
+        with:
+          cluster_name: ${{ vars.GKE_CLUSTER }}
+          location: ${{ env.REGION }}
+          project_id: ${{ vars.GCP_PROJECT_ID }}
+
+      - name: Verify LLM Gateway serving data plane
+        id: gateway-serving
+        run: |
+          python3 backend/scripts/verify-llm-gateway-serving.py \
+            --environment="${{ vars.ENV }}" \
+            --project="${{ vars.GCP_PROJECT_ID }}" \
+            --region="${{ env.REGION }}" \
+            --github-output "$GITHUB_OUTPUT"
+
       - name: Render backend runtime env
         id: runtime-env
         env:
           # Prod network flags are env_var-backed; without these the renderer
           # exits before notifications_job_* outputs are emitted (same as gcp_backend.yml).
           CLOUD_RUN_VPC_NETWORK: ${{ vars.CLOUD_RUN_VPC_NETWORK }}
           CLOUD_RUN_VPC_SUBNET: ${{ vars.CLOUD_RUN_VPC_SUBNET }}
+          OMI_LLM_GATEWAY_URL: ${{ steps.gateway-serving.outputs.gateway_url }}
         run: |
           python3 backend/scripts/render_backend_runtime_env.py --env ${{ vars.ENV }} --job notifications-job >> "$GITHUB_OUTPUT"
 
@@ -119,6 +136,19 @@ jobs:
           docker push gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:latest
           docker push gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${{ steps.image-tag.outputs.short_sha }}
 
+      - name: Probe X Flex gateway lane from the Cloud Run VPC
+        run: |
+          bash backend/scripts/probe-llm-gateway-from-cloud-run.sh \
+            --project "${{ vars.GCP_PROJECT_ID }}" \
+            --region "${{ env.REGION }}" \
+            --image "gcr.io/${{ vars.GCP_PROJECT_ID }}/${{ env.SERVICE }}:${{ steps.image-tag.outputs.short_sha }}" \
+            --gateway-url "${{ steps.gateway-serving.outputs.gateway_url }}" \
+            --network "${{ vars.CLOUD_RUN_VPC_NETWORK }}" \
+            --subnet "${{ vars.CLOUD_RUN_VPC_SUBNET }}" \
+            --vpc-egress private-ranges-only \
+            --lane omi:auto:x-memory-extraction-flex \
+            --name-suffix "${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+
       - name: Deploy to Cloud Run
         id: deploy
         uses: google-github-actions/deploy-cloudrun@v3
@@ -132,6 +162,8 @@ jobs:
           # Strip retired maintenance/Pusher keys, plus any literal shadowing a
           # Secret Manager binding (no-op on a clean target).
           flags: >-
+            ${{ steps.runtime-env.outputs.cloud_run_flags }}
+            --task-timeout=3600s
             --remove-env-vars=MEMORY_MODE,MEMORY_ENABLED_USERS,MEMORY_V3_GET_ENABLED,MEMORY_CANONICAL_MAINTENANCE_ENABLED,MEMORY_CANONICAL_CONSOLIDATION_ENABLED,MEMORY_CANONICAL_PROMOTION_CRON_ENABLED,MEMORY_CANONICAL_PROMOTION_CRON_INTERVAL_HOURS,MEMORY_CANONICAL_PROMOTION_FAST_TRACK_ENABLED,MEMORY_TYPESENSE_COLLECTION,TYPESENSE_HOST,TYPESENSE_HOST_PORT,TYPESENSE_API_KEY,HOSTED_PUSHER_API_URL,${{ steps.runtime-env.outputs.notifications_job_secret_names }}
 
       # If required, use the Cloud Run url output in later steps

backend/charts/backend-listen/dev_omi_backend_listen_values.yaml

@@ -149,6 +149,8 @@ env:
     value: "3600"
   - name: OMI_LLM_GATEWAY_FEATURE_MODE
     value: "gateway"
+  - name: OMI_LLM_GPT56_EXPLICIT_CACHE_ENABLED
+    value: "false"
   - name: OMI_LLM_CHAT_AGENT_ROUTE
     value: "gateway"
   - name: PUBLIC_SHARED_CONVERSATION_CHAT_MODE

backend/charts/backend-listen/prod_omi_backend_listen_values.yaml

@@ -129,6 +129,8 @@ env:
     value: "prod"
   - name: OMI_LLM_GATEWAY_FEATURE_MODE
     value: "gateway"
+  - name: OMI_LLM_GPT56_EXPLICIT_CACHE_ENABLED
+    value: "false"
   - name: OMI_LLM_CHAT_AGENT_ROUTE
     value: "gateway"
   - name: PUBLIC_SHARED_CONVERSATION_CHAT_MODE

backend/charts/llm-gateway/templates/backendconfig.yaml

@@ -4,6 +4,10 @@ kind: BackendConfig
 metadata:
   name: {{ .Values.service.backendConfig }}
 spec:
+  # Flex requests have a route-level 15-minute deadline. Keep the ingress
+  # connection alive slightly longer; ordinary lanes retain shorter gateway
+  # deadlines from their route artifacts.
+  timeoutSec: 960
   healthCheck:
     checkIntervalSec: 10
     timeoutSec: 5

backend/deploy/runtime_env.yaml

@@ -44,6 +44,9 @@ environments:
           OMI_LLM_GATEWAY_FEATURE_MODE:
             value: gateway
             category: rollout
+          OMI_LLM_GPT56_EXPLICIT_CACHE_ENABLED:
+            value: 'false'
+            category: rollout
           OMI_LLM_CHAT_AGENT_ROUTE:
             value: gateway
             category: rollout
@@ -857,7 +860,12 @@ environments:
           - HOSTED_PUSHER_API_URL
       jobs:
         notifications-job:
+          flags:
+            --task-timeout: 3600s
           secrets:
+            OMI_LLM_GATEWAY_SERVICE_TOKEN:
+              secret: OMI_LLM_GATEWAY_SERVICE_TOKEN
+              version: latest
             OPENAI_API_KEY:
               secret: OPENAI_API_KEY
               version: latest
@@ -871,6 +879,13 @@ environments:
               secret: PINECONE_API_KEY
               version: latest
           env:
+            OMI_LLM_GATEWAY_URL:
+              env_var: OMI_LLM_GATEWAY_URL
+              default: http://127.0.0.1:9
+              category: service_discovery
+            OMI_BACKGROUND_FLEX_CAPABLE:
+              value: 'true'
+              category: memory_rollout
             PINECONE_INDEX_NAME:
               category: integrations
               value: memories-backend-dev
@@ -947,6 +962,9 @@ environments:
             MEMORY_CANONICAL_CONSOLIDATION_ENABLED:
               value: 'true'
               category: memory_rollout
+            OMI_BACKGROUND_FLEX_CAPABLE:
+              value: 'true'
+              category: memory_rollout
             GOOGLE_CLOUD_PROJECT:
               value: based-hardware
             MEMORY_CANONICAL_GRAPH_BACKFILL_PAGE_SIZE:
@@ -1003,6 +1021,9 @@ environments:
           OMI_LLM_GATEWAY_FEATURE_MODE:
             value: gateway
             category: rollout
+          OMI_LLM_GPT56_EXPLICIT_CACHE_ENABLED:
+            value: 'false'
+            category: rollout
           OMI_LLM_CHAT_AGENT_ROUTE:
             value: gateway
             category: rollout
@@ -1753,7 +1774,12 @@ environments:
               category: rollout
       jobs:
         notifications-job:
+          flags:
+            --task-timeout: 3600s
           secrets:
+            OMI_LLM_GATEWAY_SERVICE_TOKEN:
+              secret: OMI_LLM_GATEWAY_SERVICE_TOKEN
+              version: latest
             OPENAI_API_KEY:
               secret: OPENAI_API_KEY
               version: latest
@@ -1767,6 +1793,13 @@ environments:
               secret: PINECONE_API_KEY
               version: latest
           env:
+            OMI_LLM_GATEWAY_URL:
+              env_var: OMI_LLM_GATEWAY_URL
+              default: http://127.0.0.1:9
+              category: service_discovery
+            OMI_BACKGROUND_FLEX_CAPABLE:
+              value: 'true'
+              category: memory_rollout
             PINECONE_INDEX_NAME:
               category: integrations
               value: memories-backend
@@ -1798,6 +1831,7 @@ environments:
               version: latest
           flags:
             --remove-env-vars: MEMORY_ENABLED_USERS,MEMORY_CANONICAL_PROMOTION_CRON_ENABLED,MEMORY_CANONICAL_PROMOTION_CRON_INTERVAL_HOURS,MEMORY_CANONICAL_PROMOTION_FAST_TRACK_ENABLED
+            --task-timeout: 3600s
           env:
             OMI_LLM_GATEWAY_URL:
               env_var: OMI_LLM_GATEWAY_URL
@@ -1836,6 +1870,9 @@ environments:
             MEMORY_CANONICAL_CONSOLIDATION_ENABLED:
               value: 'true'
               category: memory_rollout
+            OMI_BACKGROUND_FLEX_CAPABLE:
+              value: 'true'
+              category: memory_rollout
             OMI_LLM_GATEWAY_ALLOW_PROD_FEATURE_MODE:
               value: 'true'
               category: rollout

backend/deploy/runtime_env/_base.yaml

@@ -51,6 +51,9 @@ environment_shared:
         OMI_LLM_GATEWAY_FEATURE_MODE:
           value: gateway
           category: rollout
+        OMI_LLM_GPT56_EXPLICIT_CACHE_ENABLED:
+          value: 'false'
+          category: rollout
         OMI_LLM_CHAT_AGENT_ROUTE:
           value: gateway
           category: rollout
@@ -488,7 +491,12 @@ environment_shared:
             category: memory_rollout
     jobs:
       notifications-job:
+        flags:
+          --task-timeout: 3600s
         secrets:
+          OMI_LLM_GATEWAY_SERVICE_TOKEN:
+            secret: OMI_LLM_GATEWAY_SERVICE_TOKEN
+            version: latest
           OPENAI_API_KEY:
             secret: OPENAI_API_KEY
             version: latest
@@ -502,6 +510,13 @@ environment_shared:
             secret: PINECONE_API_KEY
             version: latest
         env:
+          OMI_LLM_GATEWAY_URL:
+            env_var: OMI_LLM_GATEWAY_URL
+            default: http://127.0.0.1:9
+            category: service_discovery
+          OMI_BACKGROUND_FLEX_CAPABLE:
+            value: 'false'
+            category: memory_rollout
           PINECONE_INDEX_NAME:
             category: integrations
           OMI_ENV_STAGE:
@@ -567,3 +582,6 @@ environment_shared:
           MEMORY_CANONICAL_CONSOLIDATION_ENABLED:
             value: 'true'
             category: memory_rollout
+          OMI_BACKGROUND_FLEX_CAPABLE:
+            value: 'false'
+            category: memory_rollout

backend/deploy/runtime_env/dev.overlay.yaml

@@ -418,7 +418,11 @@ overlay:
             version: latest
     jobs:
       notifications-job:
+        flags:
+          --task-timeout: 3600s
         env:
+          OMI_BACKGROUND_FLEX_CAPABLE:
+            value: 'true'
           GOOGLE_CLOUD_PROJECT:
             value: based-hardware
           PINECONE_INDEX_NAME:
@@ -439,6 +443,8 @@ overlay:
             value: 'false'
           MEMORY_CANONICAL_GRAPH_BACKFILL_ENABLED:
             value: 'false'
+          OMI_BACKGROUND_FLEX_CAPABLE:
+            value: 'true'
           MEMORY_CANONICAL_GRAPH_BACKFILL_PAGE_SIZE:
             value: '25'
           PINECONE_INDEX_NAME:

backend/deploy/runtime_env/prod.overlay.yaml

@@ -364,10 +364,16 @@ overlay:
             value: 'off'
     jobs:
       notifications-job:
+        flags:
+          --task-timeout: 3600s
         env:
+          OMI_BACKGROUND_FLEX_CAPABLE:
+            value: 'true'
           PINECONE_INDEX_NAME:
             value: memories-backend
       memory-maintenance-job:
+        flags:
+          --task-timeout: 3600s
         env:
           OMI_LLM_GATEWAY_ALLOW_PROD_FEATURE_MODE:
             value: 'true'
@@ -381,5 +387,7 @@ overlay:
             value: 'false'
           MEMORY_CANONICAL_GRAPH_BACKFILL_ENABLED:
             value: 'false'
+          OMI_BACKGROUND_FLEX_CAPABLE:
+            value: 'true'
           PINECONE_INDEX_NAME:
             value: memories-backend

backend/docs/llm/model_endpoint_inventory.yaml

@@ -37,7 +37,9 @@ model_config_features:
       - memories
       - memory_category
       - memory_conflict
+      - memory_conflict_flex
       - memory_l1
+      - memory_l2_flex
       - notifications
       - onboarding
       - openglass
@@ -47,6 +49,7 @@ model_config_features:
       - session_titles
       - smart_glasses
       - wrapped_analysis
+      - x_memory_extraction_flex
       - app_generator
       - app_integration
     structured_output:

backend/llm_gateway/config/generated_route_overrides.yaml

@@ -75,6 +75,18 @@ generated_route_overrides:
   - feature: memory_conflict
     primary: {provider: openai, model: gpt-5.6-luna}
     provider_options: {reasoning_effort: low}
+  - feature: memory_conflict_flex
+    primary: {provider: openai, model: gpt-5.6-luna}
+    provider_options: {reasoning_effort: low}
+    request_timeout_ms: 900000
+  - feature: memory_l2_flex
+    primary: {provider: openai, model: gpt-5.6-luna}
+    provider_options: {reasoning_effort: medium}
+    request_timeout_ms: 900000
+  - feature: x_memory_extraction_flex
+    primary: {provider: openai, model: gpt-5.6-luna}
+    provider_options: {reasoning_effort: low}
+    request_timeout_ms: 900000
   - feature: memory_l1
     primary: {provider: openai, model: gpt-5.6-luna}
     provider_options: {reasoning_effort: low}