Harden boundary delivery and data convergence - #10646

Problem

Several boundary paths treated an attempted send or write as a complete outcome even when the remote effect was unknown or only partially persisted.

At origin/main, the pusher client returned after a failed finalization send without invalidating the dead socket:

        except Exception as e:
            logger.error(f"Failed to send process_conversation request: {e} {self.uid} {self.session_id}")
            return False

Legacy memory search requested only the desired result count before filtering missing, locked, rejected, invalid, or malformed records:

def _legacy_search_memories(uid: str, query: str, *, limit: int = 5) -> List[MemorySearchMatch]:
    capped_limit = max(1, min(limit, 20))
    matches = vector_db.find_similar_memories(uid, query, threshold=0.0, limit=capped_limit)

Todoist task creation converted an omitted provider identity into the string "None" and reported success:

            if response.status_code in [200, 201]:
                task_data = response.json()
                return {"success": True, "external_task_id": str(task_data.get('id'))}

The same boundary problem appeared in transcript and speaker delivery, webhook retries, vector projection counts, and legacy X-source acknowledgement after partial projection.

Reachability / failure scenario

• backend/utils/listen_pusher_session.py::ListenPusherSession.request_conversation_processing, _transcript_flush, _audio_bytes_flush, and send_speaker_sample_request await socket sends during ordinary disconnect and cancellation paths. A failed socket could remain marked connected, while accepted work was discarded before _pusher_reconnect_loop could replay it. The fixed speaker path now delegates to _send_speaker_sample_request.
• backend/routers/pusher.py::_websocket_util_trigger runs transcript and speaker effects in process_transcript_queue and process_speaker_sample_queue. If the connection closes after one effect completes, the client cannot distinguish completed work from work that must be replayed.
• backend/utils/speaker_identification.py::extract_speaker_samples uploads through backend/utils/other/storage.py::upload_person_speech_sample_from_bytes before add_person_speech_sample commits the reference. A retryable database outcome previously created a new random object for the same logical delivery.
• backend/utils/memory/memory_service.py::_legacy_search_memories requested only limit vector candidates, then hydrated and filtered them. Stale, locked, rejected, invalid, invalidated, or malformed leading hits could therefore hide eligible lower-ranked results.
• backend/routers/memories.py::_purge_legacy_memories enumerated IDs through backend/database/memories.py::get_memories before calling delete_all_memories. Records excluded by that reader were deleted from Firestore without their vector IDs being available for cleanup.
• backend/routers/task_integrations.py::create_task_via_integration delegates to backend/utils/task_integrations_ops.py::create_task_internal. A provider 2xx response without an ID became "None", while a connection failure after send was not distinguished from a safe pre-send failure.
• backend/utils/x_connector.py::_extract_and_index persisted legacy memory documents and then called upsert_memory_vectors_batch. A partial legacy vector write could still let the source be counted as processed and acknowledged.
• backend/utils/webhooks.py::_post_dev_webhook retried permanent and transient responses alike, and backend/utils/app_integrations.py::_async_trigger_realtime_integrations lacked stable retry identity. In addition, backend/routers/users.py::enable_user_webhook_endpoint recorded synthetic success without clearing the process-local circuit breaker.

Fix

• Preserve finalization, transcript, audio, and speaker work across ambiguous outbound send failures and cancellation. On those delivery send failures, mark only the socket that failed as disconnected so a concurrent replacement socket is not cleared.
• Negotiate effect acknowledgements for transcript and speaker deliveries, carry stable delivery IDs through downstream integrations, and acknowledge only after owned effects complete. Redis delivery leases and done markers suppress cross-socket duplicates and are bounded by expiry.
• Reject new stable transcript and speaker effect deliveries instead of evicting accepted unacknowledged effect deliveries when those queues are full. Add a bounded graceful drain so audio metadata is flushed before dependent speaker extraction.
• Derive one opaque speaker object name from the stable delivery identity. Legacy callers without a delivery identity retain random UUID names. The code does not delete on ambiguous database failure because the commit may already have succeeded.
• Retry developer webhooks only for transport failures, 408, 425, 429, and 5xx, while reusing one receiver-visible idempotency key. Explicit re-enable or URL replacement resets persisted and process-local delivery health without recording synthetic success.
• Progressively over-fetch legacy vector candidates, hydrate each ID once, preserve vector order, and stop at a fixed candidate cap. The origin/main sibling _legacy_search_memories_mcp already over-fetched min(limit * 3, 60) before filtering; this change makes both API and MCP paths progressively backfill instead of relying on one fixed over-fetch.
• Use the exact IDs returned by the committed database delete for vector cleanup, chunk projection-repair writes below the Firestore batch limit, use one injected Firestore client for related reads and writes, and report actual vector write counts.
• Leave a legacy X source retryable after partial vector projection instead of acknowledging incomplete convergence. Canonical X projection acknowledgement behavior is unchanged.
• Reject provider success responses without a usable task identity. Classify failures as safe pre-send retries or ambiguous outcomes. For provider-create outcomes represented by CreateTaskResponse, preserve error_code, retryable, and ambiguous through the public response and generated clients. Existing authentication and token-refresh preflight HTTP errors remain unchanged. Ambiguous task creation is reported, not retried automatically.
• Expand high-risk workflow contracts and document the listen-to-pusher delivery guarantees and limitations.

Product invariants affected: INV-MEM-1.

Tests

• backend/tests/unit/utils/test_listen_pusher_session.py covers dead-socket invalidation, route-preserving replay, stable delivery IDs, cancellation, acknowledgement drain, and queue retention. Key cases include test_finalization_send_failure_disconnects_dead_socket_and_preserves_request, test_ack_capability_retains_and_replays_transcript_until_peer_completion, test_full_speaker_buffer_preserves_accepted_unacknowledged_delivery, and test_stale_socket_failure_does_not_disconnect_replacement.
• backend/tests/unit/test_listen_finalization_cloud_tasks.py proves acknowledgement occurs only after owned effects complete, retryable speaker work remains unacknowledged, done markers suppress cross-socket duplicates, and a full stable queue does not evict accepted work.
• backend/tests/unit/test_redis_db_cache_serialization.py covers lease ownership, done markers, owner-fenced release, bounded Redis timeouts, and fail-open behavior.
• backend/tests/unit/test_speaker_identification_delivery.py covers retryable extraction outcomes, missing-person retry, passing a one-sample cap to the transactional append helper, and stable hashed object identity across retries.
• backend/tests/unit/test_async_webhooks.py, test_async_app_integrations.py, test_async_http_infrastructure.py, test_webhook_auto_disable.py, and test_users_webhook_url_validation.py cover retry classification, stable idempotency keys, URL replacement, and explicit delivery-health reset without synthetic success.
• backend/tests/unit/test_memory_service_parity.py, test_memories_batch_delete.py, test_memories_delete_batch_chunk.py, test_memory_ledger.py, and test_memories_batch.py cover progressive backfill, candidate caps, injected Firestore use, exact committed delete IDs, 499-write repair chunks, and reported vector write counts.
• backend/tests/unit/test_x_memory_extraction_retry.py::test_pending_x_source_remains_unacknowledged_after_partial_legacy_projection proves partial legacy projection leaves the source retryable.
• backend/tests/unit/test_task_integrations_ops.py covers missing identities, unexpected 2xx responses, rate limits, server failures, safe pre-send failures, and ambiguous post-send failures. test_task_integration_due_date_validation.py::test_provider_failure_metadata_is_preserved_in_public_response covers response propagation.
• backend/testing/e2e/test_task_integrations.py verifies the expanded provider-create response at the HTTP boundary for success, provider 500, and connect timeout outcomes. backend/testing/e2e/test_webhooks.py explicitly disables delivery after URL replacement and verifies a disabled webhook neither calls the provider nor mutates its reset health snapshot.
• backend/tests/unit/test_openapi_contract.py, test_app_client_ts_generator.py, and test_app_client_dart_generator.py verify the public schema and generated clients, including test_wrapped_task_integrations_wire_dart_is_generated_from_app_client_openapi.
• backend/tests/unit/test_dev_api_lock_bypass.py models the committed delete count and verifies the route injects its Firestore client. test_developer_memory_adapter.py scopes the guard/read/delete ordering check to MemoryService.delete_external_memory. test_pusher_ghost_connections.py parses the actual speaker-age condition instead of relying on line wrapping. test_tools_router.py exercises the shared legacy-memory search boundary instead of stale vector internals.

Verification

Commit 4ea54d07e5d3e3e03fb3a393fe06e0c1abc1b143 has parent eb60a5175e96682185bba21829ccfeb98ba38c04, the latest origin/main used for publication. The final upstream advance touched no PR paths. The resulting patch is clean, and the complete listen session unit file passed again with 27 passed.

The maintainer-requested socket-identity regression is parameterized across receive and heartbeat failure paths:

python -m pytest -q tests/unit/utils/test_listen_pusher_session.py::test_stale_socket_failure_does_not_disconnect_replacement

For byte-identical prove-fail, only backend/utils/listen_pusher_session.py was restored from origin/main, and git diff origin/main -- backend/utils/listen_pusher_session.py printed nothing before pytest ran. Both variants failed at the intended boundary with:

AssertionError: stale socket failure disconnected the healthy replacement

Restoring the committed product file passed both variants. That product file is byte-identical between the prove-fail base and the final publication base.

Current-base behavioral verification:

• Conflict-sensitive grouped regression set: 108 passed.
• Listen and pusher mapped unit workflow: 462 passed.
• Listen and pusher end-to-end workflow: 18 passed.
• Conversation lifecycle workflow: 197 passed.
• Webhook delivery health workflow: 210 passed.
• Legacy memory shared workflow: 135 passed.
• Isolated lock-bypass workflow: 61 passed.
• Account deletion workflow: 235 non-E2E passed and 5 E2E passed.
• Task provider unit workflow: 19 passed.
• Task and webhook E2E workflow: 13 passed.
• OpenAPI and code generation tests: 49 passed.
• App-client OpenAPI export is current under the CI-pinned Linux Python 3.11.15 environment. Compatibility, TypeScript generation, and Dart generation are current.

Static and repository gates:

• Black 24.4.2 and CI-pinned Black 26.5.1 with --line-length 120 --skip-string-normalization: all changed Python files accepted.
• Pyright: 0 errors in both changed-product groups.
• Module-stub scan: 835 backend test files checked, 0 violations.
• Async-blocker scan: 452 synchronous and 87 asynchronous definitions checked, 0 findings.
• Import-side-effect scan: 744 product files checked, 0 violations.
• Product-file line-count ratchet: no increase across 19 changed product files.
• Workflow contracts and conversation lifecycle write guard: passed.
• Preflight selected 55 files, identified INV-MEM-1, and required no failure-class declaration because no fix: commit is present.
• git diff --check: clean.

The repository pre-push single-flight wrapper could not spawn its child process on Windows and exited with WinError 2. Its underlying checks were run directly with the pinned project environments before the guarded force-with-lease publication.

Failure-class protocol: no fix: commit was used, so no declaration is required.

Impact

User-visible successful outcomes stay the same; negotiated delivery paths now carry acknowledgement and stable identity metadata. The affected failure cases occur during ordinary reconnects, transient downstream failures, stale vector hydration, partial legacy X projection, or malformed provider success responses. They are not expected on every request, but their previous outcomes included dropped or duplicated side effects, leaked speaker objects, false task success, incomplete memory search, stale deleted projections, and legacy X sources acknowledged before all projections converged.

Limitations remain explicit:

• Effect-level acknowledgements require peer negotiation. Legacy peers continue through the compatibility path.
• Stable transcript and speaker effect deliveries are bounded and retained across reconnects, but are not durable across a client process loss.
• Finalization, transcript, audio, speaker, receive, and heartbeat failures are fenced to the socket that failed; stale failures from a replaced socket cannot disconnect the healthy replacement.
• Redis delivery fencing fails open for availability, so cross-socket duplicate suppression is degraded while Redis is unavailable.
• Webhook idempotency still depends on receivers honoring the stable idempotency key.
• Progressive memory backfill has a fixed candidate cap, so a severely stale vector index can still return fewer than requested.
• Cross-store projection is not transactional. Legacy X-source partial projection remains retryable; canonical X acknowledgement behavior is unchanged. Other Firestore/vector operations surface or log partial counts.
• Ambiguous external task creation is surfaced to the caller but is not automatically reconciled with the provider.