Conversation
Phase 1 of channel-agent multireplica: channel runtime lease columns, channel_inbox (dedup-safe receive log), agent_run (durable execution unit linked to the session execution lease), and channel_outbox (per-operation send ledger fenced by attempt + owner tokens). Old paths keep working — nothing consumes the new records yet. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Phase 2 of channel-agent multireplica: behind STELLA_CHANNEL_DURABLE_INGRESS, HandleIncoming lands events in channel_inbox and returns an ack instead of a live ChatStream. Coordinator.RoutePending drains it in one transaction under a channel advisory lock: messages resolve identity/agent/binding without an agent.Service (session-access PEP + binding advisory lock) and become queued agent_runs pinned to session and reply address; /new rotates the binding in route order under its command receipt; /abort flags the execution lease. received-but-unready events block only their own chat. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Phase 3 of channel-agent multireplica: a bounded worker claims each queued agent_run together with the session execution lease in one transaction — orphaned running rows for the session are interrupted there, and the run<->lease link is written before commit. sessionexecution gains Adopt (worker-side lease construction) and a FinishExtra callback so the run's terminal state and reply outbox ops commit inside the same finish transaction as the activity close and execution-row delete. Runtime admission reuses a lease already present in context. The reaper now terminates the linked lease atomically with marking the run interrupted; nothing re-queues a dead run. The channel side re-derives authority from the run's persisted actor facts and takes a fresh Use decision on the exact session before executing — archived targets finish as target_gone. Loops start only when STELLA_CHANNEL_DURABLE_INGRESS is set. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Phase 4 of channel-agent multireplica: channel_outbox is now consumed. pkgchannel gains OperationSender — one durable op maps to one platform call returning a receipt or a SendError classified retryable / permanent / unknown. The dispatcher claims due ops under an attempt token, sends outside the claim transaction, and completes fenced by that token; depends_on rows gate ordering so a retried chunk can never be overtaken. Reply production splits at op level (ReplyOps takes a platform length budget). Telegram is the first migrated adapter: send_text with markdown->plain fallback, tele.Error codes classify permanent vs retryable, transport failures are unknown — never resent blindly. The gateway resolves the running adapter through the notifier registry; replicas hosting no channels dispatch nothing. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Phase 5 of channel-agent multireplica. With STELLA_CHANNEL_DURABLE_INGRESS the plugin host no longer starts every enabled channel unconditionally: applyChannel goes through ChannelLeases.Ensure, which claims the channel.runtime_token/lease_until pair — losers (and DB-down replicas) treat the channel as disabled and never poll or send. A tracker Run loop renews held leases, sweeps claimable channels into reconcile, and drops lost leases back through reconcile so the poller stops promptly. Disabled or deleted intent releases the lease; the owner's applied config revision is stamped fenced by its token. Outbox dispatch now requires the held token (OwnerTokenSource) so a fenced replica cannot send, and the route/dispatch sweep covers every enabled channel — routing is claim-based, not ownership-bound. test/testbed gains ReplicaSet: N stellad processes on one PostgreSQL + shared vault key, instance 0 owning the embedded cluster. The A->B->C takeover acceptance test still needs a controllable test channel adapter. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ters Phase 6/7 core of channel-agent multireplica. ctx_session_event now records every published turn event under a session-scoped contiguous sequence (advisory-locked), with run linkage so cross-replica watchers tail by cursor; SessionHub stays the local fast path. The SSE attach endpoint falls back to the durable log when the session has an open run on another replica instead of answering 204. All six channel adapters implement pkgchannel.OperationSender: Telegram, Discord, Feishu (idempotent create uuid), QQ (scope-addressed C2C/group), Weixin (context_token via new durable Address.Token + deterministic client_id dedup), DingTalk (session webhook on the address token — the platform's only send credential). Envelope extras carry platform reply credentials across replicas. OutboundAddress gains Scope and Token. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…es session events Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The flag is consulted in the composition root before ServerConfig fields reach the constructors it configures; removing it is the rollout cleanup. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ursor A reconnecting watcher sends the standard SSE cursor; the durable replay tail honors it and answers 409 when the cursor fell behind the retained window so the client rebuilds from the transcript. Design doc for the durable channel path added under docs/design. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Review rework pass over the durable ingress pipeline: - Default-off regressions: guard the durable SSE fallback when no event store is wired (idle sessions panicked), and bind lease adoption to the target session so a child turn can never finish its parent's execution. - Single completer (plan D4): an adopted lease is finished only by the run worker, after the executor drains the reply stream — run terminal state, history and reply ops commit atomically; the forwarder keeps session bookkeeping and persists the terminal error event. - Ingress identity: command candidates only route as commands on a real slash prefix; the dedup event key is scoped by chat/thread; adapters no longer send empty acks for persisted events; the receiving bot's platform account is pinned as source_account_key and re-checked at dispatch via OwnsAccount so a swapped credential cannot deliver the old account's replies. - Send fencing: claim re-validates the channel runtime token inside the tx, each op is re-admitted before its SDK call, and a failed renew drops the local token. Renewal now returns desired state — the owner re-evaluates on revision drift or disable, and Apply failures release the lease instead of parking it. - Worker queue: per-session head-of-line enforced (a candidate never bypasses an earlier queued predecessor), archived targets fail terminally instead of wedging the sweep, actor roles are re-derived from current user state at execution, and the generic execution reaper interrupts a linked run in the same transaction. - Event stream: OpenRunID prefers the running run over queued ones, the tail exits on its own run's terminal state, seq lives on a monotonic ctx_conversation counter that pruning cannot rewind, durable replays emit SSE id: lines, and the web transport stores/replays the cursor as Last-Event-ID. The nine review reproductions land as repo regression tests. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Collaborator
Author
Review rework (commit
|
…cation inventory - Channel lease renew/sweep and RunDurableLoops are ingress-adjacent: move them behind applyManagedChannelPlugins so pollers and the outbox dispatcher only start once backends and channel runtimes exist (fixes TestRunServerStartsBackendsBeforeIngress). - Allowlist internal/channel/run_executor.go in the agent-invocation inventory: the durable worker re-checks the persisted actor via access.Use against the exact session before svc.Chat, which is the access-authorized adapter the inventory requires. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
📊 Coverage ReportTotal coverage: 62.1% (generated files excluded) Lowest-covered entries (first 200) |
- Ingress: BotAccountKey set at the single incomingMsg builders (Telegram
covers commands/media, Discord slash interactions); Telegram handleText
stops sending empty durable acks as platform replies.
- Queue: per-candidate claim tx — FIFO and target checks run before any
lease is taken; a dead target fails terminally in its own commit and a
skipped candidate leaves no unadopted lease behind.
- Lease: renew errors now reconcile instead of only dropping the token, so
an unverifiable owner stops its poller; async start failures surface via
runtime snapshots and trigger reconcile; Apply-time error snapshots no
longer earn the running stamp.
- Events: session seq counter actually wired into Append (the previous
query edit was a silent no-op) with backfill in migration 55; SSE cursor
is run-scoped ("runID:seq") and emitted after the event's last frame;
queued-only sessions attach instead of 204; failed/interrupted runs emit
a terminal error instead of a clean finish; adopted leases no longer
commit last_turn_result early; web clears the saved cursor on 409.
Collaborator
Author
Review rework round 2 (commit
|
- bot runtime: capture the unexpected-Start-failure bit before our own opCancel cancels the poll context, so an async handshake failure lands in an error snapshot that the lease tracker will retry. - channel lease: on an unverifiable renew, stop the local runtime before reconciling — the reconcile itself may fail while the DB is down, and a live ingress must not outlive a takeover. - run completion: result vocabulary unified on the activity contract (success/canceled/error) so a finished durable run no longer renders as unexplained idle. - web transport: a 409 resume clears the saved cursor and retries once without Last-Event-ID, so a truncated log reconnects instead of parking the chat in error.
Collaborator
Author
Review rework round 3 (commit
|
…t vocab in test - ErrNoRows (lease moved to another owner) now runs the same drop → stopLocal → reconcile path as unverifiable renews; a lost owner must not keep ingesting while the reconcile read fails. - single_finisher_test's hook assertion moved to the success vocabulary the worker now writes.
The adopted-lease chat path previously appended the final assistant message through mem.Append before the worker's finish — a run could show history while its terminal state, outbox reply, and activity result were still uncommitted. Now the runtime defers every non-group durable append into a ctx-carried DeferredTurnStore (installed by the worker), and the worker commits those rows through the lcm provider's new AppendSessionTurn inside the same transaction that finishes the run and appends the outbox reply. - AppendSessionTurn is get-only inside the caller tx: the conversation row is pinned by routing, and the get-or-create race re-read cannot run under an aborted transaction. - The user message stays an immediate append — it is ingress-side truth and predates execution. - Regressions: TestWorkerDeferredHistoryJoinsFinishTransaction (real Runtime → lcm → history+run+outbox one commit) and TestWorkerDeferredHistoryFailureBlocksCompletion (append failure → nothing visible).
The earlier failure test returned before any write; this one writes the real history through lcm inside the finish transaction and then fails the outbox hook, asserting run/history/outbox are all invisible. Also adds the testchan adapter (env-gated STELLA_TEST_CHANNELS) that lets testbed tests drive a real adapter → inbox → run → outbox → fake platform loop, and retracts the stale "Phase 0-5 fully" delivery claim in the plan.
Two real-process tests drive the full durable chain: a test-only channel adapter (gated on STELLA_TEST_CHANNELS) polls a fake platform's /poll and posts outbound ops to /send. - TestDurableChannelLoop: event → inbox → routed → real model call → outbox → platform send; asserts exactly one inbox/run/outbox op, one model request, redelivery dedup, and /new over durable ingress. - TestDurableChannelRestartBeforeSend: run completes, send held open, replica killed and restarted — the pending op is delivered after restart with zero additional model calls. Also registers testchan in BuiltinPlugins (env-gated) so the native admission gate recognizes the platform, and gives it a GuestPolicy decoder for unlinked-DM tests.
TestDurableChannelThreeReplicas runs four real stellad processes on one embedded PostgreSQL: D hosts the DB and API, A owns the channel lease, B runs the only worker, C waits as lease contender. Asserts bind roles to process identities: run.worker_id names B's pid, the fake platform sees send tag A before the kill and C after, and the restarted ex-owner never owns or sends while C's lease lives. Role pinning knobs: STELLA_CHANNEL_LEASE=off keeps fencing but skips the ownership race; STELLA_RUN_WORKER=off skips the run worker loop. Worker ids now embed host+pid so worker_id is a real process identity, and testchan sends carry STELLA_TESTCHAN_TAG so the test can attribute each send to a replica.
- TestDurableChannelSameEventHandoff: A receives, B's worker is pinned mid-model by a fake gate, A is killed, C takes the lease, the gate releases and B completes the ORIGINAL run — C sends its reply. Asserts one inbox/run/model-call and no send by the dead owner. - TestDurableChannelStaleOwnerPaused: A is SIGSTOP-paused holding its token (not restarted), C claims the expired lease, A resumes and is fenced out — a post-resume send is delivered by C only. - testbed gains Pause/Resume (process-group SIGSTOP/SIGCONT) and PID.
In durable mode (STELLA_CHANNEL_DURABLE_INGRESS) POST
/api/agents/{a}/sessions/{s}/messages enqueues an agent_run instead of
running the turn on the receiving replica, then streams the run's
persisted events over the same SSE contract. Idempotency-Key maps onto
run request_key so a retried send attaches rather than re-executing.
- sessionaccess.PrepareDurableSend reuses the exact Send admission path
(use check, command interception, archived rejection).
- run executor accepts Input.ExcludedTools for per-turn tool deltas.
- The finish hook skips outbox ops for channel-less (web) runs.
- POST .../stop flags the durable execution lease and cancels queued
runs, so a turn executing on another replica aborts.
Tests: TestDurableWebSend (send lands on D, executes on B, SSE replays
durable events, idempotent resend returns the original reply),
TestDurableWebCancel (stop on D cancels the running turn on B).
- StreamGroupEvents gains a 1.5s DB poll fallback: a group turn executing on another replica still surfaces its accepted messages and turn frames to this replica's subscribers. Terminal turn states are resolved from the persisted dispatch row (done/held/silent/failed), not guessed. - AbortGroupTurn now also flags the session execution lease, so a group turn running on another process aborts at its next lease check. - run.Store gains EnqueueDirect and CancelSessionTurn (the cross-replica /abort transaction); server Deps takes the narrow store, not pgxpool. - LatestTerminalGroupDispatchStates query for the poll fallback.
In durable mode notify.Dispatcher resolves targets from the channel config store and commits a "notify" outbox op per channel instead of calling the local adapter; the lease owner performs the platform send on any replica. - outbox: OpNotify kind + NotifyOp builder; Notification gains DedupKey so a retried scheduler/goal notification can reuse its queued op. - dispatch skips the source-account fence for notify ops (no triggering account exists; the channel's current bot identity is always right). - All six platform adapters plus testchan handle "notify" by delegating to their existing Notify implementation. - NotifyUser resolves the target channel by platform type from config rows in durable mode — the notifying replica need not host the channel.
In durable mode the accepted group reply becomes a send_group_reply channel_outbox op (delivery key = dispatch id) instead of an in-process publisher call. The dispatch row stays 'running' until the op reaches a terminal state, then pollPublishOutcomes marks it published, finalizes the canonical message, and completes the dispatch — on any replica. - choutbox.OpSendGroupReply + GroupReplyPayload: serializes the full replay (text/reasoning/tool events, reply-to, delivery id, requester) so the owner's adapter calls its existing Publish unchanged. - enqueueAccepted commits op + publish_started in one tx; re-enqueue dedups on the delivery key after a crash. - Web replies publish no-op immediately (the event log is their egress). - Five adapters handle the op kind; weixin/testchan have no group publisher and never produce the op. - Abort is a no-op on the receiving replica (callbacks can't cross processes); publish already happens after the turn ends.
STELLA_CHANNEL_DURABLE_INGRESS and the process-local ingress path are gone: adapters always write channel_inbox, routing/execution/outbox always run through PostgreSQL, and the legacy DM tail (queuedChat, handleResolvedIncoming, local command handlers) is deleted along with its now-dead helpers. Group publish tests are ported to durable outbox semantics (published-but-unfinalized rows stay eligible for finalization retry, and a zero-row ownership CAS now fails loudly). Role pinning for multi-replica testbeds moves to the composition root: STELLA_RUN_WORKER=off / STELLA_CHANNEL_LEASE=off exclude a replica from run execution and channel ownership while keeping fencing. Adds OTel backlog gauges (stella.channel.inbox.pending, stella.agent.run.open, stella.channel.outbox.pending, stella.channel.outbox.unknown) and updates EN/ZH deployment docs to describe the unconditional durable pipeline and its storage/network requirements.
…rns finish inside the drain budget
…ease takeover requires the previous writer proven dead
…sends bypass the lease fence
…recorded turn events replay through adapter draft/edit paths
… coverage and rollback doc
… by recovery and cancel queries
vaayne
changed the title
…mmit read-back (D4) The rollback test now performs the real outbox Append inside the finish transaction and then fails, proving already-written history AND outbox rows roll back together with the run state. Adds the uncertain-commit fault case: after a commit whose response was lost, a recovering worker reads the completed run back and never executes the turn twice.
- sessionexecution: TestFinishLostCommitAcknowledgmentCommitsExtra injects a commit that lands on the server but loses its ack, with the finish payload writing a real outbox row inside the tx — ErrOutcomeUnknown surfaces, read-back shows outcome/outbox/lease-removal all committed, and a retried Finish reports ErrLost without re-running the payload. - runtime: rename to TestWorkerCompletedRunIsNotReexecuted stating what it proves — a committed run is never reclaimed or re-executed; the commit seam is package-internal so the fault is injected at the lease layer. - fakeanthropic: TurnGate.Entered pins the exact point the gated model call parks, so tests kill a replica mid-turn instead of mid-claim. - system SameEventHandoff: waits on gate.Entered before killing A, and now counts total sends (==1), the run's sent outbox receipt, and single user/assistant history rows — duplicate sends or history can no longer pass green.
The durable run worker adopted its lease context under gctx, but errgroup cancels gctx the moment g.Wait returns — which happens mid-drain once Serve and the group-dispatch loop exit — so a claimed turn died before the accepted-work wait ever ran, committing 'canceled' instead of completing. - RunDurableLoops parents claimed turns on workCtx (cancelled only at the drain's final cancelWork step) and returns the worker's WaitInFlight so the drain's accepted-work wait covers claim→execute→finish-commit, not just the runtime's stream-close boundary. - TestDurableChannelGracefulDrain: SIGTERM the worker replica with the run parked in the gated model call; the turn finishes inside the budget, the run completes with its reply op, and the channel owner sends it. - Move sqlc.New(tx) behind memory.NewTxSessionTurnAppender — cmd/stellad's architecture boundary forbids raw sqlc refs (empty allowlist). - lcm execution test marks the previous writer's owner_pid dead; the D9 liveness fence otherwise (correctly) refuses takeover.
A claim commits its run row before ProcessOnce could raise any in-flight marker, so a drain waiting on the counter could observe zero while a committed claim was still pre-execute — teardown then cancelled the turn and finish committed 'canceled'. And waiting on the counter alone never stops the claim entry point, so a new claim could still land after the wait returned. WaitInFlight now joins Run: the loop drives ProcessOnce synchronously, so Run's exit is the authoritative point where every claimed run's finish has committed and no further claim can start. TestWorkerDrainJoinsClaimedRun ThroughFinish pins the interleaving — claim committed, executor gated, loop ctx cancelled — and asserts the drain holds until finish commits 'completed'.
…very - draft_update outbox ops: owner tails committed session events into a coalesced snapshot per run, sent through one stable platform message - DraftSender capability for telegram/discord/feishu/testchan; adapters without it simply drop drafts and keep terminal-only delivery - same-run draft barrier holds terminal ops behind in-flight draft edits; seq fencing cancels stale drafts so an old edit never overwrites the reply - C-replica attachment delivery: FileEvent bytes flow through outbox to the channel owner, verified end-to-end against the fake platform - Playwright spec 9-multireplica: sibling replica streams the gated turn, cursor resume after reconnect, reload keeps the turn, sibling stop cancels - drain: worker inflight wait now joins the Run loop so a claim committed just before Wait cannot slip past teardown
…er-call ops Two cross-replica delivery correctness fixes: - Draft dependency convergence: a draft_update successor whose predecessor ends failed/canceled/unknown can never pass the sent-only dependency gate, permanently blocking both itself and the same-run terminal barrier. The gate now treats resolved terminal states as satisfied for draft ops, and a CancelBlockedChannelOutbox sweep in claimDue cancels successors stuck behind permanently broken predecessors. Claim-time seq fences and run liveness checks keep stale edits from overwriting the terminal reply. - Per-external-operation receipts: send_reply previously issued multiple platform calls (text chunks, images, files) under one outbox row with a single receipt, so a retryable failure on a later segment resent earlier successful segments. ReplyChain now decomposes replies into one row per external call — primary send_reply, sibling send_text for overflow, one send_attachment per media — each with its own receipt. Adapters narrow send_reply to a single primary action; weixin multi-call attachment sends re-check channel ownership via OutboundOp.Guard before each SDK call, and QQ gains DraftSender via the Stream API so it no longer degrades to terminal-only live progress.
…ends A draft op that went 'unknown' may still have its edit in flight (stalled owner resuming past the attempt deadline, delayed platform apply), so the message it targeted is polluted: LatestSentDraftMessageID now abandons that identity and every later draft plus the terminal reply take a fresh message instead of editing a contaminated one. send_group_reply and notify decompose into one durable op per platform call (GroupReplyChain/NotifyChain, media folded to markers where the platform has no binary upload), pollPublishOutcomes treats a canceled link as terminal, and every adapter multi-call path (markdown to plain fallback, upload then send, card overflow, DM-open then send) re-checks the lease guard between SDK calls. Adapter classifiers now pass a pre-classified SendError through so a guard refusal stays retryable.
…d reactions enqueueAccepted passed the logical channel id as the outbox account key, so every group reply op failed OwnsAccount (account_mismatch) in production — the chain never reached the platform. Persist the receiving bot account on ctx_group_message.source_account_key at ingest and thread it through enqueueAccepted into every op of the reply chain. A channel re-bound to another platform account now rejects the old work instead of sending under the new identity; credential rotation under the same identity still owns it. Ops whose trigger had no receiving account (agent/system-origin, notifications) carry an empty key and stay unfenced. sendGroupReplyOp reactions are SDK calls too: the ack and terminal reactions now run under the op's ownership guard so a fenced-out replica stops touching the platform.
…ccount The previous fix snapshotted the trigger's observing account into reply ops, but a shared group trigger can wake members replying through a different channel and bot — OwnsAccount would reject legitimate replies. The channel lease owner's adapter now registers its platform account on channel.runtime_account_key (lease-token-fenced, never cleared), and enqueueAccepted freezes that key into the op chain at dispatch-accept. The trigger's source_account_key stays as audit data only.
GracefulDrain counted draft_update rows as sends; PausedWorkerFencing and WebCancel could pause or cancel before the worker request reached the gated fake model, leaving scripted responses unconsumed. Count only terminal ops and wait for gate.Entered before freezing or canceling.
SameEventHandoff and AttachmentDelivery park the fake model gate while waiting for replica C to claim A's lease — a 30s TTL plus the 15s sweep races the fake's 30s release backstop on loaded CI. Expire the lease row directly; the claim path under test is identical.
…a durable pipeline PostgreSQL is the source of truth for inboxes, runs, execution leases, outbox operations, and replayable session events, so replicas may share work across process boundaries: one accepts an event, another executes the run, another owns the channel lease and delivers the reply. - finish transaction commits the run's terminal state, final history, and prepared reply ops together; intermediate assistant/tool rows persist as produced - group acceptance commits message, memory, reply ops, and attachment bytes in one transaction; outgoing file bytes live in channel_outbox_attachment and senders never reopen workspace paths - session observation tails ctx_session_event exclusively (hub only wakes readers); GET resume pins an exact scope via Last-Event-ID with a fixed history boundary B, while POST registers scope without triggering a capped reload - removes in-process publishers, the deferred-turn store, and per-platform replay publishers
The 11 sequential migrations introduced by the durable-pipeline commit have not merged to main, so collapse them into a single 90000000000050 while the branch is still private. Columns added by later files move into their table's CREATE TABLE, the event_seq backfill disappears (the event log cannot have rows inside the same migration), and generated sqlc structs only change field order — the resulting schema is identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Channel ingress, agent execution, outbound delivery, and cross-replica observation are now durably coordinated in PostgreSQL: a message received on one replica can be routed, executed, answered, and observed by others, exactly once per event. The legacy in-process ingress path and its feature flag are removed — durable is the only path.
Why
Every stage used to be process-local: the receiving replica owned the binding, the turn, the live event hub, and the platform connection. A crash mid-turn lost the message; redelivery elsewhere could double-execute or double-send. Running more than one replica — or restarting during a long turn — was unsafe.
How
channel_inboxrows persist event identity/kind/version and payload before ack; dedup keys on stable event identity, not body hashes.agent_run+ctx_session_executionclaim transactions; actor/input/reply-address are versioned persisted value objects re-authorized on the claiming replica. Session takeover is writer-liveness-fenced: an expired lease whose recorded owner process is still alive — or unverifiable — cannot be taken over (resource-unavailable until resolved); a fenced writer cleans up its own expired row; the reaper uses a LEFT JOIN so orphanedrunningruns are still interrupted.FinishExtra).channel_outboxops are claimed, sent, and receipted only by the lease holder; send admission re-checks token + lease +enabledbefore every op, and a claim-timeGuardre-validates it inside multi-call ops before each SDK call after the first (markdown→plain fallback, upload→send, card overflow, DM-open→send). Failures classify retryable/permanent/unknown — unknown is never auto-resent.AccountCheckerfails ops enqueued under a swapped platform account — for group replies the fence key is the responding channel's account snapshot: the lease owner's adapter registers its platform account onchannel.runtime_account_key(token-fenced write, never cleared),enqueueAcceptedfreezes that key into the whole op chain at dispatch-accept, and the trigger's observing account stays audit-only onctx_group_message.source_account_key— so in a shared group where bot-A observes and bot-B's channel answers, ops carry bot-B, a re-bound account is rejected, and credential rotation under the same identity still owns the work; ops whose reply channel has no registered account (agent/system-origin, notifications) stay unfenced. Group replies and notifications get the same contract:GroupReplyChain/NotifyChainbuild one durable op per platform call,pollPublishOutcomeswaits for the whole chain, and acanceledlink is a terminal dispatch failure.ctx_session_eventfor its runs and folds committed events intodraft_updateops (delivery_key = live:<run-id>). Pending drafts coalesce in place, sent ones order behind the in-flight edit, andLatestSentDraftMessageIDlets a new owner recover the platform message after handoff. A seq fence cancels stale drafts and a same-run barrier holds the terminal op behind in-flight draft edits, so an old snapshot can never overwrite the final reply. Draft dependencies converge on terminal predecessor states (failed/canceled/unknown release the successor for liveness) while safety comes from identity, not timing: anunknowndraft may still have its edit in flight, so it pollutes the message it targeted —LatestSentDraftMessageIDreturns empty and later drafts plus the terminal reply take a fresh message. The abandoned preview may linger or take the late zombie edit; the final reply is a separate message it can never overwrite. Adapters implementchannel.DraftSender(TelegramBot.Edit, Discord draft/edit + component cleanup, Feishu card patch, QQ Stream API chunks, testchanmessage_idedits); adapters without it drop drafts and send only the terminal reply.ReplyChainbuilds a primarysend_reply(text, or a final edit on the live draft), siblingsend_textops for overflow segments, and onesend_attachmentop per image/file, chained byDependsOnso each carries its own receipt and retries independently. This fixes the per-receipt contract: previously onesend_replyissued every chunk and attachment under a single row, so a retryable failure on a later segment resent earlier successful segments. Adapters narrowsend_replyto one primary action; multi-call paths (weixin upload→CDN→send, feishu card overflow) re-check channel ownership viaOutboundOp.Guardbefore each additional SDK call. Attachment bytes flow through the outbox to whichever replica owns the channel — verified end-to-end against the fake platform. The same decomposition now covers every external send: group replies (GroupReplyChain, group metadata on the primary op, media folded to markers where the platform lacks binary upload) and notifications (NotifyChain, one op per segment).ctx_session_eventis the cross-process event log with run-scoped cursors, truncation detection, and a polling fallback; web send is idempotent enqueue-then-observe; group replies/notifications ride the same outbox (send_group_reply/notifyops) and group cancel propagates viacancel_requested.g.Wait()returns mid-drain (Serve and the dispatch loop exit first) and cancels it, so gctx-parented work died before the budget wait. The drain's accepted-work wait joins the workerRunloop itself, so a claim committed just before the wait cannot slip past teardown.STELLA_DATABASE_URLand one POSIXSTELLA_HOME;STELLA_CHANNEL_LEASE/STELLA_RUN_WORKERpin testbed roles. Docs (EN+ZH) cover requirements, at-least-once platform semantics, and the constraint that all replicas run one release.Deliberate limits: no hot migration of goroutines/model connections; no exactly-once promise at platform edges (idempotent platforms dedupe via keys, others get
unknown); draft-capable adapters are Telegram/Discord/Feishu/QQ/testchan — Weixin/DingTalk deliver terminal replies only; group chat replies keep the group dispatcher's accept/HOLD/snapshot policy; real-platform credential verification was not possible — see Test.Test
mise run format— clean;mise run build— clean.mise run test— full suite green: all Go package tests, 344 vitest frontend tests, and the-tags systemsuite (661s) includingTestSystem/image_historyafter fixing a real regression it caught (durable web send now encodes[]ai.ContentBlockwith kind tags viaai.MarshalTransportBlocks; a plain marshal dropped image blocks silently).mise run test:e2e -- --grep-invert @model— 25 passed, 1 skipped (live registry search, network-gated), including the new9-multireplica.spec.ts: a sibling replica (STELLA_RUN_WORKER=off) streams the primary's gated turn with SSE cursor resume, keeps it across page reload, and a stop issued to the sibling cancels the primary's run.go test -tags system -run 'TestDurable'): single-process loop, restart-before-send, three-replica takeover with pinned roles, same-event A→B→C handoff, paused stale owner (SIGSTOP/SIGCONT), paused worker fencing, DB outage/recovery, shared-Home attachment (A upload, B/C read identical bytes, stranger denied), attachment byte delivery by a non-executing replica (TestDurableChannelAttachmentDelivery— fake platform receives base64 payload, compared byte-for-byte, no duplicate send), web send/cancel,TestDurableChannelLiveProgress(mid-gate draft create → same-message edit → terminal reply; stale draft op fenced),TestDurableWebObserveLive, andTestDurableChannelGracefulDrain(SIGTERM on the worker replica mid-gated-turn → turn commits inside the drain budget → owner replica sends).internal/channel,internal/channel/outbox(split-retry, unknown-no-resend, account-mismatch, disabled-channel, cross-account group replies driven through the real append→dispatch→accept→enqueue→ProcessDue path (TestGroupReplyOpsFenceOnReplyChannelAccount: observer bot-A, responder ch-B/bot-B;TestGroupReplyOpsRejectReboundAccount: rebind rejects the frozen bot-B ops, new replies snapshot bot-B2), stale-owner, draft coalescing/fencing, draft-dependency deadlock regression: predecessor failed/canceled/unknown → successor converges and the terminal reply delivers),internal/agent/run,internal/sessionexecution(takeover fencing, reaper, tombstones, commit-then-lost-ack injection),internal/agent/runtime(real history+outbox rollback, uncertain commit read-back),internal/notify,internal/platform/home,pkg/ai(transport codec round-trip).plugins/channels/telegram/send_op_test.go, real outbox + real embedded PG + fake Bot API): mid-chain segment 429 → retry resends only the failed segment (TestSendReplyChainResumesOnlyFailedSegment); attachment failure leaves earlier sends unrepeated (TestSendReplyPartialAttachmentFailure); ownership loss claimed mid-batch is refused before later SDK calls (TestOwnershipLossBlocksLaterCallsInBatch,TestOwnershipLossBlocksWholeBatch); a markdown 400 followed by mid-op lease loss never fires the plain-text fallback (TestMarkdownFallbackBlockedByOwnershipLoss, controlTestMarkdownFallbackSendsWhileOwned); a group reply chain delivers each op with its own receipt and retries only the failed segment (TestGroupReplyChainDeliversPerCall); a group reply op enqueued under a swapped-out account is refused before any API call and its dependents cancel (TestGroupReplyOpRejectsReboundAccount), while the same account identity still delivers after rotation (TestGroupReplyOpDeliversAfterCredentialRotation), and group reactions obey the ownership guard (TestGroupReplyReactionsRespectOwnership).internal/channel/outbox/dispatch_test.go):TestTerminalReplyAbandonsPollutedDraftMessageparks the platform apply of a draft edit behind a gate, lets the row gounknown, completes the run, releases the gate after terminal delivery, and asserts the late edit lands only on the abandoned preview while the final reply lives on a fresh message identity.TestNotifyChainSplitsAndChains(segment-per-op, dependency-chained) andTestGroupReplyChainFoldsMediaAsText(no-upload platforms fold media to markers).GOOS=windows GOARCH=amd64,GOOS=linux GOARCH=amd64,GOOS=linux GOARCH=arm64,GOOS=darwin GOARCH=amd64,GOOS=darwin GOARCH=arm64— all pass after fixingowner_windows.go(x/sys/windowsdoes not exportSTILL_ACTIVE; defined locally as 259).testchanadapter, not against real platforms.web/src/features/skills/SkillsPage.tsxanti-sloptypeof windowlint warning.Refs
Closes #1297
Checklist
mise run format && mise run build && mise run test.testchansystem adapter.