Add isolation manager for per-namespace replication lanes - #11304
Draft
robholland wants to merge 5 commits into
Draft
Add isolation manager for per-namespace replication lanes#11304robholland wants to merge 5 commits into
robholland wants to merge 5 commits into
Conversation
Every replication stream sender on a shard — one per remote cluster, with one iterator per priority lane — scans the same replication task queue, so each new task page was read from persistence once per scanner. Route those reads through a shard-scoped read-through buffer over the queue's tip: overlapping scans share one persistence read, and only readers below the buffered range (deep catch-up, lanes that have lagged out of coverage) fall through to persistence. The buffer holds slim queue rows (task metadata), not converted replication payloads — event payloads only enter the pipeline at send-time conversion, which reads history through its own cache. Coverage is a contiguous task-id interval established by persistence pages; rows are immutable and the queue is append-only, so eviction just shrinks coverage from the front and there is no invalidation. Reads are bounded by the shard's exclusive-high read watermark, so covered ranges are stable once established. Capacity is ReplicationStreamReadBufferSize tasks per shard (default 0 = disabled). This also removes the read-amplification objection to adding more concurrent lanes over the queue, which replication namespace isolation (later in this stack) relies on. Co-Authored-By: Claude <noreply@anthropic.com>
This was referenced Jul 27, 2026
Refactor the stream sender's per-priority QueueReaderState arithmetic (catch-up begin watermark lookup, reader-state construction from SyncReplicationState acks, failover watermark selection) into a replicationReaderGroup type behind a new EnableReplicationReaderGroup dynamic config flag (default off). Behavior is unchanged; scope-index mapping is centralized in priorityScopeIndex, which tolerates both the single-stack (1 scope) and tiered (3 scope) persisted formats. This is groundwork for replication stream namespace isolation, which extends the persisted reader state with additional per-lane scopes. Co-Authored-By: Claude <noreply@anthropic.com>
Introduce the wire-level building blocks for replication stream namespace isolation: - SyncReplicationState gains pause_high_namespace_ids (namespaces the receiver reports as overwhelming the HIGH lane; the priority lives in the field name so a future LOW extension adds its own field), isolated_lane_states (per-lane acked watermarks keyed by namespace ID), and supports_namespace_isolation (receiver capability advertisement, so a sender never emits lane-tagged traffic to a receiver that would misroute it). - WorkflowReplicationMessages gains isolated_namespace_id — when set, the batch belongs to that namespace's dedicated lane rather than the shared priority lane — and retire_isolated_lane, which marks the lane's final message so the receiver can drop its tracking once pending work drains (a retired lane can never pin the overall ack minimum). - The receiver routes lane-tagged batches to lazily-created per-namespace task trackers (each lane is its own monotonic stream for the life of the connection), folds member-lane watermarks into the overall ack minimum, includes member-lane backlogs in HIGH flow control, and reports per-lane watermarks keyed by namespace. - NamespaceThrottler (default: noop) observes per-namespace HIGH-priority task load on the receiver and decides which namespaces to report. The sender does not tag lanes yet, so this is inert until the sender-side isolation lands. Co-Authored-By: Claude <noreply@anthropic.com>
isolationManager owns the sender-side state of HIGH-lane namespace isolation. Each namespace the receiver reports as overwhelming the shared HIGH (live) lane is split onto its own dedicated lane — an independent read cursor and wire stream, paced by a throttled severity tier (a rate class). Because every member owns its cursor, members never interact: joining, demoting, or graduating one namespace moves no other namespace's position, no cursor ever rewinds, and demotion is purely a rate-class change that moves no data. Lane transitions anchor on applied (acked) watermarks so ordered HIGH events never straddle a hand-off with a gap: a split floors the member at the shared lane's acked watermark, and graduation is gated on the member's lane having applied up to the shared cursor. Graduated members are queued for the send loop to emit a lane-retirement marker so the receiver drops the lane's tracking. Member scopes are queues.Scope values, and the persisted reader-state codec round-trips through queues.ToPersistenceScope/FromPersistenceScope (scope 0 = overall min, 1 = shared HIGH excluding isolated namespaces, 2 = LOW, 3+ = one scope per member resuming at its applied watermark). The isolated set is capped by maxIsolated, bounding per-namespace state. State machine only; the stream sender is wired to it in the next PR. Co-Authored-By: Claude <noreply@anthropic.com>
robholland
force-pushed
the
claude/temporal-pr10147-stack-3-lane-manager
branch
from
July 27, 2026 13:35
8d3ccd2 to
1678867
Compare
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
isolationManager: the sender-side state machine for HIGH-lane namespace isolation. Each namespace the receiver reports as overwhelming the shared HIGH (live) lane is split onto its own dedicated lane — an independent read cursor and wire stream, paced by a throttled severity tier (a rate class). Because every member owns its cursor:Lane transitions still anchor on applied (acked) watermarks so ordered HIGH events never straddle a hand-off with a gap: a split floors the member at the shared lane's acked watermark, and graduation is gated on the member's lane having applied up to the shared cursor — and never fires while the shared cursor is still unknown (0), which would make the gate vacuous. Graduated members are queued for the send loop to emit a lane-retirement marker; a re-split cancels any not-yet-emitted marker (it would retire the NEW lane).
Lane generations: each lane incarnation carries a generation number, because a namespace can graduate and later re-split. Cursor advances are generation-checked, so a tier loop finishing a batch from before a graduation can never fast-forward the new incarnation's cursor past tasks it hasn't sent (the classic ABA hazard). Member acked watermarks are monotonic — a stale receiver report can't rewind a persisted resume point.
Member scopes reuse the multi-cursor queue framework (
queues.Scope,tasks.NamespacePredicate), and the persisted reader-state codec round-trips throughqueues.ToPersistenceScope/FromPersistenceScope: scope 0 = overall min, 1 = shared HIGH excluding isolated namespaces, 2 = LOW, 3+ = one scope per member resuming at its applied watermark. Scope 0 is clamped to the minimum over member resume positions: replication task cleanup deletes strictly belowScopes[0]and reads no other scope, while the receiver's folded watermark cannot vouch for windows its connection has never seen (e.g. right after a reconnect, before member lanes re-form) — without the clamp, cleanup could delete a restored member's unsent window. The isolated set is capped bymaxIsolated; restored members are kept above a lowered cap (dropping them would gap their lanes), severity resets to tier 1 on reconnect, and config inputs are clamped to sane minimums (tierCount ≥ 1 etc.) so misconfiguration cannot strand namespaces.State machine only; the stream sender is wired to it in the next PR.
Why?
This is the heart of the design, reviewable in isolation as a pure, unit-tested state machine. The per-member-lane shape is what the shared read-through buffer (first in the series) makes affordable — per-lane scans at the tip are in-memory passes — and in exchange the entire rewind/rotation protocol of the shared-cursor design disappears.
How did you test it?
Potential risks
No production code paths reference the manager yet (wired next PR), so this PR is behaviorally inert. Merge-back remains best-effort: a calm namespace stays isolated while its lane lags the gate — harmless while calm, reset by any reconnect. The scope-0 clamp pins replication task cleanup at the slowest member lane's resume position; that retention is bounded by the lane's tier pacing and is exactly the window cleanup must not delete.
🤖 Generated with Claude Code