Skip to content

Add isolation manager for per-namespace replication lanes - #11304

Draft
robholland wants to merge 5 commits into
temporalio:mainfrom
robholland:claude/temporal-pr10147-stack-3-lane-manager
Draft

Add isolation manager for per-namespace replication lanes#11304
robholland wants to merge 5 commits into
temporalio:mainfrom
robholland:claude/temporal-pr10147-stack-3-lane-manager

Conversation

@robholland

@robholland robholland commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Part 4 of a 5-PR series building to replication stream namespace isolation (a restructuring of #10147): read buffer → reader group → lane protocol → isolation manager → sender isolation.
Stacked on #11303 (lane wire protocol). This PR targets main, so until the earlier PRs in the series merge its diff also includes their commits — review after they land. · Next in series: #11305 (sender isolation).

What changed?

isolationManager: the sender-side state machine for HIGH-lane namespace isolation. Each namespace the receiver reports as overwhelming the shared HIGH (live) lane is split onto its own dedicated lane — an independent read cursor and wire stream, paced by a throttled severity tier (a rate class). Because every member owns its cursor:

  • members never interact — joining, demoting, or graduating one namespace moves no other namespace's position;
  • no cursor ever rewinds, so there is no CAS machinery, no rewind-point arithmetic, and no receiver-side tracker rotation (all of which the shared-tier-cursor design in Replication stream multi-cursor #10147 required);
  • demotion is purely a rate-class change that moves no data.

Lane transitions still anchor on applied (acked) watermarks so ordered HIGH events never straddle a hand-off with a gap: a split floors the member at the shared lane's acked watermark, and graduation is gated on the member's lane having applied up to the shared cursor — and never fires while the shared cursor is still unknown (0), which would make the gate vacuous. Graduated members are queued for the send loop to emit a lane-retirement marker; a re-split cancels any not-yet-emitted marker (it would retire the NEW lane).

Lane generations: each lane incarnation carries a generation number, because a namespace can graduate and later re-split. Cursor advances are generation-checked, so a tier loop finishing a batch from before a graduation can never fast-forward the new incarnation's cursor past tasks it hasn't sent (the classic ABA hazard). Member acked watermarks are monotonic — a stale receiver report can't rewind a persisted resume point.

Member scopes reuse the multi-cursor queue framework (queues.Scope, tasks.NamespacePredicate), and the persisted reader-state codec round-trips through queues.ToPersistenceScope/FromPersistenceScope: scope 0 = overall min, 1 = shared HIGH excluding isolated namespaces, 2 = LOW, 3+ = one scope per member resuming at its applied watermark. Scope 0 is clamped to the minimum over member resume positions: replication task cleanup deletes strictly below Scopes[0] and reads no other scope, while the receiver's folded watermark cannot vouch for windows its connection has never seen (e.g. right after a reconnect, before member lanes re-form) — without the clamp, cleanup could delete a restored member's unsent window. The isolated set is capped by maxIsolated; restored members are kept above a lowered cap (dropping them would gap their lanes), severity resets to tier 1 on reconnect, and config inputs are clamped to sane minimums (tierCount ≥ 1 etc.) so misconfiguration cannot strand namespaces.

State machine only; the stream sender is wired to it in the next PR.

Why?

This is the heart of the design, reviewable in isolation as a pure, unit-tested state machine. The per-member-lane shape is what the shared read-through buffer (first in the series) makes affordable — per-lane scans at the tip are in-memory passes — and in exchange the entire rewind/rotation protocol of the shared-cursor design disappears.

How did you test it?

  • built
  • added new unit test(s) — split floors and lane scopes, member independence, the isolation cap (including slot reuse after graduation), demotion as a rate-class change capped at the deepest tier, merge-back gating on applied watermarks with cooldown and re-throttle reset (including never graduating against an unset shared cursor), retirement queueing and re-split cancellation, generation-checked and monotonic cursor advances, monotonic acked watermarks, the scope-0 cleanup clamp, config clamping, and a persistence round-trip including legacy 3-scope states

Potential risks

No production code paths reference the manager yet (wired next PR), so this PR is behaviorally inert. Merge-back remains best-effort: a calm namespace stays isolated while its lane lags the gate — harmless while calm, reset by any reconnect. The scope-0 clamp pins replication task cleanup at the slowest member lane's resume position; that retention is bounded by the lane's tier pacing and is exactly the window cleanup must not delete.

🤖 Generated with Claude Code

robholland and others added 2 commits July 24, 2026 15:40
Every replication stream sender on a shard — one per remote cluster, with
one iterator per priority lane — scans the same replication task queue,
so each new task page was read from persistence once per scanner. Route
those reads through a shard-scoped read-through buffer over the queue's
tip: overlapping scans share one persistence read, and only readers below
the buffered range (deep catch-up, lanes that have lagged out of
coverage) fall through to persistence.

The buffer holds slim queue rows (task metadata), not converted
replication payloads — event payloads only enter the pipeline at
send-time conversion, which reads history through its own cache. Coverage
is a contiguous task-id interval established by persistence pages; rows
are immutable and the queue is append-only, so eviction just shrinks
coverage from the front and there is no invalidation. Reads are bounded
by the shard's exclusive-high read watermark, so covered ranges are
stable once established.

Capacity is ReplicationStreamReadBufferSize tasks per shard (default 0 =
disabled).

This also removes the read-amplification objection to adding more
concurrent lanes over the queue, which replication namespace isolation
(later in this stack) relies on.

Co-Authored-By: Claude <noreply@anthropic.com>
robholland and others added 3 commits July 27, 2026 14:16
Refactor the stream sender's per-priority QueueReaderState arithmetic
(catch-up begin watermark lookup, reader-state construction from
SyncReplicationState acks, failover watermark selection) into a
replicationReaderGroup type behind a new EnableReplicationReaderGroup
dynamic config flag (default off). Behavior is unchanged; scope-index
mapping is centralized in priorityScopeIndex, which tolerates both the
single-stack (1 scope) and tiered (3 scope) persisted formats.

This is groundwork for replication stream namespace isolation, which
extends the persisted reader state with additional per-lane scopes.

Co-Authored-By: Claude <noreply@anthropic.com>
Introduce the wire-level building blocks for replication stream namespace
isolation:

- SyncReplicationState gains pause_high_namespace_ids (namespaces the
  receiver reports as overwhelming the HIGH lane; the priority lives in
  the field name so a future LOW extension adds its own field),
  isolated_lane_states (per-lane acked watermarks keyed by namespace ID),
  and supports_namespace_isolation (receiver capability advertisement, so
  a sender never emits lane-tagged traffic to a receiver that would
  misroute it).
- WorkflowReplicationMessages gains isolated_namespace_id — when set, the
  batch belongs to that namespace's dedicated lane rather than the shared
  priority lane — and retire_isolated_lane, which marks the lane's final
  message so the receiver can drop its tracking once pending work drains
  (a retired lane can never pin the overall ack minimum).
- The receiver routes lane-tagged batches to lazily-created per-namespace
  task trackers (each lane is its own monotonic stream for the life of
  the connection), folds member-lane watermarks into the overall ack
  minimum, includes member-lane backlogs in HIGH flow control, and
  reports per-lane watermarks keyed by namespace.
- NamespaceThrottler (default: noop) observes per-namespace HIGH-priority
  task load on the receiver and decides which namespaces to report.

The sender does not tag lanes yet, so this is inert until the sender-side
isolation lands.

Co-Authored-By: Claude <noreply@anthropic.com>
isolationManager owns the sender-side state of HIGH-lane namespace
isolation. Each namespace the receiver reports as overwhelming the
shared HIGH (live) lane is split onto its own dedicated lane — an
independent read cursor and wire stream, paced by a throttled severity
tier (a rate class). Because every member owns its cursor, members never
interact: joining, demoting, or graduating one namespace moves no other
namespace's position, no cursor ever rewinds, and demotion is purely a
rate-class change that moves no data.

Lane transitions anchor on applied (acked) watermarks so ordered HIGH
events never straddle a hand-off with a gap: a split floors the member
at the shared lane's acked watermark, and graduation is gated on the
member's lane having applied up to the shared cursor. Graduated members
are queued for the send loop to emit a lane-retirement marker so the
receiver drops the lane's tracking.

Member scopes are queues.Scope values, and the persisted reader-state
codec round-trips through queues.ToPersistenceScope/FromPersistenceScope
(scope 0 = overall min, 1 = shared HIGH excluding isolated namespaces,
2 = LOW, 3+ = one scope per member resuming at its applied watermark).
The isolated set is capped by maxIsolated, bounding per-namespace state.

State machine only; the stream sender is wired to it in the next PR.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant