Skip to content

Commit 23f4ffa

Browse files
authored
Merge pull request #29 from phoenixframework/cm-streams-ack
Optimize subcluster replication
2 parents b5f0453 + 312d275 commit 23f4ffa

32 files changed

Lines changed: 3813 additions & 370 deletions

‎README.md‎

Lines changed: 30 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -313,8 +313,9 @@ All operations are **eventually consistent**:
313313
sent alone. Receivers stage provisional chunks in shard-owned private ETS and
314314
replace visible state only after the complete slice and its terminal manifest
315315
are present.
316-
- **`replicated_anti_entropy_interval`** — interval in milliseconds for stream
317-
head advertisements and nonblocking control heartbeats. Defaults to 1,000.
316+
- **`replicated_anti_entropy_interval`** — interval in milliseconds for retrying
317+
unacknowledged stream heads and sending nonblocking control heartbeats.
318+
Acknowledged streams do not advertise unchanged heads. Defaults to 1,000.
318319
- **`replicated_peer_lease_timeout`** — time without a dist-Erlang control
319320
heartbeat before state owned by that Group peer is purged. Defaults to 15,000
320321
and must exceed the anti-entropy interval. Probes continue after expiry so a
@@ -409,16 +410,25 @@ When Group starts (or a new Erlang node connects), shards exchange
409410
nodes. This handshake:
410411

411412
1. Validates that shard counts match (raises on mismatch).
412-
2. Exchanges cluster membership lists.
413-
3. Shard 0 exchanges protocol version, origin generation, and one complete
414-
active named-cluster epoch snapshot per node. Matching data shards exchange
413+
2. Uses constant-size discovery probes, regardless of the cluster count.
414+
3. Shard 0 exchanges protocol version, origin generation, and a complete
415+
active named-cluster epoch snapshot on discovery or authority repair.
416+
Matching data shards exchange
415417
only constant-size lane/transport descriptors tied to that authority
416418
revision.
417419

418420
Constant-size heartbeats renew the peer lease. If an origin generation or
419421
cluster-epoch revision changes, the receiver requests a fresh authoritative
420422
hello; if heartbeats stop, lease expiry purges that origin's complete local
421423
view and discovery probes allow it to rejoin later.
424+
Each lease expiry advances a discovery probe epoch, so the sender can replay
425+
previously acknowledged heads when the receiver has lost its view. Retries of
426+
that same probe epoch leave acknowledged streams quiet. An authority repair
427+
request carries one token per recovery attempt: its first reply is immediate,
428+
while repeated replies for the same token and unchanged authority are limited
429+
to one every five seconds. A new authority revision is sent immediately. A
430+
heartbeat that first restores a lane route also advertises the heads of writes
431+
made while that route was absent.
422432

423433
Incremental cluster open/close controls are generation fenced, processed in
424434
revision order, and installed by shard 0 into one node-wide authority table.
@@ -481,10 +491,21 @@ ETS view and batched into one delta message per target. Process-death registry
481491
and PG removals can share one record and retain their one-event-batch behavior.
482492

483493
Receivers advance a cursor only across a contiguous sequence prefix. A gap
484-
requests the missing suffix. Repeated head advertisements recover a dropped
485-
tail even when no later write occurs. If the requested sequence is older than
486-
the bounded oplog floor, the origin sends an exact snapshot of only its own
487-
registry claims and PG memberships; absence from that snapshot is a delete.
494+
requests the missing suffix. The sender retries only unacknowledged stream
495+
heads, so a dropped tail repairs even when no later write occurs; an applied
496+
cursor acknowledgement stops those retries. Discovery after a receiver lease
497+
expiry re-advertises the sender's current heads, including previously
498+
acknowledged streams whose receiver state may have been lost. If the requested
499+
sequence is older than the bounded oplog floor, the origin sends an exact
500+
snapshot of only its own registry claims and PG memberships; absence from that
501+
snapshot is a delete.
502+
Gap requests identify the advertised head. The sender ignores requests for an
503+
already acknowledged stream or an older head, then advertises its current head
504+
on the next retry. After a complete snapshot send, repeated gap requests for
505+
the same head wait up to five seconds before retrying the full snapshot. The
506+
wait is at most half the peer lease so provisional chunks can survive between
507+
attempts; a fresh peer recovery probe clears the hold. An interrupted chunk or
508+
commit resumes from its recorded offset.
488509

489510
There are no leaders, quorum acknowledgements, per-entry replicated tombstones,
490511
or known-membership retention barriers. Oplog memory is bounded locally and

0 commit comments

Comments
 (0)