ring: make cross-machine TCP ring join robust to launch skew - #2
Draft
crypt0fairy wants to merge 1 commit into
Draft
ring: make cross-machine TCP ring join robust to launch skew#2crypt0fairy wants to merge 1 commit into
crypt0fairy wants to merge 1 commit into
Conversation
Darkbloom runs the MLX ring backend across independently-started Macs (separate processes, separate machines). Three problems made the join flaky/impossible in that setting: 1. ring.cpp: CONN_ATTEMPTS was 5, so the connect retry window was only ~5s. Two nodes launched by hand / over SSH with differing shard-load and attestation startup times routinely missed it and aborted with errno 60. Widened to 60 attempts. 2. utils.cpp TCPSocket::connect leaked the socket fd on every failed attempt, piling up SYN_SENT/CLOSED sockets that confused the peer's single accept(). Now closes the fd before retrying. 3. utils.cpp used unbounded exponential backoff (wait <<= 1), so after a few misses a node slept 16s/32s while its peer sat ready — they fell out of phase and never rendezvoused. Capped the wait at 2s so retries stay frequent and in-phase across the whole window. Verified on a 2-Mac cluster (32GB + 24GB) running sharded Mistral-24B-8bit end-to-end through the Darkbloom coordinator.
This was referenced Jun 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the TCP ring distributed backend so two nodes started independently
on separate machines reliably form the ring. The defaults assume ranks are
co-launched (all start within milliseconds); when each node is launched by hand
and spends seconds loading its model shard before reaching
accept(), theconnecting side can exhaust its retries and abort before its peer is listening.
Two files, +28/-3:
ring/ring.cpp—CONN_ATTEMPTS5 → 60. The 5-attempt window was too shortfor separately-started nodes; the connecting side aborted with
errno 60(ETIMEDOUT) before the peer reached
accept(). (No effect on co-launched setups —they connect on the first attempt.)
distributed/utils.cppTCPSocket::connect— two fixes for themany-retries-before-peer-ready case:
created each iteration; without closing, failed ones accumulate
(SYN_SENT/CLOSED) and we observed them interfere with the peer's
accept().node could be sleeping for many seconds while its peer became ready — the two
fall out of phase. Capping the wait keeps retries frequent and in-phase across
the
num_retrieswindow.Context
Found while bringing up the ring backend across two Apple-Silicon Macs
(independently launched, differing shard-load times). The fd-leak and
unbounded-backoff are general (not setup-specific) and apply to any
separately-started ring; the higher attempt count is the one tunable specific to
hand-launched nodes.
Testing
Verified end-to-end: two Macs reliably form the ring and serve a layer-sharded
model over it (pipeline-parallel inference), where the previous defaults
intermittently failed to connect.