Skip to content

[BugFix] Keep Gloo P2P operations from advancing collective tags - #79837

Open
cyanseek wants to merge 1 commit into
PaddlePaddle:developfrom
cyanseek:codex/gloo-p2p-tags
Open

cyanseek wants to merge 1 commit into
PaddlePaddle:developfrom
cyanseek:codex/gloo-p2p-tags

Conversation

@cyanseek

@cyanseek cyanseek commented Oct 2, 2026 •

Copy link
Copy Markdown

PR Category

Communication Library

PR Types

Bug fixes

Description

Fixes #79815.

Gloo send and recv advance the same per-group counter as collectives. When only some ranks exchange tensors, a later broadcast or all-reduce uses different tags across ranks and times out. Talking to multiple peers can also desynchronize the point-to-point operations themselves.

Use tag 0 for point-to-point operations. Gloo matches these messages in FIFO order per peer and direction, and Paddle already gives them a separate slot prefix from collectives. This keeps the collective counter synchronized without adding another counter that would depend on unrelated peers' traffic.

Add a four-process CPU regression covering multiple peers, reverse transfers, repeated exchanges with different payload lengths, and subsequent broadcast/all-reduce, plus a collective-only control.

Validation: the original four-rank sequence failed on the 3.3.1 CPU wheel and on the current develop process_group_gloo.cc compiled as a native module linked against that wheel. With this change, the sequence and both new tests pass. This is a targeted native build and real Gloo transport test, not a full Paddle rebuild. The test can be run on a source build with:

OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 python test/collective/test_process_group_gloo_p2p.py

The collective CMake generator cannot regenerate the whole subtree on this baseline because test/collective/fleet/testslist.csv references the missing test_fleet_static_mp_layers_deprecated.py. The added registration uses the existing generator's row parser and is formatted with the repository hook.

是否引起精度变化

否

@paddle-bot paddle-bot Bot added the contributor External developers label Oct 2, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor External developers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gloo: send/recv draw from the same per-group tag counter as collectives, so an intervening send/recv desyncs the next broadcast/all_reduce → TIMEOUT

2 participants