Skip to content

Remove unstable gather-rounds checkpoint exchange - #6813

Open
deepujain wants to merge 1 commit into
NVIDIA:mainfrom
deepujain:fix/3738-remove-gather-rounds
Open

deepujain wants to merge 1 commit into
NVIDIA:mainfrom
deepujain:fix/3738-remove-gather-rounds

Conversation

@deepujain

@deepujain deepujain commented Aug 24, 2026 •

Copy link
Copy Markdown
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

The gather_rounds checkpoint exchange path has caused numerical instability in production workloads. This change removes that implementation and configuration choice while retaining the broadcast default and the gather_object debugging option.

The load-strategy documentation and FP8 test parameterization are updated so users can no longer select the unstable path. Unknown exchange algorithms continue to fail explicitly with NotImplementedError.

Issue tracking

Fixes #3738

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Validation

  • A focused CPU check previously verified that exchange_by_distribution(..., exchange_algo="gather_rounds") now raises NotImplementedError; the repaired diff preserves that removal unchanged.
  • Repository-wide search found no remaining gather_rounds references.
  • python3 -m py_compile and git diff --check passed for all changed files.
  • Independent exact-head review found no blocking correctness or compatibility defects.

The repaired commit is rebased on current main and signed. The distributed FP8 checkpoint test requires the repository's multi-GPU environment and remains a hosted CI gate.

@copy-pr-bot

copy-pr-bot Bot commented Aug 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@deepujain
deepujain force-pushed the fix/3738-remove-gather-rounds branch from 2639092 to 4259e7e Compare September 3, 2026 02:17
@deepujain
deepujain force-pushed the fix/3738-remove-gather-rounds branch from 4259e7e to 135eb49 Compare September 24, 2026 21:49
Signed-off-by: dejain <deepujain@gmail.com>

Signed-off-by: Deepak Jain <deepujain@gmail.com>
@deepujain
deepujain force-pushed the fix/3738-remove-gather-rounds branch from 135eb49 to 42e91e4 Compare September 29, 2026 23:26
@deepujain
deepujain marked this pull request as ready for review September 29, 2026 23:55
@deepujain
deepujain requested review from a team as code owners September 29, 2026 23:55
@svcnvidia-nemo-ci
svcnvidia-nemo-ci requested a review from a team September 29, 2026 23:55
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gather rounds causes corruption in checkpoint load

2 participants