Conversation
Signed-off-by: Chen Cui <chcui@nvidia.com>
Contributor
|
Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing: Add |
Contributor
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a public Qwen3.5-35B-A3B text-only 128K SFT recipe for 16 GB200 GPUs. This migrates the validated MXFP8 launcher settings into the recipe: TP1/CP8/EP16, one MTP layer, packed CoderForge data, FP8 parameter gather/gradient-buffer reuse, CuTeDSL grouped MLP, single grouped weights, and HybridEP. BF16 is deferred to a separate change.
Includes recipe exports, pinned model and dataset revisions, focused construction/discovery/boundary tests, and a Qwen3.5 text verification card. Run evidence lives in the card; this PR makes no changes to
docs/recipe-usage.mdorexamples/models/qwen/qwen35_text/README.md.Runtime prerequisite: packed MTP correctness requires NVIDIA/Megatron-LM#7611, tested at
b5581a9e874fc5be2b91e74cdd3ccdc64a1b5204. It is still open and is not in the current MCore pin. This PR does not update dependencies and should not be treated as independently runtime-ready on the bundled pin.Validation: both the reference launcher and migrated Bridge entry point completed 100 optimizer updates on NeMo 26.10.rc3, using the same checkpoint, 16-GB200 topology, prebuilt 128K packs and scheduler. All 100 LM/MTP/global-router/local-router loss comparisons passed
abs(delta) <= 1e-6 + 0.01 * abs(reference), with finite metrics, zero skipped/NaN updates, and evaluations at steps 50/100.W&B: reference, Bridge.
The historical training evidence used Bridge base
00ca266c4abc5f8435821823a1dbe05d0c3d6033plus an uncommitted recipe precursor, frozen as source bundle SHA2565ea6d3c0c82a47f06f9f031040c2cc78641a4334d961542e25338f9c585ed863. It does not constitute a training run on this PR head. Its validated execution overrides are now recipe defaults; an all-field configuration comparison checks this equivalence on the new main base. The data was the reference's seeded 99:1 split, first 20% of training trajectories, and all validation trajectories; the recipe builder's default 5% split is not covered by those runs. The 200-step warmup/300000-step LR horizon was preserved, so this is bounded stability/parity evidence. Checkpoint resume and export were not exercised.The card records historical results in
expected_result; clean-checkout verification remains unverified. Neither a clean commit nor a public image identifier exactly describes that historical run. The shared validator now permits null environment provenance only when no item is verified, supporting draft and historical cards for any model without fabricated identities. Verified items still require a public container and immutable commit; regression tests cover model-level, hardware, FSDP variant, and weak-scaling leaves.Validation: 14 focused recipe tests previously passed in rc3, with all-field configuration equivalence. This evidence-recording update passes card validation, 108 focused validator tests, full pre-commit, and independent review.
CI follow-up: refreshed the generated catalog and guide mapping for the text card, and updated offline recipe fixtures for pinned Qwen3.5 configs. All 225 relevant recipe tests and 21 catalog tests pass locally, along with full pre-commit and independent review. The training recipe is unchanged by this fix.