Skip to content

test(functional): add MoE performance recipe proxies#4726

Closed
yaoyu-33 wants to merge 2 commits into
NVIDIA-NeMo:malay/no_load_balance_routerfrom
yaoyu-33:ci/qwen3-moe-functional-proxy
Closed

test(functional): add MoE performance recipe proxies#4726
yaoyu-33 wants to merge 2 commits into
NVIDIA-NeMo:malay/no_load_balance_routerfrom
yaoyu-33:ci/qwen3-moe-functional-proxy

Conversation

@yaoyu-33

@yaoyu-33 yaoyu-33 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds reduced, single-node functional proxies for the production Qwen3 30B-A3B, DeepSeek V3, and GPT-OSS 120B performance recipes. These three model families were implicated by the same downstream Blackwell MXFP8 grouped-MLP failure, but the existing Bridge functional suite did not exercise their production HybridEP and TE op-fuser paths.

This PR is intentionally stacked on #4679. The proxies therefore execute and guard the corrected natural-routing path. Retarget to main after #4679 merges.

Coverage

  • retain the existing 8-GPU H100 Qwen3 FP8 current-scaling proxy
  • run one 8-GPU GB200 L0 job with three isolated torchrun invocations:
    • Qwen3 30B-A3B from the exact 8-GPU MXFP8 constructor
    • DeepSeek V3 from the production 256-GPU MXFP8 constructor, reduced to four decoder layers, 16 experts, PP4, and EP2 while retaining MTP and its production-style pipeline layout
    • GPT-OSS 120B from the production 64-GPU MXFP8 constructor, reduced to two layers and EP8 while retaining the 120B width, 128 experts, top-4 routing, and alternating sliding attention
  • assert natural routing, HybridEP/flex dispatch, grouped GEMM, MXFP8, discrete expert weights, the TE op-fuser, overlap settings, and each recipe-specific CUDA-graph mode
  • use a null tokenizer with mock data so the tests do not require tokenizer or training-dataset downloads
  • mirror the relevant production executor environment, with DeepSeek-only layernorm settings scoped to its invocation

Why this matrix

A Qwen-only proxy guards the shared grouped-MLP operator path, but it cannot cover DeepSeek MTP/pipeline assembly or the GPT-OSS 120B provider and expert topology. Those differences were present in the downstream failures and are not represented by the current DeepSeek PP1/FSDP proxy or GPT-OSS 20B EP1 L1 test.

Other downstream failure classes were intentionally not converted into expensive GPU jobs: Llama packed-recipe construction is already enumerated by the dedicated recipe unit regression plus packed-LoRA functional coverage, while evaluator, performance-threshold, golden-data, and infrastructure failures belong in their respective test layers.

Validation

  • Ruff lint passes for the functional module
  • shell syntax checks pass for both launchers
  • pytest collection in the project container: 4 functional tests collected
  • targeted recipe unit tests: 32 passed
  • real DeepSeek V3 and GPT-OSS production constructors were instantiated in the project container with simulated GB200 capability, and all proxy topology/path assertions passed
  • the GB200 matrix uses one x8 runner job with a 60-minute timeout; the H100 Qwen proxy remains one x8 job with a 30-minute timeout

Full H100/GB200 runtime validation is delegated to this PR functional CI.

CI resource note

These tests are L0 because the objective is PR-time failure detection. They request eight GPUs, which is supported by the dynamic runner matrix but exceeds the general two-GPU guidance in CONTRIBUTING.md; automation/CI-maintainer approval is requested for this targeted exception.

@copy-pr-bot

copy-pr-bot Bot commented Jul 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 force-pushed the ci/qwen3-moe-functional-proxy branch from f5b7d87 to 377dcf2 Compare July 8, 2026 05:15
@yaoyu-33

yaoyu-33 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 377dcf2

@yaoyu-33 yaoyu-33 force-pushed the ci/qwen3-moe-functional-proxy branch from 377dcf2 to 41f84e8 Compare July 8, 2026 05:16
@yaoyu-33

yaoyu-33 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 41f84e8

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 force-pushed the ci/qwen3-moe-functional-proxy branch from 41f84e8 to 6c63214 Compare July 8, 2026 05:19
@yaoyu-33

yaoyu-33 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 6c63214

@yaoyu-33

yaoyu-33 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Implementation and independent review are complete. The latest CI run is waiting at the protected test environment before the H100 and GB200 functional jobs can start: https://github.com/NVIDIA-NeMo/Megatron-Bridge/actions/runs/28919465324\n\nThe GB200 proxy explicitly guards natural routing, the discrete-weight MXFP8 TE op-fuser path, and partial TE graph settings. CI-maintainer approval is requested for the targeted 8-GPU L0 exception described in the PR body.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 changed the title test(functional): add Qwen3 MoE performance proxies test(functional): add MoE performance recipe proxies Jul 8, 2026
@yaoyu-33

yaoyu-33 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test a46fa53

@yaoyu-33

yaoyu-33 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

The updated three-model matrix is ready on commit a46fa53. Lint and pre-flight passed; the current functional run is waiting for approval on the protected test environment: https://github.com/NVIDIA-NeMo/Megatron-Bridge/actions/runs/28920614465

This run contains one GB200 x8 L0 job for Qwen3, GPT-OSS 120B, and DeepSeek V3 plus the existing H100 Qwen proxy. CI-maintainer approval is requested for the targeted x8 jobs described in the PR body.

@yaoyu-33

yaoyu-33 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #4742. The replacement uses an upstream branch and splits Qwen3, DeepSeek V3, and GPT-OSS 120B into independent model-specific L0 modules and launchers.

@yaoyu-33 yaoyu-33 closed this Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant