Skip to content

fix(recipe): enable VR in perf configs#4724

Merged
malay-nagda merged 1 commit into
NVIDIA-NeMo:r0.5.0from
kocchop:faysal/fix_vr_configs_r0.5.0
Jul 8, 2026
Merged

fix(recipe): enable VR in perf configs#4724
malay-nagda merged 1 commit into
NVIDIA-NeMo:r0.5.0from
kocchop:faysal/fix_vr_configs_r0.5.0

Conversation

@kocchop

@kocchop kocchop commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What does this PR do ?

Fixes few VR perf configs

Changelog

  • gpt_oss 120B / qwen3 235B & 30B: apply full-iteration CUDA-graph configs on VR200 (and mxfp8 fp8_output_proj for gpt_oss 120B)
  • perf_plugins/executors: add vr200 to the GB200/GB300 platform lists (NVL72 topology, cuDNN LayerNorm, sm_100+ overlap, NUMA divisor)

GitHub Actions CI

See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

- gpt_oss 120B / qwen3 235B & 30B: apply full-iteration CUDA-graph configs
  on VR200 (and mxfp8 fp8_output_proj for gpt_oss 120B)
- perf_plugins/executors: add vr200 to the GB200/GB300 platform lists
  (NVL72 topology, cuDNN LayerNorm, sm_100+ overlap, NUMA divisor)

Signed-off-by: Md Fahim Faysal Khan <mdfahimfaysa@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added area:perf Performance optimizations and benchmarking cherry-pick community-request feature New capabilities, enhancements, or enablement work needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer r0.5.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. labels Jul 8, 2026
@dingqingy-nv

Copy link
Copy Markdown
Contributor

/claude review

@claude

claude Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Light review — LGTM

Clean, consistent VR200 enablement. VR200 is treated the same as GB200/GB300 (NVL72 topology, 4 GPUs/node so numa_divisor=2, SM100 dispatcher, cuDNN-LN branch), matching the existing base-config aliases and argument_parser.py (vr200: 4). The new/updated vr200 config functions mirror their gb200/gb300 siblings, including the fp8_mx guard on set_full_iter_cg_configs and the mxfp8 fp8_output_proj toggle for gpt_oss_120b.

Observations (non-blocking):

  • utils/executors.py:110 still sets NCCL_NET_GDR_LEVEL / NCCL_NET_GDR_C2C only for gpu == gb200. If VR200 wants the same GDR tuning it was not extended here, likely intentional (GB200-specific NCCL workaround), just confirm.
  • No new unit/functional coverage was added for the vr200 config paths. test_perf_config_integration.py only instantiates the _h100 variants; a lightweight instantiation assertion for the vr200 functions would guard against a missing base-config alias silently breaking get_workload_base_config(gpu=vr200).

Suggested test cases (perf configs touched: qwen3_235b_a22b vr200, qwen3_30b_a3b vr200, gpt_oss_120b vr200):

  • qwen3_235b_a22b vr200 bf16 (QWEN3_235B_A22B_PRETRAIN_CONFIG_VR200_BF16_V2)
  • qwen3_235b_a22b vr200 fp8_mx (QWEN3_235B_A22B_PRETRAIN_CONFIG_VR200_FP8_MX_V2)
  • qwen3_235b_a22b vr200 nvfp4 (QWEN3_235B_A22B_PRETRAIN_CONFIG_VR200_NVFP4_V2)
  • qwen3_30b_a3b vr200 bf16 (QWEN3_30B_A3B_PRETRAIN_CONFIG_VR200_BF16_V1)
  • qwen3_30b_a3b vr200 fp8_mx (QWEN3_30B_A3B_PRETRAIN_CONFIG_VR200_FP8_MX_V1)
  • gpt_oss_120b vr200 bf16 (GPT_OSS_120B_PRETRAIN_CONFIG_VR200_BF16_V2)
  • gpt_oss_120b vr200 fp8_mx (GPT_OSS_120B_PRETRAIN_CONFIG_VR200_FP8_MX_V2)

Each should instantiate the corresponding pretrain_config_vr200(precision=...) function and assert it builds without error (and that the fp8_mx path sets the full-iter CG / fp8_output_proj fields).

@malay-nagda malay-nagda added the docs-only With great power comes great responsibility. label Jul 8, 2026
@malay-nagda

Copy link
Copy Markdown
Contributor

/ok to test 6a5ba94

@malay-nagda malay-nagda enabled auto-merge (squash) July 8, 2026 06:35
@malay-nagda malay-nagda merged commit a3b4858 into NVIDIA-NeMo:r0.5.0 Jul 8, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:perf Performance optimizations and benchmarking cherry-pick docs-only With great power comes great responsibility. feature New capabilities, enhancements, or enablement work needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer r0.5.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants