Describe the bug
TestDynamicInferenceNVLS::test_batch_invariant_prefill_matches_full_forward fails its exact logits comparison between dynamic prefill and the full-sequence forward. The recorded rank-0 pytest summary is 1 failed, 130 passed.
This occurred in PR #7725, run 37701357027, H100 hybrid-MoE unit job 113078675475.
Steps/Code to reproduce bug
Use the tested CI checkout 722eb084a072fd5b07118ab6b4dfe926d10f5a9d, the synthetic merge of PR head a7a6534eec8c8b0968810b7d307c198e749a333d into main 54f18cd2195a0fc22c145268c9c1c193e9a3026e, with the existing CI data mounts and GPU development environment. The original job ran eight H100 workers.
Run the existing test with the same installed CI environment and eight GPU workers:
uv run --no-sync python -m torch.distributed.run \
--standalone --nproc-per-node=8 -m pytest -vv -s \
tests/unit_tests/inference/test_hybrid_moe.py::TestDynamicInferenceNVLS::test_batch_invariant_prefill_matches_full_forward
The test seeds the inference RNG, enables batch-invariant mode, uses the NVLS dispatcher and Torch grouped GEMM, and compares the same 64-token input with and without the dynamic inference context. CUDA graphs are disabled for this comparison. It clears the inherited NVTE environment and explicitly selects FlashAttention 4 when available, otherwise 3; an inherited FA4-disable environment therefore does not constrain this explicit test configuration.
The original failure is:
torch.testing.assert_close(inference_logits, full_logits, atol=0, rtol=0)
AssertionError: Tensor-likes are not equal!
Mismatched elements: 1232 / 8192 (15.0%)
Greatest absolute difference: 0.00390625 at index (0, 9, 109)
Greatest relative difference: 2.171875 at index (0, 18, 82)
Expected behavior
Dynamic prefill and the full-sequence forward should produce identical logits under this test's batch-invariant configuration, as required by its existing atol=0, rtol=0 assertion.
Additional context
-
The runtime banner identifies NVIDIA PyTorch 26.09, PyTorch 2.14.0a0+b2c75dd; the pulled image digest is sha256:b542beea582085c4738ee3e68262751a4c8a1d462255761b7d4299c52a872c14.
-
Source: test configuration and assertion.
-
This is a logits-equality failure, not proof of changed generated tokens or degraded model quality. No matched baseline or isolated backend control is claimed here.
-
The model's layer pattern is ME* (Mamba/MoE). The explicit FlashAttention configuration alone does not establish that an attention kernel caused this mismatch. The responsible operation has not been isolated.
-
Related broader parity work is tracked in #7831. This report records the concrete CI failure; it does not establish the BF16 decode-state cause from #7856 or the GDP CUDA-graph token mismatch from #7845.
-
Proposed temporary containment is a dev-only skip of this one existing test, preserving its body and equality tolerance and leaving the other 130 cases selected. Re-enable it after the underlying parity failure is resolved and validated.
-
This report is left unassigned; no owner or team mentions are included.
Describe the bug
TestDynamicInferenceNVLS::test_batch_invariant_prefill_matches_full_forwardfails its exact logits comparison between dynamic prefill and the full-sequence forward. The recorded rank-0 pytest summary is 1 failed, 130 passed.This occurred in PR #7725, run 37701357027, H100 hybrid-MoE unit job 113078675475.
Steps/Code to reproduce bug
Use the tested CI checkout
722eb084a072fd5b07118ab6b4dfe926d10f5a9d, the synthetic merge of PR heada7a6534eec8c8b0968810b7d307c198e749a333dinto main54f18cd2195a0fc22c145268c9c1c193e9a3026e, with the existing CI data mounts and GPU development environment. The original job ran eight H100 workers.Run the existing test with the same installed CI environment and eight GPU workers:
The test seeds the inference RNG, enables batch-invariant mode, uses the NVLS dispatcher and Torch grouped GEMM, and compares the same 64-token input with and without the dynamic inference context. CUDA graphs are disabled for this comparison. It clears the inherited NVTE environment and explicitly selects FlashAttention 4 when available, otherwise 3; an inherited FA4-disable environment therefore does not constrain this explicit test configuration.
The original failure is:
Expected behavior
Dynamic prefill and the full-sequence forward should produce identical logits under this test's batch-invariant configuration, as required by its existing
atol=0, rtol=0assertion.Additional context
The runtime banner identifies NVIDIA PyTorch 26.09, PyTorch 2.14.0a0+b2c75dd; the pulled image digest is
sha256:b542beea582085c4738ee3e68262751a4c8a1d462255761b7d4299c52a872c14.Source: test configuration and assertion.
This is a logits-equality failure, not proof of changed generated tokens or degraded model quality. No matched baseline or isolated backend control is claimed here.
The model's layer pattern is
ME*(Mamba/MoE). The explicit FlashAttention configuration alone does not establish that an attention kernel caused this mismatch. The responsible operation has not been isolated.Related broader parity work is tracked in #7831. This report records the concrete CI failure; it does not establish the BF16 decode-state cause from #7856 or the GDP CUDA-graph token mismatch from #7845.
Proposed temporary containment is a dev-only skip of this one existing test, preserving its body and equality tolerance and leaving the other 130 cases selected. Re-enable it after the underlying parity failure is resolved and validated.
This report is left unassigned; no owner or team mentions are included.