test(functional): relax grpo_topp_topk gen_kl_error bound 0.03 -> 0.032#3341
Merged
Conversation
The top-p/top-k GRPO functional test asserts max(train/gen_kl_error) < 0.03, but top-p/top-k truncated sampling produces a larger, noisier generation-vs- training logprob mismatch than greedy, so the metric sits right at the bound and flakes. Observed on a recent run: the check failed at 0.03014 (barely over 0.03), requiring a rerun (fast_L1_Functional_Tests_GRPO_3, run 30111867247). gen_kl_error across recent GRPO_3 runs ranges ~0.009-0.025 on passing runs with the flake at ~0.030. Bump to 0.032 to clear the observed flake while keeping the bound tight (regression-sensitive). Signed-off-by: Terry Kong <terryk@nvidia.com>
terrykong
changed the title
terrykong
force-pushed
the
terryk/relax-topp-topk-gen-kl-error
branch
from
July 24, 2026 21:21
a122c60 to
bb191ba
Compare
terrykong
added
the
CI:Lfast
Collaborator
Author
|
/ok to test bb191ba |
terrykong
enabled auto-merge (squash)
July 24, 2026 21:42
yfw
approved these changes
Jul 24, 2026
macandro96
pushed a commit
that referenced
this pull request
Jul 25, 2026
…32 (#3341) Signed-off-by: Terry Kong <terryk@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Relax the
grpo_topp_topkfunctional-test convergence bound:Why
tests/functional/grpo_topp_topk.shassertsmax(train/gen_kl_error) < 0.03.gen_kl_errormeasures the mismatch between the logprobs vLLM used at generation time and the ones the training policy recomputes for the same tokens. Top-p/top-k truncated sampling produces a larger and noisier mismatch than greedy decoding, so the metric sits right at the0.03bound and flakes.Observed failure:
fast_L1_Functional_Tests_GRPO_3failed at 0.03014 — barely over — and needed a rerun:Data
Mining
gen_kl_errorfor this test across recentGRPO_3runs:0.03has essentially no margin over the passing distribution.0.032clears the observed flake (0.0301) while keeping the bound tight so it stays sensitive to a genuine regression. (The bound has been0.03since the test was added in #2053 and was never tuned against observed variance.)🤖 Generated with Claude Code