Repository navigation
Add the legacy DeepSeek-R1 and GPT-OSS-120B drafters - #116
Merged
arav-agarwal2 merged 1 commit intoOct 9, 2026
Merged
Conversation
Carry three weight-identified drafters over from legacy submissions: - deepseek-r1: the native MTP head (layer 61) in deepseek-ai/DeepSeek-R1 @ 56d4cbbb and centml/DeepSeek-R1-NVFP4-v2-mlpinf @ 93947a0d. The NVFP4 checkpoint is post-training quantized on the MLPerf Inference deepseek-r1 calibration set and keeps the MTP layer. - gpt-oss-120b: nvidia/gpt-oss-120b-Eagle3-long-context @ 633caf45, the repository nvidia/gpt-oss-120b-Eagle3 redirects to. The head is unquantized. They are recorded as task-force seed entries (§2.9.4) with the other initial approvals in 2026-09-C1, so 2026-10-C1 submissions can use them. The bundled-list tests now cover the new heads, and the README lists them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
arav-agarwal2
approved these changes
Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the speculative-decoding drafters carried over from legacy submissions for the two non-agentic benchmarks to
src/submission_checker/data/approved_drafters.yaml. Until now neither benchmark had an entry, soapproved-drafterrejected every DeepSeek-R1 or GPT-OSS-120B point that uses speculative decoding.model_idweight_checksumdeepseek-r1deepseek-ai/DeepSeek-R1git-sha1:56d4cbbb4d29f4355bab4b9a39ccb717a14ad5addeepseek-r1centml/DeepSeek-R1-NVFP4-v2-mlpinfgit-sha1:93947a0d7bd04f73ff98636f6b18ff5839e7aaf9gpt-oss-120bnvidia/gpt-oss-120b-Eagle3-long-contextgit-sha1:633caf45f31288cbb70ee237f7c939db707ecc94nvidia/gpt-oss-120b-Eagle3redirects to this repository. The head is unquantized (quant_algo: null).All three repositories are public and ungated on Hugging Face at the pinned revisions.
Approval cohort
The new entries are recorded as task-force seed entries (§2.9.4: "the benchmark task force seeds the list with ≥ 1 approved head/drafter per benchmark model"). They use
2026-09-C1, the same cohort as the initial approvals, so the two-cohort lead time makes them eligible from target cohort2026-10-C1.Changes
approved_drafters.yaml: three entries, plus header comments on where they come from and why they use this cohort.tests/submission_checker/test_checks_specdec_agentic.py:_REFERENCE_HEADSnow includes the new heads, so the bundled-list, cohort and lead-time tests cover them.README.md: the drafter section lists the new heads and their eligibility.Test plan
pytest tests/submission_checker/test_checks_specdec_agentic.py: 36 passedruff checkclean.ruff format --checkalready flags this test file onmain, on lines this PR does not touch.🤖 Generated with Claude Code