Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 18 additions & 7 deletions endpoints_rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -371,7 +371,7 @@ Each benchmark has a **reference implementation** published in the MLPerf Endpoi
- The **dataset** used for performance and accuracy runs (Hugging Face dataset ID or download URL, plus the canonical split and any preprocessing recipe).
- The **reference chat template** (Hugging Face chat-template string or the equivalent message-formatting spec). Submissions MUST use the reference chat template; alternative templates that produce different tokenized output are not permitted.
- The **reference server / sampling parameters**: temperature, top-k, top-p, repetition penalty, greedy-vs-stochastic decoding flag, max output tokens, stop sequences. These MUST be set per the benchmark definition; submissions MUST NOT modify them.
- The **approved drafter list** for the benchmark, if any — one or more approved draft models/heads, each with its ID, precision, algorithm, and default per-point configuration. Submitters select any approved drafter; the list is published and versioned per submission round alongside the model list (see [§3.2](#32-supported-models)). See [§2.9.4](#294-speculative-decoding) for eligibility, approval, and verification requirements.
- The **approved drafter list** for the benchmark, if any — one or more approved draft models/heads, each with its ID, precision, algorithm, and default per-point configuration. A native head shipped inside one of the benchmark's approved checkpoints is approved with that checkpoint and need not be listed separately ([§2.9.4](#294-speculative-decoding)). Submitters select any approved drafter; the list is published and versioned per submission round alongside the model list (see [§3.2](#32-supported-models)). See [§2.9.4](#294-speculative-decoding) for eligibility, approval, and verification requirements.
- The accuracy evaluation methodology and quality target.
- The endpoint API interface.
- **Fixed configuration:** Configuration parameters explicitly designated as fixed by the reference implementation MUST NOT be modified by submitters.
Expand Down Expand Up @@ -432,7 +432,9 @@ Memory footprint is a legitimate dimension of a serving configuration, and the a
> [!CAUTION]
> **`[TENTATIVE — Subject to change after 2026-10-12]`**
**v1.0 change.** Speculative decoding is now governed by a **curated approved-drafter-list model, per benchmark**, rather than a single fixed drafter designated by the benchmark definition. Speculative decoding is permitted for any benchmark for which the benchmark task force has approved one or more drafters, following the process below. Benchmarks with no approved drafter continue to disallow speculative decoding entirely.
**v1.0 change.** Speculative decoding is now governed by a **curated approved-drafter-list model, per benchmark**, rather than a single fixed drafter designated by the benchmark definition. Speculative decoding is permitted for any benchmark for which the benchmark task force has approved one or more drafters, following the process below, or whose approved checkpoint ships a native head (see *Native heads* below). Benchmarks with no approved drafter continue to disallow speculative decoding entirely.

List of approved drafter list is available [here](https://github.com/mlcommons/endpoints-submission-cli/blob/main/src/submission_checker/data/approved_drafters.yaml).

**Approved drafter list.** Each benchmark's set of eligible drafters (MTP head, EAGLE-style head, or analogous module) is a **curated, published list**, not a single fixed drafter:

Expand All @@ -441,14 +443,23 @@ Memory footprint is a legitimate dimension of a serving configuration, and the a
- A newly approved drafter may first be used in a submission where the review eligibility date for a submission is **at least two cohorts dates after** the cohort in which the drafter was approved. Approval is recorded against the cohort in which the updated list is published.
- The approved list is published in the reference repository, versioned per submission round, alongside the model list (see [§3.2](#32-supported-models)).

**Native heads.** A speculative-decoding head (for example an MTP, DSpark, or EAGLE-style head) that the model publisher ships inside a checkpoint approved for the benchmark is **approved with that checkpoint**, in the checkpoint's approval cohort, and needs no separate list entry. The reference repository's list of approved checkpoints notes which of them ship a native head — for Agentic Inference, see [Approved Checkpoints and Speculative-Decoding Heads](https://github.com/mlcommons/endpoints/blob/main/examples/10_Agentic_Inference/README.md#approved-checkpoints-and-speculative-decoding-heads).

The following are **not allowed** with a native head:

- **A head that is not the canonical checkpoint's own.** A head added, replaced, or retrained by anyone other than the canonical model's publisher — including in a third-party quantized checkpoint — or added after the approved revision, is not native and requires an entry on the approved list.
- **Modifying the head**, other than *PTQ on drafter weights* below.
- **Using it as a different kind of drafter** than its publisher documents (e.g., running an MTP head as an EAGLE-style head). Per-point configuration may vary as for any approved drafter.
- **A head the WG has excluded** for failing *Drafter eligibility* below.

**Drafter eligibility.** The following disqualify a drafter from the approved list:

- **Not open-weight.** The weights are not downloadable by anyone who agrees to the publisher's license terms — private hosting, or access requiring manual/discretionary approval, disqualifies. An automatic license click-through with the same terms and instant access for anyone (e.g., Meta's Llama license gate) does not disqualify.
- **Input-based optimization ([§2.2.1](#221-general-rules)).** Build-time incorporation of the benchmark performance dataset's content — deliberate fine-tuning or distillation on it. (Incidental pretraining-corpus overlap does not disqualify; a corpus-disclosure or release-date test is not required of the proposer.)
- **Trained for benchmark performance.** A drafter trained, fine-tuned, or distilled specifically to perform well on this benchmark — including against benchmark-like traffic that reproduces its task mix, prompt style, or length distribution — even where the benchmark dataset itself was never used. A drafter published for general use, and adopted for the benchmark because it happens to suit it, does not disqualify.
- **Undisclosed or QAT-style quantization.** Quantization-aware training, or post-training quantization without disclosure of the calibration set and methodology, per [§2.9.3](#293-model-weight-rules).

Submitters select any drafter from the benchmark's approved list. Using a drafter **not on the approved list** is not permitted.
Submitters select any drafter from the benchmark's approved list, or a native head of one of its approved checkpoints. Using any other drafter is not permitted.

A list entry identifies a drafter in one of two ways:

Expand All @@ -468,7 +479,7 @@ The following are **also disallowed** at run time:

**PTQ on drafter weights.** The drafter weights MAY be post-training quantized under the same conditions as the canonical model ([§2.9.3](#293-model-weight-rules)): calibration-only, using only the published calibration set, no gradient updates, disclosed in the submission YAML, and subject to the accuracy gate. The drafter MUST NOT be modified in any other training-side sense — see *Drafter eligibility* above.

**Leaving the drafter unused.** A submission is not required to load or use a drafter shipped with the canonical checkpoint; see [§2.9.3](#293-model-weight-rules). Where a benchmark has no approved drafter ([§2.9.1](#291-reference-implementation)), speculative decoding is not available for that benchmark at all — and a drafter shipped with the canonical checkpoint but absent from the approved list may not be used.
**Leaving the drafter unused.** A submission is not required to load or use a drafter shipped with the canonical checkpoint; see [§2.9.3](#293-model-weight-rules). Where a benchmark has no approved drafter — no list entry and no native head of an approved checkpoint ([§2.9.1](#291-reference-implementation)) — speculative decoding is not available for that benchmark at all.

**Model equivalence for drafters.** No drafter-specific equivalence standard applies. A speculative-decoding submission is model equivalent on the same two general tests as any other optimization:

Expand Down Expand Up @@ -669,7 +680,7 @@ The set of supported benchmark models is defined per submission round and mainta
> [!NOTE]
> The model list for each submission round is published in the MLPerf Endpoints reference repository at least 6 weeks before the submission round opens. New models may be proposed to the working group per the benchmark roadmap process defined in the MLPerf General Submission Rules §4.3.
**Approved drafter lists.** For benchmarks that support speculative decoding, the approved drafter list is published in the reference repository alongside the model list, versioned per submission round. The approval process, eligibility criteria, and the lead time required before a newly approved drafter may be used are defined in [§2.9.4](#294-speculative-decoding).
**Approved drafter lists.** For benchmarks that support speculative decoding, the approved drafter list is published in the reference repository alongside the model list, versioned per submission round. The approval process, eligibility criteria, and the lead time required before a newly approved drafter may be used are defined in [§2.9.4](#294-speculative-decoding). Native heads of the listed checkpoints are approved with them and need not appear on the list ([§2.9.4](#294-speculative-decoding)).

### 3.3 Weight Transformations

Expand Down Expand Up @@ -1679,8 +1690,8 @@ The compliance validator — run by the submitter before submission and by MLCom
| **Accuracy** | Accuracy results are present for all points required by §5.3 and satisfy the applicable single-turn or multi-turn gate in §4.3. | Reject submission. |
| **Seed-set validity** | For an initial submission, every point must record the same seed set, and that set must have been published for `target_cohort` or one of the three immediately preceding cohorts. For an amendment, every new or replacement point must match the original submission's bound seed set; the four-cohort adoption test is not reapplied using the amendment's later cohort. See [Submission Rules §4.6](endpoints_submission_rules.md#46-seed-rotation). | Reject submission. |
| **Configuration consistency** | Same model, endpoint configuration, software stack, and seed set across all measurement points. | Flag inconsistencies. |
| **Approved drafter** | For points using speculative decoding, the disclosed drafter matches an entry on the benchmark's published approved drafter list — by weight checksum, or by target checksum plus configuration for a configuration-identified entry ([§2.9.4](#294-speculative-decoding)/[§3.2](#32-supported-models)). | Reject non-conforming points. |
| **Drafter approval lead time** | The drafter used was approved at least two cohorts before the submission's `target_cohort` ([§2.9.4](#294-speculative-decoding)). | Reject non-conforming points. |
| **Approved drafter** | For points using speculative decoding, the disclosed drafter matches an entry on the benchmark's published approved drafter list — by weight checksum, or by target checksum plus configuration for a configuration-identified entry — or is a native head, identified by the model ID and checksum of an approved checkpoint that ships it ([§2.9.4](#294-speculative-decoding)/[§3.2](#32-supported-models)). | Reject non-conforming points. |
| **Drafter approval lead time** | The drafter used was approved at least two cohorts before the submission's `target_cohort`; a native head counts from its checkpoint's approval cohort ([§2.9.4](#294-speculative-decoding)). | Reject non-conforming points. |

### 9.2 Manual Review Focus Areas

Expand Down
Loading