Fix reference-client interactivity checks and bundle approved agentic drafters

README.md

@@ -454,11 +454,21 @@ not satisfy High Concurrency coverage.
 | `drafter-approval-lead-time` | §2.9.4 | Approved at least two cohorts before `target_cohort` |
 | `drafter-list-registry` | §2.9.4 | Warns when the drafter list itself cannot be read |
 
-The approved list ships as data (`src/submission_checker/data/approved_drafters.yaml`)
-and is **empty** — §2.9.4's list has not been published yet, and an empty registry means
-speculative decoding is not permitted for any benchmark, which is §2.9.4's own rule for a
-benchmark with no approved drafter. Point `--approved-drafters FILE` or
-`$MLPERF_ENDPOINTS_APPROVED_DRAFTERS` at a published list.
+The approved list ships as data (`src/submission_checker/data/approved_drafters.yaml`),
+transcribed from the agentic reference README's "Approved Checkpoints and
+Speculative-Decoding Heads" (mlcommons/endpoints `examples/10_Agentic_Inference`): the
+Kimi K3 DSpark heads, and the native heads of the approved DeepSeek-V4.1-Flash and
+Qwen3.6-35B-A3B checkpoints. Each is weight-identified by its Hugging Face revision
+(`weight_checksum: git-sha1:<revision>`), which a point's `speculative_decoding` block must
+declare to match. A benchmark with no entry has no approved drafter, so speculative
+decoding stays disallowed for it (§2.9.4). Point `--approved-drafters FILE` or
+`$MLPERF_ENDPOINTS_APPROVED_DRAFTERS` at a newer published list.
+
+All six initial drafter approvals are recorded against `2026-10-C1`, matching the
+published seed-set cohort. The two-cohort approval lead time makes their earliest
+eligible target cohort `2026-11-C1`.
+Approval cohorts are recorded per drafter, so future additions can carry later
+cohorts in the same registry.
 
 The published sets ship as data (`src/submission_checker/data/seed_sets.yaml`), mirrored
 from the policies repo's `seedset.yaml`. The file's `cohort-id` is the cohort its sets
@@ -478,7 +488,7 @@ set published after this release.
 | `metric-consistency-tpot-p90` | §9.1 | Reported TPOT P90 present, finite, strictly positive |
 | `metric-consistency-tps-per-user` | §9.1 | Stored `tps_per_user` matches `1000 / tpot_p90_ms` |
 | `metric-consistency-tps-per-kw` | §4.5.3 | Stored `system_tps_per_kw` matches `system_tps / point_power_kw` — see [per-point normalisation](#per-point-normalisation-453) |
-| `agentic-metric-consistency` | §4.1 | `e2e_avg_interactivity` is derivable from its reported inputs |
+| `agentic-metric-consistency` | §4.1 | `e2e_avg_interactivity` is derivable from its reported inputs: the per-turn sums, or the client's `output_sequence_lengths.total / (latency.total / 1e9)` when no sample failed (latency is in nanoseconds) |
 
 ### Accuracy (§15)
 

src/submission_checker/data/approved_drafters.yaml

@@ -1,13 +1,14 @@
 # Approved speculative-decoding drafters (§2.9.4), per benchmark.
 #
 # §2.9.4 makes this "a curated, published list, not a single fixed drafter",
-# published in the reference repository and versioned per submission round. No such
-# list has been published yet, so this file ships empty and the checker treats an
-# empty registry as "speculative decoding is not yet permitted for any benchmark" —
-# which is exactly §2.9.4's rule for a benchmark with no approved drafter.
+# published in the reference repository and versioned per submission round. The
+# agentic benchmarks' list is published in mlcommons/endpoints
+# examples/10_Agentic_Inference/README.md, "Approved Checkpoints and
+# Speculative-Decoding Heads"; the entries below transcribe it. A benchmark absent
+# here still has no approved drafter, so speculative decoding stays disallowed for it.
 #
 # Override with `--approved-drafters FILE` or $MLPERF_ENDPOINTS_APPROVED_DRAFTERS
-# once the task force publishes one, without waiting for a checker release.
+# when the task force publishes a newer list, without waiting for a checker release.
 #
 # Shape, per §2.9.4's two identification forms:
 #
@@ -21,5 +22,48 @@
 #       target_checksum: sha256:...      # configuration-identified
 #       configuration:
 #         exit_layer: 24
+#
+# How the reference entries are transcribed:
+#
+# - Weight-identified. MTP and DSpark heads carry their own weights (§2.9.4's
+#   configuration form is for drafters with none). A head native to an approved
+#   checkpoint is identified by that checkpoint.
+# - weight_checksum is the Hugging Face revision the README pins, as
+#   `git-sha1:<revision>`. A revision is the git commit of the model repository; its
+#   tree pins every weight file's SHA-256 through the LFS pointers, so it identifies
+#   the published weights exactly.
+# - These initial entries are recorded against 2026-10-C1, matching the published
+#   cohort in seed_sets.yaml. The two-cohort approval lead time therefore makes
+#   them eligible starting with target cohort 2026-11-C1.
+#   Future entries record their own approval cohorts alongside these entries.
+
+drafters:
+  # Kimi K3: separately published DSpark heads (endpoints#494).
+  - benchmark: kimi-k3
+    approved_cohort: 2026-10-C1
+    model_id: RadixArk/Kimi-K3-DSpark
+    weight_checksum: git-sha1:3c5bac301d9cf392706189d82ed947feca6c2f0f
+  - benchmark: kimi-k3
+    approved_cohort: 2026-10-C1
+    model_id: Inferact/Kimi-K3-DSpark
+    weight_checksum: git-sha1:cf6b8244620e7ea4b0651d214f28e89eac75bed6
+
+  # DeepSeek-V4.1-Flash: the native DSpark head in the approved checkpoint (endpoints#519).
+  - benchmark: deepseek-v4_1-flash
+    approved_cohort: 2026-10-C1
+    model_id: deepseek-ai/DeepSeek-V4.1-Flash
+    weight_checksum: git-sha1:dba1be0a40aa45a94ad051997016db3960a90277
 
-drafters: []
+  # Qwen3.6-35B-A3B: the native MTP heads in each approved checkpoint (endpoints#494).
+  - benchmark: qwen3_6-35b-a3b
+    approved_cohort: 2026-10-C1
+    model_id: Qwen/Qwen3.6-35B-A3B
+    weight_checksum: git-sha1:995ad96eacd98c81ed38be0c5b274b04031597b0
+  - benchmark: qwen3_6-35b-a3b
+    approved_cohort: 2026-10-C1
+    model_id: Qwen/Qwen3.6-35B-A3B-FP8
+    weight_checksum: git-sha1:95a723d08a9490559dae23d0cff1d9466213d989
+  - benchmark: qwen3_6-35b-a3b
+    approved_cohort: 2026-10-C1
+    model_id: nvidia/Qwen3.6-35B-A3B-NVFP4
+    weight_checksum: git-sha1:1355db6a052410cfd62085d94b58866fd0f2c3c5

src/submission_checker/data/messages.yaml

@@ -108,8 +108,9 @@ agentic-metric-consistency:
   spec: §4.1
   messages:
     pass: e2e_avg_interactivity={derived:.4f} tok/s across completed turns
-    fail: e2e_avg_interactivity {stored!r} is reported, but the summary states no output_tokens_per_turn_total
-      / e2e_turn_time_seconds_total to derive it from (§4.1)
+    fail: e2e_avg_interactivity {stored!r} is reported, but the summary states neither output_tokens_per_turn_total
+      / e2e_turn_time_seconds_total nor a failure-free output_sequence_lengths.total / latency.total to derive
+      it from (§4.1)
     fail-2: stored e2e_avg_interactivity {stored:.4f} ≠ derived {derived:.4f} (rel err {rel_err:.1%})
 agentic-osl-range:
   title: Agentic OSL range

src/submission_checker/models/file/point_summary.py

@@ -65,6 +65,8 @@ class PointSummary(BaseModel):
     ttft: PercentileStats = Field(default_factory=PercentileStats)
     tpot: PercentileStats = Field(default_factory=PercentileStats)
     output_sequence_lengths: PercentileStats = Field(default_factory=PercentileStats)
+    #: Per-sample request latency in nanoseconds; ``total`` is its sum over the samples.
+    latency: PercentileStats = Field(default_factory=PercentileStats)
 
     #: §4.1 agentic inputs. Summed across every completed turn of every trajectory;
     #: e2e_turn_time excludes tool-call execution time.
@@ -145,11 +147,21 @@ def e2e_avg_interactivity(self) -> float | None:
         """§4.1: ``sum(output_tokens_per_turn) / sum(e2e_turn_time_seconds)``.
 
         The agentic analogue of ``tps_per_user`` — the output-token rate across
-        completed turns, one scalar per measurement point. ``None`` for a single-turn
-        benchmark, which reports neither input.
+        completed turns, one scalar per measurement point. Also derivable for
+        single-turn reports; ``None`` when usable input totals are unavailable.
+
+        The reference client reports the same two sums under its own names:
+        ``output_sequence_lengths.total`` (tokens) and ``latency.total`` (nanoseconds of
+        per-turn request latency, so tool-call time between turns is not in it). Those
+        are used when the named sums are absent, and only when no sample failed: output
+        tokens are recorded for successful turns only, latency for every terminal one,
+        which is why the client itself omits the ratio then.
         """
         tokens = self.output_tokens_per_turn_total
         seconds = self.e2e_turn_time_seconds_total
+        if tokens is None and seconds is None and self.n_samples_failed == 0:
+            tokens = self.output_sequence_lengths.total
+            seconds = self.latency.total / 1e9
         if tokens is None or seconds is None or seconds <= 0:
             return None
         return tokens / seconds

tests/submission_checker/test_checks_specdec_agentic.py

@@ -69,10 +69,71 @@ def _errors(binding: DrafterBinding, rule: str) -> list:
 
 @pytest.mark.unit
 class TestDrafterList:
-    def test_bundled_list_ships_empty(self) -> None:
-        """§2.9.4's list is not published yet; empty means "none approved"."""
+    #: The approved heads in mlcommons/endpoints examples/10_Agentic_Inference/README.md,
+    #: "Approved Checkpoints and Speculative-Decoding Heads": benchmark -> {model_id: revision}.
+    #: Qwen and DeepSeek-V4.1-Flash heads are native to their approved checkpoints.
+    _REFERENCE_HEADS = {
+        "kimi-k3": {
+            "RadixArk/Kimi-K3-DSpark": "3c5bac301d9cf392706189d82ed947feca6c2f0f",
+            "Inferact/Kimi-K3-DSpark": "cf6b8244620e7ea4b0651d214f28e89eac75bed6",
+        },
+        "deepseek-v4_1-flash": {
+            "deepseek-ai/DeepSeek-V4.1-Flash": "dba1be0a40aa45a94ad051997016db3960a90277",
+        },
+        "qwen3_6-35b-a3b": {
+            "Qwen/Qwen3.6-35B-A3B": "995ad96eacd98c81ed38be0c5b274b04031597b0",
+            "Qwen/Qwen3.6-35B-A3B-FP8": "95a723d08a9490559dae23d0cff1d9466213d989",
+            "nvidia/Qwen3.6-35B-A3B-NVFP4": "1355db6a052410cfd62085d94b58866fd0f2c3c5",
+        },
+    }
+
+    def test_bundled_list_carries_the_reference_approved_heads(self) -> None:
         assert bundled_drafters_path().is_file()
-        assert load_approved_drafters() == {}
+        bundled = {
+            benchmark: {d.model_id: d.weight_checksum for d in drafters}
+            for benchmark, drafters in load_approved_drafters().items()
+        }
+        assert bundled == {
+            benchmark: {model_id: f"git-sha1:{rev}" for model_id, rev in heads.items()}
+            for benchmark, heads in self._REFERENCE_HEADS.items()
+        }
+
+    def test_bundled_heads_record_the_initial_2026_10_c1_cohort(self) -> None:
+        """Initial approvals use the cohort published alongside the seed sets."""
+        cohorts = {
+            d.model_id: d.approved_cohort for ds in load_approved_drafters().values() for d in ds
+        }
+        assert cohorts == {
+            model_id: "2026-10-C1"
+            for heads in self._REFERENCE_HEADS.values()
+            for model_id in heads
+        }
+
+    @pytest.mark.parametrize(
+        ("target_cohort", "expected_severity"),
+        [
+            ("2026-10-C1", Severity.ERROR),
+            ("2026-11-C0", Severity.ERROR),
+            ("2026-11-C1", Severity.INFO),
+        ],
+    )
+    def test_bundled_heads_respect_their_approval_lead_time(
+        self, tmp_path: Path, target_cohort: str, expected_severity: Severity
+    ) -> None:
+        for benchmark, approved in load_approved_drafters().items():
+            for drafter in approved:
+                binding = DrafterBinding(
+                    points=[
+                        _point(tmp_path, {"weight_checksum": drafter.weight_checksum}, target_cohort)
+                    ],
+                    approved=approved,
+                    benchmark=benchmark,
+                    model_dir=tmp_path,
+                )
+                results = [
+                    r for r in binding._check_results if r.rule == "drafter-approval-lead-time"
+                ]
+                assert [r.severity for r in results] == [expected_severity], drafter.model_id
 
     def test_entries_group_by_benchmark(self, tmp_path: Path) -> None:
         path = tmp_path / "d.yaml"
@@ -266,3 +327,59 @@ def test_zero_turn_time_is_not_a_division_error(self, tmp_path: Path) -> None:
             e2e_turn_time_seconds_total=0.0,
         )
         assert summary.e2e_avg_interactivity is None
+
+    # The reference client writes the same two sums under its own names,
+    # output_sequence_lengths.total (tokens) and latency.total (ns), and leaves
+    # e2e_avg_interactivity out when any sample failed (endpoints metrics/report.py).
+
+    def test_derived_from_the_clients_token_and_latency_totals(self) -> None:
+        summary = PointSummary(
+            n_samples_completed=1,
+            duration_ns=1.0,
+            output_sequence_lengths={"total": 6000.0},
+            latency={"total": 120_000_000_000.0},
+        )
+        assert summary.e2e_avg_interactivity == pytest.approx(50.0)
+
+    def test_client_report_with_its_own_value_passes(self, tmp_path: Path) -> None:
+        """A measured client report: 30,048,888 tokens over 65,037.28 s of turn latency."""
+        result = self._result(
+            tmp_path,
+            output_sequence_lengths={"total": 30048888},
+            latency={"total": 65037277147776},
+            e2e_avg_interactivity=462.02561542857495,
+        )
+        hits = self._hits(result)
+        assert hits and all(r.severity == Severity.INFO for r in hits)
+
+    def test_client_report_with_a_wrong_value_errors(self, tmp_path: Path) -> None:
+        result = self._result(
+            tmp_path,
+            output_sequence_lengths={"total": 6000.0},
+            latency={"total": 120_000_000_000.0},
+            e2e_avg_interactivity=999.0,
+        )
+        errors = [r for r in self._hits(result) if r.severity == Severity.ERROR]
+        assert [r.key for r in errors] == ["fail-2"]  # a mismatch, not missing inputs
+
+    def test_client_totals_are_not_paired_when_a_sample_failed(self) -> None:
+        """Tokens cover successful turns only and latency every terminal one."""
+        totals = {
+            "n_samples_completed": 1,
+            "duration_ns": 1.0,
+            "output_sequence_lengths": {"total": 6000.0},
+            "latency": {"total": 120_000_000_000.0},
+        }
+        assert PointSummary(**totals).e2e_avg_interactivity == pytest.approx(50.0)
+        assert PointSummary(**totals, n_samples_failed=1).e2e_avg_interactivity is None
+
+    def test_the_named_sums_take_precedence_over_the_client_totals(self) -> None:
+        summary = PointSummary(
+            n_samples_completed=1,
+            duration_ns=1.0,
+            output_tokens_per_turn_total=6000.0,
+            e2e_turn_time_seconds_total=120.0,
+            output_sequence_lengths={"total": 1.0},
+            latency={"total": 1_000_000_000.0},
+        )
+        assert summary.e2e_avg_interactivity == pytest.approx(50.0)