From d059fa2a53e92051714ced0306565a9e291f7e88 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 12:16:54 -0700 Subject: [PATCH 01/23] docs: gap analysis and plan for video-generation pareto support The client can already drive video-generation endpoints, but every shipped video config is a single measurement point. This records what is missing to run one as a multi-point pareto sweep, verified against 4235a9c, with requirements cited against endpoints_policies v1.0_rules_dev @ 6b0b1ef. That branch is unmerged and moving, so citations are marked as-of. Two things are already correct for a tokenless workload and need no change: Report.qps is n_completed over the tracked duration, which for video is videos/second, and a per-request latency percentile is already emitted with P90 in the default grid. The per-user rate mirrors v1.0's latency-reciprocal definition rather than dividing throughput by concurrency. The blockers are that no minimum-duration floor exists in the stop predicate; that min_issue_duration_ms is a poisson count-sizer rather than a floor and is rejected for concurrency; that steady-state detection is gated on a resolved tokenizer and streaming, which a video run lacks by construction, while v1.0 4.4 makes that verdict the official reporting basis; and that config-lock requires target_concurrency == 1 and so fails at every point of a curve. Includes a workaround that satisfies both the duration floor and 6.4's whole-pass requirement by sizing the sample count to the duration, an itemised task breakdown with dependencies and sizes, and an explicit out-of-scope section. Co-Authored-By: Claude Sonnet 4.6 --- docs/videogen_pareto_support.md | 212 ++++++++++++++++++++++++++++++++ 1 file changed, 212 insertions(+) create mode 100644 docs/videogen_pareto_support.md diff --git a/docs/videogen_pareto_support.md b/docs/videogen_pareto_support.md new file mode 100644 index 00000000..9f586922 --- /dev/null +++ b/docs/videogen_pareto_support.md @@ -0,0 +1,212 @@ +# Video-generation pareto support — gaps and plan + +Status: proposal · Baseline: `4235a9c` · Scope: this repository only. + +> **Rules basis.** Requirements below are cited against `endpoints_policies` branch +> `v1.0_rules_dev` @ `6b0b1ef` (2026-09-22). That branch is **unmerged and still moving**; every +> rule citation here is as-of that commit and should be re-checked before being relied on. + +The client already drives video-generation endpoints (`videogen/`, +`examples/09_Wan22_VideoGen_Example/`), but every shipped video config is a **single measurement +point**. This records what is missing to run one as a **multi-point pareto sweep**, and itemises +the pending work in §5. + +This covers the client slice only. §6 lists what it deliberately excludes — admissibility under +the Endpoints rules, the accuracy target, submitter-side tooling, and the serving stack are all +out of scope here. + +## 1. Target shape + +Points at fixed concurrency under `ConcurrencyScheduler`, each sustaining a minimum steady-state +window, **plus one Offline point** (§5.7 requires it for every non-agentic benchmark), plus +accuracy runs — 5 for a non-agentic benchmark, one per mandatory point (§5.3). + +Two rules shape a video run in particular: + +- **§6.4** — total samples issued at a point MUST be a positive integer multiple of the dataset + size. Runs end on a whole pass, not on a clock. +- **§6.2** — minimum steady-state duration is 600 s in the Ultra Low Concurrency region (1–32) + and **1200 s** in the Low, Medium, and High Concurrency regions. + +Per-point metrics cannot be token-derived — a response is one artifact, not a token stream: + +| Token metric | Video-generation substitute | +| --- | --- | +| `system_tps` | `completed_videos / elapsed_seconds` | +| `tps_per_user` (v1.0: `1000 / tpot_p90_ms`) | `1000 / latency_p90_ms` — the same latency-reciprocal shape | +| `ttft_*` | end-to-end issue → complete-response percentile | + +The per-user metric mirrors v1.0's definition rather than dividing throughput by concurrency: v1.0 +defines per-user rate from the tail latency a user actually experiences, and the video analog +should keep that shape. + +**Percentile: P90.** This now *matches* v1.0 rather than diverging from it — §4.1 moved TTFT from +P95 (v0.7) to P90, with an explicit versioning note. The independent reason still holds: a stable +P95 needs roughly twice the completed queries of a P90, and since one video takes tens of seconds +to minutes, a P95 at concurrency 1 would need on the order of 100+ videos per point. Already +available — `DEFAULT_PERCENTILES` carries both +(`async_utils/services/metrics_aggregator/registry.py:411-423`). + +**Offline point.** `max_throughput` is the Offline load pattern, so the existing +`examples/09_Wan22_VideoGen_Example/offline_wan22*.yaml` configs are already the right shape for +it. Per §5.7 the reported concurrency is the dataset cardinality, throughput is the only metric of +interest, and latency metrics explicitly do not apply. + +## 2. Already works — no change needed + +- **System throughput.** `Report.qps` = `n_completed / duration_s`, duration from the + `tracked_duration_ns` counter (`metrics/report.py:360-361`, `:390-396`). For video that *is* + videos/second. The legacy LoadGen window is poisson-only + (`commands/benchmark/pipeline.py:222-225`), so concurrency uses the native window. +- **Per-request latency percentile.** `sample_latency_ns` → `latency` + (`metrics/report.py:50`, `:234`), P90 and P95 both in the default grid. +- **Per-user rate is already derivable.** `1000 / latency_p90_ms` needs no new measurement — the + P90 is in the emitted grid. `target_concurrency` also lands in `result_summary.json` via + `run_config`, so points are self-identifying. +- **`ConcurrencyScheduler`.** Semaphore released on every terminal result, errors included + (`load_generator/strategy.py`). +- **Phase isolation.** `max_issue_duration_ms` bounds only the performance phase + (`commands/benchmark/watchdog.py`). +- **Sample order is infinite** — never raises `StopIteration` (`load_generator/sample_order.py`). +- **Accuracy.** VBench path runs; the accuracy phase mirrors the performance load pattern. +- **Audit.** TEST04 already accepts `concurrency` + (`compliance/audit_test/output_caching_test.py`). +- **Missing tokenizer is handled gracefully**, not fatally (`commands/benchmark/execute.py`). + +## 3. Gaps + +| # | Gap | Evidence | Kind | +| --- | --- | --- | --- | +| 1 | Stop predicate is a pure OR of stop-requested / count-reached / `max_duration_ns` exceeded. No branch holds off the count stop until a wall-clock floor passes, so a phase is count-driven or capped — never floored. | `load_generator/session.py:866-880` | code | +| 2 | `min_issue_duration_ms` is a poisson count-sizer (`target_qps × duration`), not a floor. Rejected for non-poisson patterns; relaxing that validator alone is insufficient — `total_samples_to_issue()` then raises, since concurrency has no `target_qps`. | `config/schema.py:1021-1027`, `config/runtime_settings.py:249` | code | +| 3 | Steady state cannot certify a tokenless window — see §3.1. | `metrics/steady_state_diagnostics.py` | code | +| 4 | The per-user rate is not emitted as a named field; `Report` carries `qps` and `tps` only. Minor — it is one reciprocal of the already-emitted `latency` P90. | `metrics/report.py:250` | code | +| 5 | Config lock requires `target_concurrency == 1`, so it fails at every point of a curve above 1. `_resolve_model` also raises `KeyError` for a model absent from a ruleset, and the lock expects a `temperature` field the video request type lacks. | `compliance/checker.py:149`, `:115-119` | code | +| 6 | `VideoGenAdapter` mirrors the video *path string* into `response_output`, which the OSL trigger tokenizes. Harmless only because the model name resolves to no tokenizer; supplying one yields meaningless OSL/TPS. | `videogen/adapter.py` | latent | +| 7 | No sweep driver — one invocation is one point, no per-point report-dir convention, no cross-point aggregation. No shipped video config uses concurrency > 1. `publish_submission.py` publishes a single run. | — | tooling | + +Gaps 1 and 2 together mean the steady-state window a pareto point needs **cannot currently be +expressed** for a concurrency run. + +### 3.1 Why steady state cannot gate a video run + +This is the most consequential gap. Under v1.0 §4.4 the detected steady-state window is not a +diagnostic — it is **the official reporting basis**: a point's metrics are computed over that +window, with whole-run values kept only as supplementary. §4.4 names the gating metric as TPOT at +P50 and P90. A workload with no TPOT therefore has no gating metric, and so no official result — +not merely a missing diagnostic section. + +See [`steady_state_diagnostics.md`](steady_state_diagnostics.md) for the algorithm; this is only +why it does not apply. + +- **The exclusion is explicit, not accidental.** `steady_state_profile` + (`commands/benchmark/pipeline.py:96-118`) returns `None` — no collection — unless the run has a + resolved tokenizer *and* streaming, on top of `settings.steady_state.enabled` being set at all + (it is off by default). A video run fails both conditions by construction. +- The reasoning is sound for its purpose: without streaming there is no `TpotTrigger`, so every + plateau gate would see an empty TPOT series and every verdict would be `found: false`. +- `tpot_ns` derives from tokens after the first chunk; TPOT is the gated metric, TTFT is + diagnostic only. `MIN_DUR_FLOOR_S` is already 600 s but never reached. +- Outcome is a `None` verdict, not an error: `Report.steady_state` is `SteadyState | None` + (`metrics/report.py:289`, `:456`). +- There is no way for a workload to declare a different gating metric — eligibility comes from + `Profile.supported` (`metrics/steady_state_diagnostics.py:311`) and `SteadyStateConfig` + (`config/schema.py:947`, `:994`) carries no model allowlist (`:958`). + +The tension is between a defensible client design and the v1.0 rule: the client correctly declines +to emit a verdict it cannot compute, while §4.4 makes that verdict the basis of the official +result. Closing it needs a gating metric that exists for tokenless workloads, not a change on +either side alone. + +## 4. Workaround available today + +Since a run must end on a whole dataset pass (§6.4) *and* meet a duration floor (§6.2), and no +floor setting exists (gaps 1–2), size the count to the duration instead of capping the clock: + +1. Calibrate — run the point briefly to estimate sustained throughput at that concurrency. +2. Choose the smallest integer `N` where `N × dataset_size / throughput` comfortably exceeds the + region's minimum duration. +3. Issue exactly that many samples, with no clock cap: + +```yaml +settings: + runtime: + n_samples_to_issue: 1240 # N x dataset_size (e.g. 5 x 248) — an exact multiple + load_pattern: + type: "concurrency" + target_concurrency: 1 +``` + +The phase then ends on the count, at a pass boundary, having run at least the required duration — +satisfying both rules without a client patch. `n_samples_to_issue` is returned verbatim and the +sample order is infinite, so no clamping interferes. + +- **The duration is achieved, not enforced.** If throughput is lower than calibrated the point + simply runs longer, which is safe; if higher, it may undershoot the floor, so size `N` with + margin and check the achieved duration afterwards. +- Every point above concurrency 1 still fails the compliance checker (gap 5). +- Do **not** use `min_issue_duration_ms` for this (gap 2). +- Capping with `max_issue_duration_ms` instead would end the run mid-pass and violate §6.4; it + remains useful only as a runaway guard set well above the expected finish. + +## 5. Pending work + +Every code task carries tests — `AGENTS.md` sets >90% coverage and requires an explicit marker. +`videogen/` already has unit and integration suites (`tests/unit/videogen/`, +`tests/integration/videogen/`) to extend rather than start from. + +### A. Core runtime + +| # | Task | Fixes | Depends on | Size | +| --- | --- | --- | --- | --- | +| A1 | Duration floor: gate the count branch on minimum elapsed time (AND semantics), and accept the setting for `concurrency`. Must land as one change — relaxing the validator alone makes `total_samples_to_issue()` raise. Prefer a new setting name over widening `min_issue_duration_ms`. | 1, 2 | — | S | +| A2 | Emit the per-user rate (`1000 / latency_p90_ms`) as a named field rather than leaving every consumer to derive it. | 4 | — | XS | +| A3 | Modality-aware steady-state gating: gate on an alternative metric where the gated one has no samples, or report un-gated with a reason instead of returning `None`. Requires a way for a workload to declare its gating metric. | 3 | — | M | +| A4 | Make the reported latency percentile explicit (P90, per §4.1) rather than leaving consumers to pick from the grid. | — | A2 | XS | +| A5 | Remove the tokenizer footgun: stop routing a path through `response_output`, or mark the field non-tokenizable for artifact-output adapters. | 6 | — | S | + +### B. Compliance and validation + +| # | Task | Fixes | Depends on | Size | +| --- | --- | --- | --- | --- | +| B1 | Validate multi-point curves: config lock currently requires `target_concurrency == 1`, and expects a `temperature` field artifact adapters lack. Redesigns what `check_submission` covers. | 5 | A1 | L | +| B2 | Register a video benchmark as a ruleset model and dataset (`config/rulesets/mlcommons/`). Not needed to *run*, but without it there is no golden accuracy, no validity thresholds, and `_resolve_model` raises. | 5 | — | S | + +### C. Configuration and tooling + +| # | Task | Fixes | Depends on | Size | +| --- | --- | --- | --- | --- | +| C1 | Add concurrency-region example configs sized per §6.4. The shipped configs cover only `max_throughput` (the Offline point) and `concurrency: 1`. | 7 | A1 | XS | +| C2 | Decide whether a multi-point sweep driver belongs here or in submitter tooling, then build or document accordingly. | 7 | — | M | +| C3 | Multi-point publish layout — `publish_submission.py` publishes one run into one scenario directory. | 7 | C2 | M | + +### D. Documentation + +| # | Task | Depends on | +| --- | --- | --- | +| D1 | Update this document as items land; update `AGENTS.md` if any module moves or is added. | any | +| D2 | Note the modality constraint in `steady_state_diagnostics.md` once A3 settles. | A3 | + +**Critical path:** A1 → C1 → (B1, C2). A1 unblocks a legitimate multi-point run; B1 is the only +item that is unavoidable rather than convenient. + +## 6. Out of scope for this repository + +Listed so the boundary is explicit. None are tracked here. + +- Whether a video-generation benchmark is admissible under the Endpoints rules, and any rule text + it would need — a working-group question. +- The accuracy quality target itself. The scorer exists; the threshold is a benchmark-definition + decision, not a client one. +- Submitter-side result layout, aggregation, and visualisation of a finished curve. +- The model server and serving stack under test. +- Executing sweeps and producing results. + +## 7. Open questions + +- Should a sweep driver live here or in submitter tooling? Nothing here orchestrates multiple + points today. +- Sizing `N` per §6.4 needs a throughput estimate before each point. Should calibration be a + documented manual step, or something the client can do itself? +- §4.5 introduces provisioned-power normalisation of throughput, marked tentative. Nothing here + computes it, and the units are not yet settled — worth tracking, not building. From bf3af6f40900be7482c769500e7c4381331e5a95 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 12:27:56 -0700 Subject: [PATCH 02/23] docs: reframe as pareto sweep support, video as motivating case Retitles and rescopes docs/videogen_pareto_support.md to docs/pareto_sweep_support.md. Five of the seven gaps are not specific to video at all. No minimum duration floor exists in the stop predicate, min_issue_duration_ms is a poisson count-sizer rejected for concurrency, config lock requires target_concurrency == 1, there is no sweep driver, and the per-user rate is not emitted. Each blocks a multi-point curve for any model, which is why existing curves are built by running each concurrency separately and stitching the results downstream. Only two gaps are tokenless-specific: steady state cannot certify a window without TPOT, and the video adapter routes a file path through the field the OSL trigger tokenizes. The gap table now carries a Scope column separating the two, and the task breakdown is grouped the same way. Video generation remains the motivating case, since it is what surfaced all of this, but it is no longer presented as the scope. Co-Authored-By: Claude Sonnet 4.6 --- docs/pareto_sweep_support.md | 222 ++++++++++++++++++++++++++++++++ docs/videogen_pareto_support.md | 212 ------------------------------ 2 files changed, 222 insertions(+), 212 deletions(-) create mode 100644 docs/pareto_sweep_support.md delete mode 100644 docs/videogen_pareto_support.md diff --git a/docs/pareto_sweep_support.md b/docs/pareto_sweep_support.md new file mode 100644 index 00000000..d35203e8 --- /dev/null +++ b/docs/pareto_sweep_support.md @@ -0,0 +1,222 @@ +# Multi-point pareto sweep support: gaps and plan + +Status: proposal · Baseline: `4235a9c` · Scope: this repository only. + +> **Rules basis.** Requirements below are cited against `endpoints_policies` branch +> `v1.0_rules_dev` @ `6b0b1ef` (2026-09-22). That branch is **unmerged and still moving**; every +> rule citation here is as-of that commit and should be re-checked before being relied on. + +An Endpoints submission is a **pareto curve**: several measurement points at different +concurrencies, each sustaining a minimum steady-state window, plus an Offline point and accuracy +runs. The client measures **one point per invocation**. Everything needed to get from one point to +a curve, and to verify that the points are valid, is either missing or only reachable by working +around a setting that does something else. + +Five of the seven gaps in §3 apply to **any** model. Existing text-model curves were produced by +running each concurrency separately and stitching the results together downstream, which is the +workaround for exactly these gaps. + +**Video generation is the motivating case**, not the scope. It was the workload that surfaced +this, and it adds two gaps of its own (§3.1, §3.2) because its responses are single artifacts +rather than token streams. Those are marked `tokenless` in the gap table. + +This covers the client slice only. §6 lists what it deliberately excludes. + +## 1. Target shape + +Per v1.0 §5.3 and §5.7, a non-agentic submission is four mandatory concurrency points (Ultra Low +1-32, Low, Medium, High), three submitter's-choice points, one Offline point, and five accuracy +runs. + +Two rules govern how a single point must run: + +- **§6.4**: total samples issued at a point MUST be a positive integer multiple of the dataset + size. A point ends on a whole pass, not on a clock. +- **§6.2**: minimum steady-state duration is 600 s in the Ultra Low Concurrency region and + **1200 s** in the Low, Medium, and High regions. + +Neither is expressible today for a concurrency-scheduled run: see gaps 1 and 2. + +### 1.1 Metrics for a tokenless workload + +For video generation the per-point metrics cannot be token-derived, because a response is one +artifact rather than a token stream: + +| Token metric | Substitute | +| --- | --- | +| `system_tps` | `completed_videos / elapsed_seconds` | +| `tps_per_user` (v1.0: `1000 / tpot_p90_ms`) | `1000 / latency_p90_ms`, the same latency-reciprocal shape | +| `ttft_*` | end-to-end issue to complete-response percentile | + +The per-user metric mirrors v1.0's definition rather than dividing throughput by concurrency, +because v1.0 derives per-user rate from the tail latency a user actually experiences. + +**Percentile: P90.** This *matches* v1.0 rather than diverging from it, since §4.1 moved TTFT from +P95 to P90 with an explicit versioning note. The independent reason also holds: a stable P95 needs +roughly twice the completed queries of a P90, and one video takes tens of seconds to minutes, so a +P95 at concurrency 1 would need on the order of 100+ videos per point. `DEFAULT_PERCENTILES` +already carries both (`async_utils/services/metrics_aggregator/registry.py:411-423`). + +**Offline point.** `max_throughput` is the Offline load pattern, so the existing +`examples/09_Wan22_VideoGen_Example/offline_wan22*.yaml` configs are already the right shape. Per +§5.7 the reported concurrency is the dataset cardinality, throughput is the only metric of +interest, and latency metrics explicitly do not apply. + +## 2. Already works, no change needed + +- **System throughput.** `Report.qps` = `n_completed / duration_s`, duration from the + `tracked_duration_ns` counter (`metrics/report.py:360-361`, `:390-396`). For a video workload + that *is* videos/second. The legacy LoadGen window is poisson-only + (`commands/benchmark/pipeline.py:222-225`), so concurrency uses the native window. +- **Per-request latency percentile.** `sample_latency_ns` maps to `latency` + (`metrics/report.py:50`, `:234`), with P90 and P95 both in the default grid. +- **`ConcurrencyScheduler`.** Semaphore released on every terminal result, errors included + (`load_generator/strategy.py`). +- **Phase isolation.** `max_issue_duration_ms` bounds only the performance phase + (`commands/benchmark/watchdog.py`). +- **Sample order is infinite**, never raising `StopIteration` (`load_generator/sample_order.py`). +- **Point identity.** `target_concurrency` lands in `result_summary.json` via `run_config`. +- **Audit.** TEST04 already accepts `concurrency` + (`compliance/audit_test/output_caching_test.py`). +- **Missing tokenizer is handled gracefully**, not fatally (`commands/benchmark/execute.py`). + +## 3. Gaps + +`Scope` distinguishes gaps that block any model from those specific to a tokenless workload. + +| # | Gap | Evidence | Scope | Kind | +| --- | --- | --- | --- | --- | +| 1 | Stop predicate is a pure OR of stop-requested / count-reached / `max_duration_ns` exceeded. No branch holds off the count stop until a wall-clock floor passes, so a phase is count-driven or capped, never floored. | `load_generator/session.py:866-880` | any model | code | +| 2 | `min_issue_duration_ms` is a poisson count-sizer (`target_qps × duration`), not a floor. Rejected for non-poisson patterns, and relaxing that validator alone is insufficient because `total_samples_to_issue()` then raises, since concurrency has no `target_qps`. | `config/schema.py:1021-1027`, `config/runtime_settings.py:249` | any model | code | +| 3 | Config lock requires `target_concurrency == 1`, so it fails at every point of a curve above 1. `_resolve_model` also raises `KeyError` for a model absent from a ruleset. | `compliance/checker.py:149` | any model | code | +| 4 | No sweep driver. One invocation is one point, with no per-point report-dir convention and no cross-point aggregation. `publish_submission.py` publishes a single run into a single scenario directory. | n/a | any model | tooling | +| 5 | The per-user rate is not emitted as a named field; `Report` carries `qps` and `tps` only. Minor, since it is one reciprocal of the already-emitted `latency` P90. | `metrics/report.py:250` | any model | code | +| 6 | Steady state cannot certify a tokenless window. See §3.1. | `metrics/steady_state_diagnostics.py` | tokenless | code | +| 7 | `VideoGenAdapter` mirrors the video *path string* into `response_output`, which the OSL trigger tokenizes. Harmless only because the model name resolves to no tokenizer; supplying one yields meaningless OSL and TPS. See §3.2. | `videogen/adapter.py` | tokenless | latent | + +Gaps 1 and 2 together mean the steady-state window a pareto point needs **cannot currently be +expressed** for a concurrency run, for any model. Gap 3 means that even a correctly measured curve +cannot be validated by this repository's checker. + +### 3.1 Why steady state cannot gate a tokenless run + +This is the most consequential of the tokenless gaps. Under v1.0 §4.4 the detected steady-state +window is not a diagnostic. It is **the official reporting basis**: a point's metrics are computed +over that window, with whole-run values kept only as supplementary. §4.4 names the gating metric +as TPOT at P50 and P90. A workload with no TPOT therefore has no gating metric, and so no official +result. + +See [`steady_state_diagnostics.md`](steady_state_diagnostics.md) for the algorithm; this is only +why it does not apply. + +- **The exclusion is explicit, not accidental.** `steady_state_profile` + (`commands/benchmark/pipeline.py:96-118`) returns `None`, meaning no collection, unless the run + has a resolved tokenizer *and* streaming, on top of `settings.steady_state.enabled` being set at + all (it is off by default). A tokenless run fails both conditions by construction. +- The reasoning is sound for its purpose: without streaming there is no `TpotTrigger`, so every + plateau gate would see an empty TPOT series and every verdict would be `found: false`. +- Outcome is a `None` verdict, not an error: `Report.steady_state` is `SteadyState | None` + (`metrics/report.py:289`, `:456`). +- There is no way for a workload to declare a different gating metric. Eligibility comes from + `Profile.supported` (`metrics/steady_state_diagnostics.py:311`), and `SteadyStateConfig` + (`config/schema.py:947`, `:994`) carries no model allowlist (`:958`). + +The tension is between a defensible client design and the v1.0 rule. The client correctly declines +to emit a verdict it cannot compute, while §4.4 makes that verdict the basis of the official +result. Closing it needs a gating metric that exists for tokenless workloads, not a change on +either side alone. + +### 3.2 Tokenizer footgun for artifact responses + +`VideoGenAdapter` puts the response's file path into `response_output`, which is the field the +output-sequence-length trigger tokenizes. Nothing breaks today only because the model name does +not resolve to a tokenizer, so token metrics stay disabled. Supplying `--tokenizer` would silently +produce OSL and TPS figures computed from a filesystem path. + +## 4. Workaround available today + +Since a point must end on a whole dataset pass (§6.4) *and* meet a duration floor (§6.2), and no +floor setting exists (gaps 1 and 2), size the count to the duration instead of capping the clock: + +1. Calibrate. Run the point briefly to estimate sustained throughput at that concurrency. +2. Choose the smallest integer `N` where `N × dataset_size / throughput` comfortably exceeds the + region's minimum duration. +3. Issue exactly that many samples, with no clock cap: + +```yaml +settings: + runtime: + n_samples_to_issue: 1240 # N x dataset_size (e.g. 5 x 248), an exact multiple + load_pattern: + type: "concurrency" + target_concurrency: 1 +``` + +The phase then ends on the count, at a pass boundary, having run at least the required duration, +satisfying both rules without a client patch. `n_samples_to_issue` is returned verbatim and the +sample order is infinite, so nothing clamps it. + +- **The duration is achieved, not enforced.** If throughput is lower than calibrated the point + simply runs longer, which is safe. If higher, it may undershoot the floor, so size `N` with + margin and check the achieved duration afterwards. +- Every point above concurrency 1 still fails the compliance checker (gap 3). +- Do **not** use `min_issue_duration_ms` for this (gap 2). +- Capping with `max_issue_duration_ms` instead would end the run mid-pass and violate §6.4. It + remains useful only as a runaway guard set well above the expected finish. + +## 5. Pending work + +Every code task carries tests. `AGENTS.md` sets >90% coverage and requires an explicit marker. + +### A. Core runtime (any model) + +| # | Task | Fixes | Depends on | Size | +| --- | --- | --- | --- | --- | +| A1 | Duration floor: gate the count branch on minimum elapsed time (AND semantics), and accept the setting for `concurrency`. Must land as one change, because relaxing the validator alone makes `total_samples_to_issue()` raise. Prefer a new setting name over widening `min_issue_duration_ms`. | 1, 2 | n/a | S | +| A2 | Emit the per-user rate as a named field rather than leaving every consumer to derive it. | 5 | n/a | XS | +| A3 | Make the reported latency percentile explicit (P90, per §4.1). | n/a | A2 | XS | + +### B. Compliance and tooling (any model) + +| # | Task | Fixes | Depends on | Size | +| --- | --- | --- | --- | --- | +| B1 | Validate multi-point curves. Config lock requires `target_concurrency == 1` and expects a `temperature` field artifact adapters lack. Redesigns what `check_submission` covers. | 3 | A1 | L | +| B2 | Decide whether a multi-point sweep driver belongs here or in submitter tooling, then build or document accordingly. | 4 | n/a | M | +| B3 | Multi-point publish layout for `publish_submission.py`. | 4 | B2 | M | + +### C. Tokenless workloads + +| # | Task | Fixes | Depends on | Size | +| --- | --- | --- | --- | --- | +| C1 | Modality-aware steady-state gating: gate on an alternative metric where the gated one has no samples (end-to-end latency exists for every workload), or report un-gated with a reason instead of returning `None`. Requires a way for a workload to declare its gating metric. | 6 | n/a | M | +| C2 | Remove the tokenizer footgun: stop routing a path through `response_output`, or mark the field non-tokenizable for artifact-output adapters. | 7 | n/a | S | +| C3 | Add concurrency-region example configs sized per §6.4. The shipped video configs cover only `max_throughput` (the Offline point) and `concurrency: 1`. | 4 | A1 | XS | + +### D. Documentation + +| # | Task | Depends on | +| --- | --- | --- | +| D1 | Update this document as items land; update `AGENTS.md` if any module moves or is added. | any | +| D2 | Note the modality constraint in `steady_state_diagnostics.md` once C1 settles. | C1 | + +**Critical path:** A1 → C3 → (B1, B2). A1 unblocks a legitimate multi-point run for every model. +B1 is the only item that is unavoidable rather than convenient. + +## 6. Out of scope for this repository + +- Whether a video-generation benchmark is admissible under the Endpoints rules, and any rule text + it would need. That is a working-group question. +- The accuracy quality target. The scorer exists; the threshold is a benchmark-definition + decision, not a client one. +- Submitter-side result layout, aggregation, and visualisation of a finished curve. +- The model server and serving stack under test. +- Executing sweeps and producing results. + +## 7. Open questions + +- Should a sweep driver live here or in submitter tooling? Nothing here orchestrates multiple + points today. +- Sizing `N` per §6.4 needs a throughput estimate before each point. Should calibration be a + documented manual step, or something the client can do itself? +- §4.5 introduces provisioned-power normalisation of throughput, marked tentative. Nothing here + computes it and the units are not yet settled, so it is worth tracking rather than building. diff --git a/docs/videogen_pareto_support.md b/docs/videogen_pareto_support.md deleted file mode 100644 index 9f586922..00000000 --- a/docs/videogen_pareto_support.md +++ /dev/null @@ -1,212 +0,0 @@ -# Video-generation pareto support — gaps and plan - -Status: proposal · Baseline: `4235a9c` · Scope: this repository only. - -> **Rules basis.** Requirements below are cited against `endpoints_policies` branch -> `v1.0_rules_dev` @ `6b0b1ef` (2026-09-22). That branch is **unmerged and still moving**; every -> rule citation here is as-of that commit and should be re-checked before being relied on. - -The client already drives video-generation endpoints (`videogen/`, -`examples/09_Wan22_VideoGen_Example/`), but every shipped video config is a **single measurement -point**. This records what is missing to run one as a **multi-point pareto sweep**, and itemises -the pending work in §5. - -This covers the client slice only. §6 lists what it deliberately excludes — admissibility under -the Endpoints rules, the accuracy target, submitter-side tooling, and the serving stack are all -out of scope here. - -## 1. Target shape - -Points at fixed concurrency under `ConcurrencyScheduler`, each sustaining a minimum steady-state -window, **plus one Offline point** (§5.7 requires it for every non-agentic benchmark), plus -accuracy runs — 5 for a non-agentic benchmark, one per mandatory point (§5.3). - -Two rules shape a video run in particular: - -- **§6.4** — total samples issued at a point MUST be a positive integer multiple of the dataset - size. Runs end on a whole pass, not on a clock. -- **§6.2** — minimum steady-state duration is 600 s in the Ultra Low Concurrency region (1–32) - and **1200 s** in the Low, Medium, and High Concurrency regions. - -Per-point metrics cannot be token-derived — a response is one artifact, not a token stream: - -| Token metric | Video-generation substitute | -| --- | --- | -| `system_tps` | `completed_videos / elapsed_seconds` | -| `tps_per_user` (v1.0: `1000 / tpot_p90_ms`) | `1000 / latency_p90_ms` — the same latency-reciprocal shape | -| `ttft_*` | end-to-end issue → complete-response percentile | - -The per-user metric mirrors v1.0's definition rather than dividing throughput by concurrency: v1.0 -defines per-user rate from the tail latency a user actually experiences, and the video analog -should keep that shape. - -**Percentile: P90.** This now *matches* v1.0 rather than diverging from it — §4.1 moved TTFT from -P95 (v0.7) to P90, with an explicit versioning note. The independent reason still holds: a stable -P95 needs roughly twice the completed queries of a P90, and since one video takes tens of seconds -to minutes, a P95 at concurrency 1 would need on the order of 100+ videos per point. Already -available — `DEFAULT_PERCENTILES` carries both -(`async_utils/services/metrics_aggregator/registry.py:411-423`). - -**Offline point.** `max_throughput` is the Offline load pattern, so the existing -`examples/09_Wan22_VideoGen_Example/offline_wan22*.yaml` configs are already the right shape for -it. Per §5.7 the reported concurrency is the dataset cardinality, throughput is the only metric of -interest, and latency metrics explicitly do not apply. - -## 2. Already works — no change needed - -- **System throughput.** `Report.qps` = `n_completed / duration_s`, duration from the - `tracked_duration_ns` counter (`metrics/report.py:360-361`, `:390-396`). For video that *is* - videos/second. The legacy LoadGen window is poisson-only - (`commands/benchmark/pipeline.py:222-225`), so concurrency uses the native window. -- **Per-request latency percentile.** `sample_latency_ns` → `latency` - (`metrics/report.py:50`, `:234`), P90 and P95 both in the default grid. -- **Per-user rate is already derivable.** `1000 / latency_p90_ms` needs no new measurement — the - P90 is in the emitted grid. `target_concurrency` also lands in `result_summary.json` via - `run_config`, so points are self-identifying. -- **`ConcurrencyScheduler`.** Semaphore released on every terminal result, errors included - (`load_generator/strategy.py`). -- **Phase isolation.** `max_issue_duration_ms` bounds only the performance phase - (`commands/benchmark/watchdog.py`). -- **Sample order is infinite** — never raises `StopIteration` (`load_generator/sample_order.py`). -- **Accuracy.** VBench path runs; the accuracy phase mirrors the performance load pattern. -- **Audit.** TEST04 already accepts `concurrency` - (`compliance/audit_test/output_caching_test.py`). -- **Missing tokenizer is handled gracefully**, not fatally (`commands/benchmark/execute.py`). - -## 3. Gaps - -| # | Gap | Evidence | Kind | -| --- | --- | --- | --- | -| 1 | Stop predicate is a pure OR of stop-requested / count-reached / `max_duration_ns` exceeded. No branch holds off the count stop until a wall-clock floor passes, so a phase is count-driven or capped — never floored. | `load_generator/session.py:866-880` | code | -| 2 | `min_issue_duration_ms` is a poisson count-sizer (`target_qps × duration`), not a floor. Rejected for non-poisson patterns; relaxing that validator alone is insufficient — `total_samples_to_issue()` then raises, since concurrency has no `target_qps`. | `config/schema.py:1021-1027`, `config/runtime_settings.py:249` | code | -| 3 | Steady state cannot certify a tokenless window — see §3.1. | `metrics/steady_state_diagnostics.py` | code | -| 4 | The per-user rate is not emitted as a named field; `Report` carries `qps` and `tps` only. Minor — it is one reciprocal of the already-emitted `latency` P90. | `metrics/report.py:250` | code | -| 5 | Config lock requires `target_concurrency == 1`, so it fails at every point of a curve above 1. `_resolve_model` also raises `KeyError` for a model absent from a ruleset, and the lock expects a `temperature` field the video request type lacks. | `compliance/checker.py:149`, `:115-119` | code | -| 6 | `VideoGenAdapter` mirrors the video *path string* into `response_output`, which the OSL trigger tokenizes. Harmless only because the model name resolves to no tokenizer; supplying one yields meaningless OSL/TPS. | `videogen/adapter.py` | latent | -| 7 | No sweep driver — one invocation is one point, no per-point report-dir convention, no cross-point aggregation. No shipped video config uses concurrency > 1. `publish_submission.py` publishes a single run. | — | tooling | - -Gaps 1 and 2 together mean the steady-state window a pareto point needs **cannot currently be -expressed** for a concurrency run. - -### 3.1 Why steady state cannot gate a video run - -This is the most consequential gap. Under v1.0 §4.4 the detected steady-state window is not a -diagnostic — it is **the official reporting basis**: a point's metrics are computed over that -window, with whole-run values kept only as supplementary. §4.4 names the gating metric as TPOT at -P50 and P90. A workload with no TPOT therefore has no gating metric, and so no official result — -not merely a missing diagnostic section. - -See [`steady_state_diagnostics.md`](steady_state_diagnostics.md) for the algorithm; this is only -why it does not apply. - -- **The exclusion is explicit, not accidental.** `steady_state_profile` - (`commands/benchmark/pipeline.py:96-118`) returns `None` — no collection — unless the run has a - resolved tokenizer *and* streaming, on top of `settings.steady_state.enabled` being set at all - (it is off by default). A video run fails both conditions by construction. -- The reasoning is sound for its purpose: without streaming there is no `TpotTrigger`, so every - plateau gate would see an empty TPOT series and every verdict would be `found: false`. -- `tpot_ns` derives from tokens after the first chunk; TPOT is the gated metric, TTFT is - diagnostic only. `MIN_DUR_FLOOR_S` is already 600 s but never reached. -- Outcome is a `None` verdict, not an error: `Report.steady_state` is `SteadyState | None` - (`metrics/report.py:289`, `:456`). -- There is no way for a workload to declare a different gating metric — eligibility comes from - `Profile.supported` (`metrics/steady_state_diagnostics.py:311`) and `SteadyStateConfig` - (`config/schema.py:947`, `:994`) carries no model allowlist (`:958`). - -The tension is between a defensible client design and the v1.0 rule: the client correctly declines -to emit a verdict it cannot compute, while §4.4 makes that verdict the basis of the official -result. Closing it needs a gating metric that exists for tokenless workloads, not a change on -either side alone. - -## 4. Workaround available today - -Since a run must end on a whole dataset pass (§6.4) *and* meet a duration floor (§6.2), and no -floor setting exists (gaps 1–2), size the count to the duration instead of capping the clock: - -1. Calibrate — run the point briefly to estimate sustained throughput at that concurrency. -2. Choose the smallest integer `N` where `N × dataset_size / throughput` comfortably exceeds the - region's minimum duration. -3. Issue exactly that many samples, with no clock cap: - -```yaml -settings: - runtime: - n_samples_to_issue: 1240 # N x dataset_size (e.g. 5 x 248) — an exact multiple - load_pattern: - type: "concurrency" - target_concurrency: 1 -``` - -The phase then ends on the count, at a pass boundary, having run at least the required duration — -satisfying both rules without a client patch. `n_samples_to_issue` is returned verbatim and the -sample order is infinite, so no clamping interferes. - -- **The duration is achieved, not enforced.** If throughput is lower than calibrated the point - simply runs longer, which is safe; if higher, it may undershoot the floor, so size `N` with - margin and check the achieved duration afterwards. -- Every point above concurrency 1 still fails the compliance checker (gap 5). -- Do **not** use `min_issue_duration_ms` for this (gap 2). -- Capping with `max_issue_duration_ms` instead would end the run mid-pass and violate §6.4; it - remains useful only as a runaway guard set well above the expected finish. - -## 5. Pending work - -Every code task carries tests — `AGENTS.md` sets >90% coverage and requires an explicit marker. -`videogen/` already has unit and integration suites (`tests/unit/videogen/`, -`tests/integration/videogen/`) to extend rather than start from. - -### A. Core runtime - -| # | Task | Fixes | Depends on | Size | -| --- | --- | --- | --- | --- | -| A1 | Duration floor: gate the count branch on minimum elapsed time (AND semantics), and accept the setting for `concurrency`. Must land as one change — relaxing the validator alone makes `total_samples_to_issue()` raise. Prefer a new setting name over widening `min_issue_duration_ms`. | 1, 2 | — | S | -| A2 | Emit the per-user rate (`1000 / latency_p90_ms`) as a named field rather than leaving every consumer to derive it. | 4 | — | XS | -| A3 | Modality-aware steady-state gating: gate on an alternative metric where the gated one has no samples, or report un-gated with a reason instead of returning `None`. Requires a way for a workload to declare its gating metric. | 3 | — | M | -| A4 | Make the reported latency percentile explicit (P90, per §4.1) rather than leaving consumers to pick from the grid. | — | A2 | XS | -| A5 | Remove the tokenizer footgun: stop routing a path through `response_output`, or mark the field non-tokenizable for artifact-output adapters. | 6 | — | S | - -### B. Compliance and validation - -| # | Task | Fixes | Depends on | Size | -| --- | --- | --- | --- | --- | -| B1 | Validate multi-point curves: config lock currently requires `target_concurrency == 1`, and expects a `temperature` field artifact adapters lack. Redesigns what `check_submission` covers. | 5 | A1 | L | -| B2 | Register a video benchmark as a ruleset model and dataset (`config/rulesets/mlcommons/`). Not needed to *run*, but without it there is no golden accuracy, no validity thresholds, and `_resolve_model` raises. | 5 | — | S | - -### C. Configuration and tooling - -| # | Task | Fixes | Depends on | Size | -| --- | --- | --- | --- | --- | -| C1 | Add concurrency-region example configs sized per §6.4. The shipped configs cover only `max_throughput` (the Offline point) and `concurrency: 1`. | 7 | A1 | XS | -| C2 | Decide whether a multi-point sweep driver belongs here or in submitter tooling, then build or document accordingly. | 7 | — | M | -| C3 | Multi-point publish layout — `publish_submission.py` publishes one run into one scenario directory. | 7 | C2 | M | - -### D. Documentation - -| # | Task | Depends on | -| --- | --- | --- | -| D1 | Update this document as items land; update `AGENTS.md` if any module moves or is added. | any | -| D2 | Note the modality constraint in `steady_state_diagnostics.md` once A3 settles. | A3 | - -**Critical path:** A1 → C1 → (B1, C2). A1 unblocks a legitimate multi-point run; B1 is the only -item that is unavoidable rather than convenient. - -## 6. Out of scope for this repository - -Listed so the boundary is explicit. None are tracked here. - -- Whether a video-generation benchmark is admissible under the Endpoints rules, and any rule text - it would need — a working-group question. -- The accuracy quality target itself. The scorer exists; the threshold is a benchmark-definition - decision, not a client one. -- Submitter-side result layout, aggregation, and visualisation of a finished curve. -- The model server and serving stack under test. -- Executing sweeps and producing results. - -## 7. Open questions - -- Should a sweep driver live here or in submitter tooling? Nothing here orchestrates multiple - points today. -- Sizing `N` per §6.4 needs a throughput estimate before each point. Should calibration be a - documented manual step, or something the client can do itself? -- §4.5 introduces provisioned-power normalisation of throughput, marked tentative. Nothing here - computes it, and the units are not yet settled — worth tracking, not building. From eb66fdd3f1fea12050a39ed1d9d4c625064ed981 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 13:35:48 -0700 Subject: [PATCH 03/23] docs: align title with the file name Co-Authored-By: Claude Sonnet 4.6 --- docs/pareto_sweep_support.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/pareto_sweep_support.md b/docs/pareto_sweep_support.md index d35203e8..5a0a03a2 100644 --- a/docs/pareto_sweep_support.md +++ b/docs/pareto_sweep_support.md @@ -1,4 +1,4 @@ -# Multi-point pareto sweep support: gaps and plan +# Pareto sweep support: gaps and plan Status: proposal · Baseline: `4235a9c` · Scope: this repository only. From 7e898b945c914571577f8e70317bec16da26f8d9 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 13:40:18 -0700 Subject: [PATCH 04/23] docs: restore the ruleset registration task dropped in the reframe Regrouping the task tables lost 'register a video benchmark as a ruleset model and dataset', and trimmed the temperature-field clause from the config-lock gap. Both are restored, with registration in the tokenless group since text models are already registered. Co-Authored-By: Claude Sonnet 4.6 --- docs/pareto_sweep_support.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/pareto_sweep_support.md b/docs/pareto_sweep_support.md index 5a0a03a2..8bb82ecc 100644 --- a/docs/pareto_sweep_support.md +++ b/docs/pareto_sweep_support.md @@ -88,7 +88,7 @@ interest, and latency metrics explicitly do not apply. | --- | --- | --- | --- | --- | | 1 | Stop predicate is a pure OR of stop-requested / count-reached / `max_duration_ns` exceeded. No branch holds off the count stop until a wall-clock floor passes, so a phase is count-driven or capped, never floored. | `load_generator/session.py:866-880` | any model | code | | 2 | `min_issue_duration_ms` is a poisson count-sizer (`target_qps × duration`), not a floor. Rejected for non-poisson patterns, and relaxing that validator alone is insufficient because `total_samples_to_issue()` then raises, since concurrency has no `target_qps`. | `config/schema.py:1021-1027`, `config/runtime_settings.py:249` | any model | code | -| 3 | Config lock requires `target_concurrency == 1`, so it fails at every point of a curve above 1. `_resolve_model` also raises `KeyError` for a model absent from a ruleset. | `compliance/checker.py:149` | any model | code | +| 3 | Config lock requires `target_concurrency == 1`, so it fails at every point of a curve above 1. `_resolve_model` also raises `KeyError` for a model absent from a ruleset, and the lock expects a `temperature` field artifact adapters do not carry. | `compliance/checker.py:149`, `:115-119` | any model | code | | 4 | No sweep driver. One invocation is one point, with no per-point report-dir convention and no cross-point aggregation. `publish_submission.py` publishes a single run into a single scenario directory. | n/a | any model | tooling | | 5 | The per-user rate is not emitted as a named field; `Report` carries `qps` and `tps` only. Minor, since it is one reciprocal of the already-emitted `latency` P90. | `metrics/report.py:250` | any model | code | | 6 | Steady state cannot certify a tokenless window. See §3.1. | `metrics/steady_state_diagnostics.py` | tokenless | code | @@ -191,6 +191,7 @@ Every code task carries tests. `AGENTS.md` sets >90% coverage and requires an ex | C1 | Modality-aware steady-state gating: gate on an alternative metric where the gated one has no samples (end-to-end latency exists for every workload), or report un-gated with a reason instead of returning `None`. Requires a way for a workload to declare its gating metric. | 6 | n/a | M | | C2 | Remove the tokenizer footgun: stop routing a path through `response_output`, or mark the field non-tokenizable for artifact-output adapters. | 7 | n/a | S | | C3 | Add concurrency-region example configs sized per §6.4. The shipped video configs cover only `max_throughput` (the Offline point) and `concurrency: 1`. | 4 | A1 | XS | +| C4 | Register the benchmark as a ruleset model and dataset (`config/rulesets/mlcommons/`). Not needed to *run*, but without it there is no golden accuracy, no validity thresholds, and `_resolve_model` raises. | 3 | n/a | S | ### D. Documentation From 0b2f9ebaed1cb1f15ea3f76f200777a608382ca2 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 14:01:23 -0700 Subject: [PATCH 05/23] docs: retitle and rescope to the text-to-video benchmark Renames docs/pareto_sweep_support.md to docs/t2v_benchmark_gaps.md and leads with the workload rather than the mechanism, since the document exists to get a text-to-video benchmark running. The gap analysis is unchanged. The Scope column stays, because which gaps also block other models is useful to a reader deciding what to fix first: two gaps are specific to a tokenless workload, and the other five block a multi-point curve for any model. Co-Authored-By: Claude Sonnet 4.6 --- ...sweep_support.md => t2v_benchmark_gaps.md} | 28 ++++++++++--------- 1 file changed, 15 insertions(+), 13 deletions(-) rename docs/{pareto_sweep_support.md => t2v_benchmark_gaps.md} (91%) diff --git a/docs/pareto_sweep_support.md b/docs/t2v_benchmark_gaps.md similarity index 91% rename from docs/pareto_sweep_support.md rename to docs/t2v_benchmark_gaps.md index 8bb82ecc..2f831b29 100644 --- a/docs/pareto_sweep_support.md +++ b/docs/t2v_benchmark_gaps.md @@ -1,4 +1,4 @@ -# Pareto sweep support: gaps and plan +# Text-to-video benchmark: closing the client gaps Status: proposal · Baseline: `4235a9c` · Scope: this repository only. @@ -6,19 +6,21 @@ Status: proposal · Baseline: `4235a9c` · Scope: this repository only. > `v1.0_rules_dev` @ `6b0b1ef` (2026-09-22). That branch is **unmerged and still moving**; every > rule citation here is as-of that commit and should be re-checked before being relied on. -An Endpoints submission is a **pareto curve**: several measurement points at different -concurrencies, each sustaining a minimum steady-state window, plus an Offline point and accuracy -runs. The client measures **one point per invocation**. Everything needed to get from one point to -a curve, and to verify that the points are valid, is either missing or only reachable by working -around a setting that does something else. +Running a text-to-video workload as an Endpoints submission needs a **pareto curve**: several +measurement points at different concurrencies, each sustaining a minimum steady-state window, plus +an Offline point and accuracy runs. Two things stand in the way. The client measures **one point +per invocation**, and a video response is a **single artifact rather than a token stream**, which +the metric and steady-state machinery assumes throughout. -Five of the seven gaps in §3 apply to **any** model. Existing text-model curves were produced by -running each concurrency separately and stitching the results together downstream, which is the -workaround for exactly these gaps. +This records the distance between what the client does today and what a text-to-video submission +needs, and itemises the work in §5. The adapter itself (`videogen/`) and the example workload +(`examples/09_Wan22_VideoGen_Example/`) already exist; what is missing is everything around them. -**Video generation is the motivating case**, not the scope. It was the workload that surfaced -this, and it adds two gaps of its own (§3.1, §3.2) because its responses are single artifacts -rather than token streams. Those are marked `tokenless` in the gap table. +Two of the seven gaps are specific to a tokenless workload (§3.1, §3.2). The other five block a +multi-point curve for **any** model and are marked `any model` in the gap table, so fixing them +unblocks video and every other benchmark at once. Existing text-model curves are produced by +running each concurrency separately and stitching the results downstream, which is the workaround +for exactly those five. This covers the client slice only. §6 lists what it deliberately excludes. @@ -37,7 +39,7 @@ Two rules govern how a single point must run: Neither is expressible today for a concurrency-scheduled run: see gaps 1 and 2. -### 1.1 Metrics for a tokenless workload +### 1.1 Metrics for video generation For video generation the per-point metrics cannot be token-derived, because a response is one artifact rather than a token stream: From 177e0d1c9914bdf3b62b5f69d2647d6679ccc8ed Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 14:04:53 -0700 Subject: [PATCH 06/23] docs: add a category-level summary of what is missing Opens with the seven categories and whether each is tokenless-specific or blocks any model, so the shape of the work is visible before the detail. Per-gap evidence stays in section 3 and the itemised tasks in section 5. Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index 2f831b29..7ae18c25 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -24,6 +24,22 @@ for exactly those five. This covers the client slice only. §6 lists what it deliberately excludes. +## 0. What is missing + +| Category | Missing | Scope | +| --- | --- | --- | +| Run control | A duration floor, so a point can meet a minimum steady-state window | any model | +| Validation | Checking a curve rather than a single single-stream point | any model | +| Orchestration | A sweep driver, and a publish layout for multiple points | any model | +| Metrics | The per-user rate as a named field | any model | +| Tokenless support | Steady-state gating without TPOT, and artifact-safe responses | tokenless | +| Registration | A ruleset model and dataset entry for the benchmark | tokenless | +| Examples | Concurrency-region configs sized to whole dataset passes | tokenless | + +Nothing in the serving path is missing: the adapter and the example workload already run. The gaps +are in measuring a curve, proving it valid, and describing the benchmark to the tooling. Each +category is broken out in §3, with the work itemised in §5. + ## 1. Target shape Per v1.0 §5.3 and §5.7, a non-agentic submission is four mandatory concurrency points (Ultra Low From 09d5e569ecfd62d5c77acdf9a6f6e4e0f16ebbee Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 14:11:18 -0700 Subject: [PATCH 07/23] docs: name the missing categories consistently Naming, Missing metrics, Duration support, Multi-point support, Validation support, Tokenless support, Example configs. Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index 7ae18c25..cfe23dd5 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -28,13 +28,13 @@ This covers the client slice only. §6 lists what it deliberately excludes. | Category | Missing | Scope | | --- | --- | --- | -| Run control | A duration floor, so a point can meet a minimum steady-state window | any model | -| Validation | Checking a curve rather than a single single-stream point | any model | -| Orchestration | A sweep driver, and a publish layout for multiple points | any model | -| Metrics | The per-user rate as a named field | any model | +| Naming | A ruleset model and dataset entry, so the tooling recognises the benchmark at all | tokenless | +| Missing metrics | The per-user rate as a named field | any model | +| Duration support | A floor, so a point can meet a minimum steady-state window | any model | +| Multi-point support | A sweep driver, and a publish layout for more than one point | any model | +| Validation support | Checking a curve rather than a single single-stream point | any model | | Tokenless support | Steady-state gating without TPOT, and artifact-safe responses | tokenless | -| Registration | A ruleset model and dataset entry for the benchmark | tokenless | -| Examples | Concurrency-region configs sized to whole dataset passes | tokenless | +| Example configs | Concurrency-region configs sized to whole dataset passes | tokenless | Nothing in the serving path is missing: the adapter and the example workload already run. The gaps are in measuring a curve, proving it valid, and describing the benchmark to the tooling. Each From 55bf1f00742f380844e60e2688014e7ec4677bff Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 14:13:49 -0700 Subject: [PATCH 08/23] docs: drop the Scope column from the summary table Which gaps also affect other models is still recorded per gap in the section 3 table; the opening summary does not need it. Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index cfe23dd5..4f4b1591 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -26,15 +26,15 @@ This covers the client slice only. §6 lists what it deliberately excludes. ## 0. What is missing -| Category | Missing | Scope | -| --- | --- | --- | -| Naming | A ruleset model and dataset entry, so the tooling recognises the benchmark at all | tokenless | -| Missing metrics | The per-user rate as a named field | any model | -| Duration support | A floor, so a point can meet a minimum steady-state window | any model | -| Multi-point support | A sweep driver, and a publish layout for more than one point | any model | -| Validation support | Checking a curve rather than a single single-stream point | any model | -| Tokenless support | Steady-state gating without TPOT, and artifact-safe responses | tokenless | -| Example configs | Concurrency-region configs sized to whole dataset passes | tokenless | +| Category | Missing | +| --- | --- | +| Naming | A ruleset model and dataset entry, so the tooling recognises the benchmark at all | +| Missing metrics | The per-user rate as a named field | +| Duration support | A floor, so a point can meet a minimum steady-state window | +| Multi-point support | A sweep driver, and a publish layout for more than one point | +| Validation support | Checking a curve rather than a single single-stream point | +| Tokenless support | Steady-state gating without TPOT, and artifact-safe responses | +| Example configs | Concurrency-region configs sized to whole dataset passes | Nothing in the serving path is missing: the adapter and the example workload already run. The gaps are in measuring a curve, proving it valid, and describing the benchmark to the tooling. Each From 8f7d2322fcd9b435ffe9f9c9a9e9f9f4502ce184 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 14:17:12 -0700 Subject: [PATCH 09/23] docs: order sections 3 and 5 by the section 0 categories, link them Section 0 did not match the order of anything that followed. Gaps and tasks are now both ordered by the same seven categories, and each category links to its task group. Splits the old config-lock gap in two, since it conflated Naming (_resolve_model raising for an unregistered model) with Validation (the single-stream config lock). Eight gaps now, renumbered, with the references in sections 1 and 4 updated to match. Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 112 ++++++++++++++++++++++--------------- 1 file changed, 68 insertions(+), 44 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index 4f4b1591..ddae95a1 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -16,7 +16,8 @@ This records the distance between what the client does today and what a text-to- needs, and itemises the work in §5. The adapter itself (`videogen/`) and the example workload (`examples/09_Wan22_VideoGen_Example/`) already exist; what is missing is everything around them. -Two of the seven gaps are specific to a tokenless workload (§3.1, §3.2). The other five block a +Three of the eight gaps are specific to a tokenless workload, two of them detailed in §3.1 and +§3.2. The other five block a multi-point curve for **any** model and are marked `any model` in the gap table, so fixing them unblocks video and every other benchmark at once. Existing text-model curves are produced by running each concurrency separately and stitching the results downstream, which is the workaround @@ -28,17 +29,17 @@ This covers the client slice only. §6 lists what it deliberately excludes. | Category | Missing | | --- | --- | -| Naming | A ruleset model and dataset entry, so the tooling recognises the benchmark at all | -| Missing metrics | The per-user rate as a named field | -| Duration support | A floor, so a point can meet a minimum steady-state window | -| Multi-point support | A sweep driver, and a publish layout for more than one point | -| Validation support | Checking a curve rather than a single single-stream point | -| Tokenless support | Steady-state gating without TPOT, and artifact-safe responses | -| Example configs | Concurrency-region configs sized to whole dataset passes | +| [Naming](#a-naming) | A ruleset model and dataset entry, so the tooling recognises the benchmark at all | +| [Missing metrics](#b-missing-metrics) | The per-user rate as a named field | +| [Duration support](#c-duration-support) | A floor, so a point can meet a minimum steady-state window | +| [Multi-point support](#d-multi-point-support) | A sweep driver, and a publish layout for more than one point | +| [Validation support](#e-validation-support) | Checking a curve rather than a single single-stream point | +| [Tokenless support](#f-tokenless-support) | Steady-state gating without TPOT, and artifact-safe responses | +| [Example configs](#g-example-configs) | Concurrency-region configs sized to whole dataset passes | Nothing in the serving path is missing: the adapter and the example workload already run. The gaps -are in measuring a curve, proving it valid, and describing the benchmark to the tooling. Each -category is broken out in §3, with the work itemised in §5. +are in measuring a curve, proving it valid, and describing the benchmark to the tooling. §3 and §5 +follow the same order as this table. ## 1. Target shape @@ -53,7 +54,7 @@ Two rules govern how a single point must run: - **§6.2**: minimum steady-state duration is 600 s in the Ultra Low Concurrency region and **1200 s** in the Low, Medium, and High regions. -Neither is expressible today for a concurrency-scheduled run: see gaps 1 and 2. +Neither is expressible today for a concurrency-scheduled run: see gaps 3 and 4. ### 1.1 Metrics for video generation @@ -100,20 +101,22 @@ interest, and latency metrics explicitly do not apply. ## 3. Gaps -`Scope` distinguishes gaps that block any model from those specific to a tokenless workload. +Ordered by the categories in §0. `Scope` distinguishes gaps that block any model from those +specific to a tokenless workload. | # | Gap | Evidence | Scope | Kind | | --- | --- | --- | --- | --- | -| 1 | Stop predicate is a pure OR of stop-requested / count-reached / `max_duration_ns` exceeded. No branch holds off the count stop until a wall-clock floor passes, so a phase is count-driven or capped, never floored. | `load_generator/session.py:866-880` | any model | code | -| 2 | `min_issue_duration_ms` is a poisson count-sizer (`target_qps × duration`), not a floor. Rejected for non-poisson patterns, and relaxing that validator alone is insufficient because `total_samples_to_issue()` then raises, since concurrency has no `target_qps`. | `config/schema.py:1021-1027`, `config/runtime_settings.py:249` | any model | code | -| 3 | Config lock requires `target_concurrency == 1`, so it fails at every point of a curve above 1. `_resolve_model` also raises `KeyError` for a model absent from a ruleset, and the lock expects a `temperature` field artifact adapters do not carry. | `compliance/checker.py:149`, `:115-119` | any model | code | -| 4 | No sweep driver. One invocation is one point, with no per-point report-dir convention and no cross-point aggregation. `publish_submission.py` publishes a single run into a single scenario directory. | n/a | any model | tooling | -| 5 | The per-user rate is not emitted as a named field; `Report` carries `qps` and `tps` only. Minor, since it is one reciprocal of the already-emitted `latency` P90. | `metrics/report.py:250` | any model | code | -| 6 | Steady state cannot certify a tokenless window. See §3.1. | `metrics/steady_state_diagnostics.py` | tokenless | code | -| 7 | `VideoGenAdapter` mirrors the video *path string* into `response_output`, which the OSL trigger tokenizes. Harmless only because the model name resolves to no tokenizer; supplying one yields meaningless OSL and TPS. See §3.2. | `videogen/adapter.py` | tokenless | latent | - -Gaps 1 and 2 together mean the steady-state window a pareto point needs **cannot currently be -expressed** for a concurrency run, for any model. Gap 3 means that even a correctly measured curve +| 1 | No ruleset entry exists for the benchmark, so `_resolve_model` raises `KeyError` before any other check runs. Without one there is also no golden accuracy and no validity thresholds. | `compliance/checker.py:156-164` | tokenless | code | +| 2 | The per-user rate is not emitted as a named field; `Report` carries `qps` and `tps` only. Minor, since it is one reciprocal of the already-emitted `latency` P90. | `metrics/report.py:250` | any model | code | +| 3 | Stop predicate is a pure OR of stop-requested / count-reached / `max_duration_ns` exceeded. No branch holds off the count stop until a wall-clock floor passes, so a phase is count-driven or capped, never floored. | `load_generator/session.py:866-880` | any model | code | +| 4 | `min_issue_duration_ms` is a poisson count-sizer (`target_qps × duration`), not a floor. Rejected for non-poisson patterns, and relaxing that validator alone is insufficient because `total_samples_to_issue()` then raises, since concurrency has no `target_qps`. | `config/schema.py:1021-1027`, `config/runtime_settings.py:249` | any model | code | +| 5 | No sweep driver. One invocation is one point, with no per-point report-dir convention and no cross-point aggregation. `publish_submission.py` publishes a single run into a single scenario directory. | n/a | any model | tooling | +| 6 | Config lock requires `target_concurrency == 1`, so it fails at every point of a curve above 1, and expects a `temperature` field artifact adapters do not carry. | `compliance/checker.py:149`, `:115-119` | any model | code | +| 7 | Steady state cannot certify a tokenless window. See §3.1. | `metrics/steady_state_diagnostics.py` | tokenless | code | +| 8 | `VideoGenAdapter` mirrors the video *path string* into `response_output`, which the OSL trigger tokenizes. Harmless only because the model name resolves to no tokenizer; supplying one yields meaningless OSL and TPS. See §3.2. | `videogen/adapter.py` | tokenless | latent | + +Gaps 3 and 4 together mean the steady-state window a pareto point needs **cannot currently be +expressed** for a concurrency run, for any model. Gap 6 means that even a correctly measured curve cannot be validated by this repository's checker. ### 3.1 Why steady state cannot gate a tokenless run @@ -154,7 +157,7 @@ produce OSL and TPS figures computed from a filesystem path. ## 4. Workaround available today Since a point must end on a whole dataset pass (§6.4) *and* meet a duration floor (§6.2), and no -floor setting exists (gaps 1 and 2), size the count to the duration instead of capping the clock: +floor setting exists (gaps 3 and 4), size the count to the duration instead of capping the clock: 1. Calibrate. Run the point briefly to estimate sustained throughput at that concurrency. 2. Choose the smallest integer `N` where `N × dataset_size / throughput` comfortably exceeds the @@ -177,49 +180,70 @@ sample order is infinite, so nothing clamps it. - **The duration is achieved, not enforced.** If throughput is lower than calibrated the point simply runs longer, which is safe. If higher, it may undershoot the floor, so size `N` with margin and check the achieved duration afterwards. -- Every point above concurrency 1 still fails the compliance checker (gap 3). -- Do **not** use `min_issue_duration_ms` for this (gap 2). +- Every point above concurrency 1 still fails the compliance checker (gap 6). +- Do **not** use `min_issue_duration_ms` for this (gap 4). - Capping with `max_issue_duration_ms` instead would end the run mid-pass and violate §6.4. It remains useful only as a runaway guard set well above the expected finish. ## 5. Pending work -Every code task carries tests. `AGENTS.md` sets >90% coverage and requires an explicit marker. +Grouped by the categories in §0. Every code task carries tests; `AGENTS.md` sets >90% coverage and +requires an explicit marker. -### A. Core runtime (any model) +### A. Naming | # | Task | Fixes | Depends on | Size | | --- | --- | --- | --- | --- | -| A1 | Duration floor: gate the count branch on minimum elapsed time (AND semantics), and accept the setting for `concurrency`. Must land as one change, because relaxing the validator alone makes `total_samples_to_issue()` raise. Prefer a new setting name over widening `min_issue_duration_ms`. | 1, 2 | n/a | S | -| A2 | Emit the per-user rate as a named field rather than leaving every consumer to derive it. | 5 | n/a | XS | -| A3 | Make the reported latency percentile explicit (P90, per §4.1). | n/a | A2 | XS | +| A1 | Register the benchmark as a ruleset model and dataset (`config/rulesets/mlcommons/`). Not needed to *run*, but without it there is no golden accuracy, no validity thresholds, and `_resolve_model` raises. | 1 | n/a | S | -### B. Compliance and tooling (any model) +### B. Missing metrics | # | Task | Fixes | Depends on | Size | | --- | --- | --- | --- | --- | -| B1 | Validate multi-point curves. Config lock requires `target_concurrency == 1` and expects a `temperature` field artifact adapters lack. Redesigns what `check_submission` covers. | 3 | A1 | L | -| B2 | Decide whether a multi-point sweep driver belongs here or in submitter tooling, then build or document accordingly. | 4 | n/a | M | -| B3 | Multi-point publish layout for `publish_submission.py`. | 4 | B2 | M | +| B1 | Emit the per-user rate as a named field rather than leaving every consumer to derive it. | 2 | n/a | XS | +| B2 | Make the reported latency percentile explicit (P90, per §4.1). | n/a | B1 | XS | -### C. Tokenless workloads +### C. Duration support | # | Task | Fixes | Depends on | Size | | --- | --- | --- | --- | --- | -| C1 | Modality-aware steady-state gating: gate on an alternative metric where the gated one has no samples (end-to-end latency exists for every workload), or report un-gated with a reason instead of returning `None`. Requires a way for a workload to declare its gating metric. | 6 | n/a | M | -| C2 | Remove the tokenizer footgun: stop routing a path through `response_output`, or mark the field non-tokenizable for artifact-output adapters. | 7 | n/a | S | -| C3 | Add concurrency-region example configs sized per §6.4. The shipped video configs cover only `max_throughput` (the Offline point) and `concurrency: 1`. | 4 | A1 | XS | -| C4 | Register the benchmark as a ruleset model and dataset (`config/rulesets/mlcommons/`). Not needed to *run*, but without it there is no golden accuracy, no validity thresholds, and `_resolve_model` raises. | 3 | n/a | S | +| C1 | Duration floor: gate the count branch on minimum elapsed time (AND semantics), and accept the setting for `concurrency`. Must land as one change, because relaxing the validator alone makes `total_samples_to_issue()` raise. Prefer a new setting name over widening `min_issue_duration_ms`. | 3, 4 | n/a | S | -### D. Documentation +### D. Multi-point support + +| # | Task | Fixes | Depends on | Size | +| --- | --- | --- | --- | --- | +| D1 | Decide whether a multi-point sweep driver belongs here or in submitter tooling, then build or document accordingly. | 5 | n/a | M | +| D2 | Multi-point publish layout for `publish_submission.py`. | 5 | D1 | M | + +### E. Validation support + +| # | Task | Fixes | Depends on | Size | +| --- | --- | --- | --- | --- | +| E1 | Validate multi-point curves. Redesigns what `check_submission` covers, since the config lock is built around a single single-stream point. | 6 | C1 | L | + +### F. Tokenless support + +| # | Task | Fixes | Depends on | Size | +| --- | --- | --- | --- | --- | +| F1 | Modality-aware steady-state gating: gate on an alternative metric where the gated one has no samples (end-to-end latency exists for every workload), or report un-gated with a reason instead of returning `None`. Requires a way for a workload to declare its gating metric. | 7 | n/a | M | +| F2 | Remove the tokenizer footgun: stop routing a path through `response_output`, or mark the field non-tokenizable for artifact-output adapters. | 8 | n/a | S | + +### G. Example configs + +| # | Task | Fixes | Depends on | Size | +| --- | --- | --- | --- | --- | +| G1 | Add concurrency-region example configs sized per §6.4. The shipped video configs cover only `max_throughput` (the Offline point) and `concurrency: 1`. | 5 | C1 | XS | + +### H. Documentation | # | Task | Depends on | | --- | --- | --- | -| D1 | Update this document as items land; update `AGENTS.md` if any module moves or is added. | any | -| D2 | Note the modality constraint in `steady_state_diagnostics.md` once C1 settles. | C1 | +| H1 | Update this document as items land; update `AGENTS.md` if any module moves or is added. | any | +| H2 | Note the modality constraint in `steady_state_diagnostics.md` once F1 settles. | F1 | -**Critical path:** A1 → C3 → (B1, B2). A1 unblocks a legitimate multi-point run for every model. -B1 is the only item that is unavoidable rather than convenient. +**Critical path:** C1 → G1 → (E1, D1). C1 unblocks a legitimate multi-point run for every model. +E1 is the only item that is unavoidable rather than convenient. ## 6. Out of scope for this repository From 0fd0fdcd69a142cc2367408f2e68133be042af5c Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 14:17:38 -0700 Subject: [PATCH 10/23] docs: rewrap the intro paragraph Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index ddae95a1..ce77c0cc 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -17,11 +17,10 @@ needs, and itemises the work in §5. The adapter itself (`videogen/`) and the ex (`examples/09_Wan22_VideoGen_Example/`) already exist; what is missing is everything around them. Three of the eight gaps are specific to a tokenless workload, two of them detailed in §3.1 and -§3.2. The other five block a -multi-point curve for **any** model and are marked `any model` in the gap table, so fixing them -unblocks video and every other benchmark at once. Existing text-model curves are produced by -running each concurrency separately and stitching the results downstream, which is the workaround -for exactly those five. +§3.2. The other five block a multi-point curve for **any** model and are marked `any model` in +the gap table, so fixing them unblocks video and every other benchmark at once. Existing +text-model curves are produced by running each concurrency separately and stitching the results +downstream, which is the workaround for exactly those five. This covers the client slice only. §6 lists what it deliberately excludes. From 4e97c947592ae21d90ce4c0c90d061a8899cca04 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 14:23:46 -0700 Subject: [PATCH 11/23] docs: drop the admissibility caveat from out of scope Admissibility is settled; rules text will be added. The bullet framed it as an open working-group question, which is no longer accurate. Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index ce77c0cc..baabb81f 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -246,8 +246,6 @@ E1 is the only item that is unavoidable rather than convenient. ## 6. Out of scope for this repository -- Whether a video-generation benchmark is admissible under the Endpoints rules, and any rule text - it would need. That is a working-group question. - The accuracy quality target. The scorer exists; the threshold is a benchmark-definition decision, not a client one. - Submitter-side result layout, aggregation, and visualisation of a finished curve. From 609ac88aabeab7dfaa53cc1519c2625642c6f1b1 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 14:27:46 -0700 Subject: [PATCH 12/23] docs: clarify the scope of the no-gaps-in-serving claim 'Nothing in the serving path is missing' could be read as covering the model server. It means the client side of the request path. Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index baabb81f..9982bdac 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -36,7 +36,8 @@ This covers the client slice only. §6 lists what it deliberately excludes. | [Tokenless support](#f-tokenless-support) | Steady-state gating without TPOT, and artifact-safe responses | | [Example configs](#g-example-configs) | Concurrency-region configs sized to whole dataset passes | -Nothing in the serving path is missing: the adapter and the example workload already run. The gaps +Nothing is missing on the client side of the request path: the adapter and the example workload +already run. The gaps are in measuring a curve, proving it valid, and describing the benchmark to the tooling. §3 and §5 follow the same order as this table. From 8fbd0b2b0234fff978393acfa9b27e60aa59c56c Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 14:59:41 -0700 Subject: [PATCH 13/23] docs: drop the open questions section Each item is either answered or already tracked as a task: calibration is answered by measured per-video latency, the sweep-driver question is task D1, and power normalisation is tracked outside this repository. Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index 9982bdac..942980be 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -252,12 +252,3 @@ E1 is the only item that is unavoidable rather than convenient. - Submitter-side result layout, aggregation, and visualisation of a finished curve. - The model server and serving stack under test. - Executing sweeps and producing results. - -## 7. Open questions - -- Should a sweep driver live here or in submitter tooling? Nothing here orchestrates multiple - points today. -- Sizing `N` per §6.4 needs a throughput estimate before each point. Should calibration be a - documented manual step, or something the client can do itself? -- §4.5 introduces provisioned-power normalisation of throughput, marked tentative. Nothing here - computes it and the units are not yet settled, so it is worth tracking rather than building. From dc4f9342160562584b0e3bbe972f2322a34f5e42 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 15:32:23 -0700 Subject: [PATCH 14/23] docs: itemise the intro and use plainer words Four dense paragraphs become short lists: what blocks a submission, what already exists, how the eight gaps split, and the scope. Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 43 +++++++++++++++++++++++--------------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index 942980be..f6736211 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -6,23 +6,32 @@ Status: proposal · Baseline: `4235a9c` · Scope: this repository only. > `v1.0_rules_dev` @ `6b0b1ef` (2026-09-22). That branch is **unmerged and still moving**; every > rule citation here is as-of that commit and should be re-checked before being relied on. -Running a text-to-video workload as an Endpoints submission needs a **pareto curve**: several -measurement points at different concurrencies, each sustaining a minimum steady-state window, plus -an Offline point and accuracy runs. Two things stand in the way. The client measures **one point -per invocation**, and a video response is a **single artifact rather than a token stream**, which -the metric and steady-state machinery assumes throughout. - -This records the distance between what the client does today and what a text-to-video submission -needs, and itemises the work in §5. The adapter itself (`videogen/`) and the example workload -(`examples/09_Wan22_VideoGen_Example/`) already exist; what is missing is everything around them. - -Three of the eight gaps are specific to a tokenless workload, two of them detailed in §3.1 and -§3.2. The other five block a multi-point curve for **any** model and are marked `any model` in -the gap table, so fixing them unblocks video and every other benchmark at once. Existing -text-model curves are produced by running each concurrency separately and stitching the results -downstream, which is the workaround for exactly those five. - -This covers the client slice only. §6 lists what it deliberately excludes. +An Endpoints submission needs a **pareto curve**: several runs at different concurrency levels, +plus one Offline run and a set of accuracy runs. Each run has to hold steady for a minimum time. + +**Two things block that today:** + +- The client measures **one point per run**. There is no way to sweep several. +- A video reply is **one file, not a stream of tokens**. The metric and steady-state code assume + tokens everywhere. + +**What already exists:** + +- The adapter, `videogen/`. +- The example workload, `examples/09_Wan22_VideoGen_Example/`. + +Everything around them is what is missing, and §5 lists that work. + +**How the gaps split:** + +- Eight gaps in total (§3). +- **Three** only matter for a workload with no tokens. Two of those are explained in §3.1 and §3.2. +- **Five** block a multi-point curve for **any** model, marked `any model` in the gap table. +- So fixing those five helps video and every other benchmark at the same time. +- Text curves are built today by running each concurrency on its own and joining the results + afterwards. That is the workaround for those same five gaps. + +**Scope:** the client only. §6 lists what is left out. ## 0. What is missing From bd70a29ba66ddc415b92231c856915217503cca3 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 15:34:29 -0700 Subject: [PATCH 15/23] docs: shorten the blockers heading Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index f6736211..c56d1fe1 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -9,7 +9,7 @@ Status: proposal · Baseline: `4235a9c` · Scope: this repository only. An Endpoints submission needs a **pareto curve**: several runs at different concurrency levels, plus one Offline run and a set of accuracy runs. Each run has to hold steady for a minimum time. -**Two things block that today:** +**Blockers:** - The client measures **one point per run**. There is no way to sweep several. - A video reply is **one file, not a stream of tokens**. The metric and steady-state code assume From 34218a30ffb4b04f4c5d0ac02c27c1b2bfad7ede Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 15:40:45 -0700 Subject: [PATCH 16/23] docs: write counts as digits Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index c56d1fe1..f5119986 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -7,12 +7,12 @@ Status: proposal · Baseline: `4235a9c` · Scope: this repository only. > rule citation here is as-of that commit and should be re-checked before being relied on. An Endpoints submission needs a **pareto curve**: several runs at different concurrency levels, -plus one Offline run and a set of accuracy runs. Each run has to hold steady for a minimum time. +plus 1 Offline run and a set of accuracy runs. Each run has to hold steady for a minimum time. **Blockers:** -- The client measures **one point per run**. There is no way to sweep several. -- A video reply is **one file, not a stream of tokens**. The metric and steady-state code assume +- The client measures **1 point per run**. There is no way to sweep several. +- A video reply is **1 file, not a stream of tokens**. The metric and steady-state code assume tokens everywhere. **What already exists:** @@ -24,12 +24,12 @@ Everything around them is what is missing, and §5 lists that work. **How the gaps split:** -- Eight gaps in total (§3). -- **Three** only matter for a workload with no tokens. Two of those are explained in §3.1 and §3.2. -- **Five** block a multi-point curve for **any** model, marked `any model` in the gap table. -- So fixing those five helps video and every other benchmark at the same time. +- 8 gaps in total (§3). +- **3** only matter for a workload with no tokens. 2 of those are explained in §3.1 and §3.2. +- **5** block a multi-point curve for **any** model, marked `any model` in the gap table. +- So fixing those 5 helps video and every other benchmark at the same time. - Text curves are built today by running each concurrency on its own and joining the results - afterwards. That is the workaround for those same five gaps. + afterwards. That is the workaround for those same 5 gaps. **Scope:** the client only. §6 lists what is left out. @@ -40,7 +40,7 @@ Everything around them is what is missing, and §5 lists that work. | [Naming](#a-naming) | A ruleset model and dataset entry, so the tooling recognises the benchmark at all | | [Missing metrics](#b-missing-metrics) | The per-user rate as a named field | | [Duration support](#c-duration-support) | A floor, so a point can meet a minimum steady-state window | -| [Multi-point support](#d-multi-point-support) | A sweep driver, and a publish layout for more than one point | +| [Multi-point support](#d-multi-point-support) | A sweep driver, and a publish layout for more than 1 point | | [Validation support](#e-validation-support) | Checking a curve rather than a single single-stream point | | [Tokenless support](#f-tokenless-support) | Steady-state gating without TPOT, and artifact-safe responses | | [Example configs](#g-example-configs) | Concurrency-region configs sized to whole dataset passes | @@ -52,11 +52,11 @@ follow the same order as this table. ## 1. Target shape -Per v1.0 §5.3 and §5.7, a non-agentic submission is four mandatory concurrency points (Ultra Low -1-32, Low, Medium, High), three submitter's-choice points, one Offline point, and five accuracy +Per v1.0 §5.3 and §5.7, a non-agentic submission is 4 mandatory concurrency points (Ultra Low +1-32, Low, Medium, High), 3 submitter's-choice points, 1 Offline point, and 5 accuracy runs. -Two rules govern how a single point must run: +2 rules govern how a single point must run: - **§6.4**: total samples issued at a point MUST be a positive integer multiple of the dataset size. A point ends on a whole pass, not on a clock. @@ -67,7 +67,7 @@ Neither is expressible today for a concurrency-scheduled run: see gaps 3 and 4. ### 1.1 Metrics for video generation -For video generation the per-point metrics cannot be token-derived, because a response is one +For video generation the per-point metrics cannot be token-derived, because a response is 1 artifact rather than a token stream: | Token metric | Substitute | @@ -81,7 +81,7 @@ because v1.0 derives per-user rate from the tail latency a user actually experie **Percentile: P90.** This *matches* v1.0 rather than diverging from it, since §4.1 moved TTFT from P95 to P90 with an explicit versioning note. The independent reason also holds: a stable P95 needs -roughly twice the completed queries of a P90, and one video takes tens of seconds to minutes, so a +roughly twice the completed queries of a P90, and 1 video takes tens of seconds to minutes, so a P95 at concurrency 1 would need on the order of 100+ videos per point. `DEFAULT_PERCENTILES` already carries both (`async_utils/services/metrics_aggregator/registry.py:411-423`). @@ -116,10 +116,10 @@ specific to a tokenless workload. | # | Gap | Evidence | Scope | Kind | | --- | --- | --- | --- | --- | | 1 | No ruleset entry exists for the benchmark, so `_resolve_model` raises `KeyError` before any other check runs. Without one there is also no golden accuracy and no validity thresholds. | `compliance/checker.py:156-164` | tokenless | code | -| 2 | The per-user rate is not emitted as a named field; `Report` carries `qps` and `tps` only. Minor, since it is one reciprocal of the already-emitted `latency` P90. | `metrics/report.py:250` | any model | code | +| 2 | The per-user rate is not emitted as a named field; `Report` carries `qps` and `tps` only. Minor, since it is the reciprocal of the already-emitted `latency` P90. | `metrics/report.py:250` | any model | code | | 3 | Stop predicate is a pure OR of stop-requested / count-reached / `max_duration_ns` exceeded. No branch holds off the count stop until a wall-clock floor passes, so a phase is count-driven or capped, never floored. | `load_generator/session.py:866-880` | any model | code | | 4 | `min_issue_duration_ms` is a poisson count-sizer (`target_qps × duration`), not a floor. Rejected for non-poisson patterns, and relaxing that validator alone is insufficient because `total_samples_to_issue()` then raises, since concurrency has no `target_qps`. | `config/schema.py:1021-1027`, `config/runtime_settings.py:249` | any model | code | -| 5 | No sweep driver. One invocation is one point, with no per-point report-dir convention and no cross-point aggregation. `publish_submission.py` publishes a single run into a single scenario directory. | n/a | any model | tooling | +| 5 | No sweep driver. 1 invocation is 1 point, with no per-point report-dir convention and no cross-point aggregation. `publish_submission.py` publishes a single run into a single scenario directory. | n/a | any model | tooling | | 6 | Config lock requires `target_concurrency == 1`, so it fails at every point of a curve above 1, and expects a `temperature` field artifact adapters do not carry. | `compliance/checker.py:149`, `:115-119` | any model | code | | 7 | Steady state cannot certify a tokenless window. See §3.1. | `metrics/steady_state_diagnostics.py` | tokenless | code | | 8 | `VideoGenAdapter` mirrors the video *path string* into `response_output`, which the OSL trigger tokenizes. Harmless only because the model name resolves to no tokenizer; supplying one yields meaningless OSL and TPS. See §3.2. | `videogen/adapter.py` | tokenless | latent | From fe989055448cc538820412fb126ca1cb6b9c3ab1 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 15:54:36 -0700 Subject: [PATCH 17/23] docs: scope to what T2V actually needs The document listed 8 gaps, 5 of which do not block a T2V benchmark. They surfaced while working out how to run a sweep, which is not the same as blocking it: text curves already work around all 5, and the same workaround serves video. Leaves 3 gaps plus example configs. Only one is a hard blocker: steady state cannot certify a window without tokens, and v1.0 4.4 makes that verdict the official reporting basis. The 5 become a short context note in 3.3, kept so the section 4 recipe still makes sense. One of them was also wrong to list at all: compliance/checker.py is scoped to Edge-Agentic submissions and reachable only through a standalone script, so it is not the validator for an Endpoints pareto submission and never was on this path. Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 124 +++++++++++++++---------------------- 1 file changed, 50 insertions(+), 74 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index f5119986..0fb33c49 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -22,14 +22,12 @@ plus 1 Offline run and a set of accuracy runs. Each run has to hold steady for a Everything around them is what is missing, and §5 lists that work. -**How the gaps split:** +**The gaps:** -- 8 gaps in total (§3). -- **3** only matter for a workload with no tokens. 2 of those are explained in §3.1 and §3.2. -- **5** block a multi-point curve for **any** model, marked `any model` in the gap table. -- So fixing those 5 helps video and every other benchmark at the same time. -- Text curves are built today by running each concurrency on its own and joining the results - afterwards. That is the workaround for those same 5 gaps. +- 3 gaps (§3), plus example configs. +- Only 1 is a hard blocker: steady state cannot certify a window without tokens (§3.1). +- The client also has limits that affect every model, not just video. They do **not** block a + T2V benchmark, because the same workaround text curves already use works here too. See §3.3. **Scope:** the client only. §6 lists what is left out. @@ -38,17 +36,11 @@ Everything around them is what is missing, and §5 lists that work. | Category | Missing | | --- | --- | | [Naming](#a-naming) | A ruleset model and dataset entry, so the tooling recognises the benchmark at all | -| [Missing metrics](#b-missing-metrics) | The per-user rate as a named field | -| [Duration support](#c-duration-support) | A floor, so a point can meet a minimum steady-state window | -| [Multi-point support](#d-multi-point-support) | A sweep driver, and a publish layout for more than 1 point | -| [Validation support](#e-validation-support) | Checking a curve rather than a single single-stream point | -| [Tokenless support](#f-tokenless-support) | Steady-state gating without TPOT, and artifact-safe responses | -| [Example configs](#g-example-configs) | Concurrency-region configs sized to whole dataset passes | - -Nothing is missing on the client side of the request path: the adapter and the example workload -already run. The gaps -are in measuring a curve, proving it valid, and describing the benchmark to the tooling. §3 and §5 -follow the same order as this table. +| [Tokenless support](#b-tokenless-support) | Steady-state gating without TPOT, and artifact-safe responses | +| [Example configs](#c-example-configs) | Concurrency-region configs sized to whole dataset passes | + +The adapter and the example workload already run. What is missing is the description of the +benchmark to the tooling, and the handling of a reply that carries no tokens. ## 1. Target shape @@ -63,7 +55,7 @@ runs. - **§6.2**: minimum steady-state duration is 600 s in the Ultra Low Concurrency region and **1200 s** in the Low, Medium, and High regions. -Neither is expressible today for a concurrency-scheduled run: see gaps 3 and 4. +Neither is expressible today for a concurrency-scheduled run: see §3.3. ### 1.1 Metrics for video generation @@ -110,23 +102,13 @@ interest, and latency metrics explicitly do not apply. ## 3. Gaps -Ordered by the categories in §0. `Scope` distinguishes gaps that block any model from those -specific to a tokenless workload. +Ordered by the categories in §0. -| # | Gap | Evidence | Scope | Kind | -| --- | --- | --- | --- | --- | -| 1 | No ruleset entry exists for the benchmark, so `_resolve_model` raises `KeyError` before any other check runs. Without one there is also no golden accuracy and no validity thresholds. | `compliance/checker.py:156-164` | tokenless | code | -| 2 | The per-user rate is not emitted as a named field; `Report` carries `qps` and `tps` only. Minor, since it is the reciprocal of the already-emitted `latency` P90. | `metrics/report.py:250` | any model | code | -| 3 | Stop predicate is a pure OR of stop-requested / count-reached / `max_duration_ns` exceeded. No branch holds off the count stop until a wall-clock floor passes, so a phase is count-driven or capped, never floored. | `load_generator/session.py:866-880` | any model | code | -| 4 | `min_issue_duration_ms` is a poisson count-sizer (`target_qps × duration`), not a floor. Rejected for non-poisson patterns, and relaxing that validator alone is insufficient because `total_samples_to_issue()` then raises, since concurrency has no `target_qps`. | `config/schema.py:1021-1027`, `config/runtime_settings.py:249` | any model | code | -| 5 | No sweep driver. 1 invocation is 1 point, with no per-point report-dir convention and no cross-point aggregation. `publish_submission.py` publishes a single run into a single scenario directory. | n/a | any model | tooling | -| 6 | Config lock requires `target_concurrency == 1`, so it fails at every point of a curve above 1, and expects a `temperature` field artifact adapters do not carry. | `compliance/checker.py:149`, `:115-119` | any model | code | -| 7 | Steady state cannot certify a tokenless window. See §3.1. | `metrics/steady_state_diagnostics.py` | tokenless | code | -| 8 | `VideoGenAdapter` mirrors the video *path string* into `response_output`, which the OSL trigger tokenizes. Harmless only because the model name resolves to no tokenizer; supplying one yields meaningless OSL and TPS. See §3.2. | `videogen/adapter.py` | tokenless | latent | - -Gaps 3 and 4 together mean the steady-state window a pareto point needs **cannot currently be -expressed** for a concurrency run, for any model. Gap 6 means that even a correctly measured curve -cannot be validated by this repository's checker. +| # | Gap | Evidence | Kind | +| --- | --- | --- | --- | +| 1 | No ruleset entry exists for the benchmark, so `_resolve_model` raises `KeyError` before any other check runs. Without one there is also no golden accuracy and no validity thresholds. | `compliance/checker.py:156-164` | code | +| 2 | Steady state cannot certify a window with no tokens. See §3.1. **This is the only hard blocker.** | `metrics/steady_state_diagnostics.py` | code | +| 3 | `VideoGenAdapter` mirrors the video *path string* into `response_output`, which the OSL trigger tokenizes. Harmless only because the model name resolves to no tokenizer; supplying one yields meaningless OSL and TPS. See §3.2. | `videogen/adapter.py` | latent | ### 3.1 Why steady state cannot gate a tokenless run @@ -163,10 +145,31 @@ output-sequence-length trigger tokenizes. Nothing breaks today only because the not resolve to a tokenizer, so token metrics stay disabled. Supplying `--tokenizer` would silently produce OSL and TPS figures computed from a filesystem path. +### 3.3 Client limits that are not T2V problems + +Working out how to run a sweep surfaced several limits in the client. None of them block a T2V +benchmark, and they are recorded here only so the §4 recipe makes sense. + +- There is no minimum-duration floor in the run loop, and `min_issue_duration_ms` is a poisson + sample-count sizer rather than a floor, rejected for `concurrency` + (`load_generator/session.py:866-880`, `config/schema.py:1021-1027`, + `config/runtime_settings.py:249`). The §4 recipe meets the floor without either. +- One invocation measures one point. There is no sweep driver and no multi-point publish layout. + Existing text curves are built by running each concurrency on its own and joining the results + afterwards; the same applies here. +- The per-user rate is not emitted as a named field, but it is the reciprocal of a value already + reported (`metrics/report.py:250`), so it is one division in post-processing. +- `compliance/checker.py` requires `target_concurrency == 1`, but it is scoped to Edge-Agentic + (BFCL v4) submissions and is reachable only through `scripts/check_compliance.py`. It is not the + validator for an Endpoints pareto submission, so it is not on this path. + +Fixing any of these would help every benchmark, not just video. That is a separate argument and +not a prerequisite here. + ## 4. Workaround available today Since a point must end on a whole dataset pass (§6.4) *and* meet a duration floor (§6.2), and no -floor setting exists (gaps 3 and 4), size the count to the duration instead of capping the clock: +floor setting exists (§3.3), size the count to the duration instead of capping the clock: 1. Calibrate. Run the point briefly to estimate sustained throughput at that concurrency. 2. Choose the smallest integer `N` where `N × dataset_size / throughput` comfortably exceeds the @@ -189,8 +192,7 @@ sample order is infinite, so nothing clamps it. - **The duration is achieved, not enforced.** If throughput is lower than calibrated the point simply runs longer, which is safe. If higher, it may undershoot the floor, so size `N` with margin and check the achieved duration afterwards. -- Every point above concurrency 1 still fails the compliance checker (gap 6). -- Do **not** use `min_issue_duration_ms` for this (gap 4). +- Do **not** use `min_issue_duration_ms` for this (§3.3). - Capping with `max_issue_duration_ms` instead would end the run mid-pass and violate §6.4. It remains useful only as a runaway guard set well above the expected finish. @@ -205,54 +207,28 @@ requires an explicit marker. | --- | --- | --- | --- | --- | | A1 | Register the benchmark as a ruleset model and dataset (`config/rulesets/mlcommons/`). Not needed to *run*, but without it there is no golden accuracy, no validity thresholds, and `_resolve_model` raises. | 1 | n/a | S | -### B. Missing metrics - -| # | Task | Fixes | Depends on | Size | -| --- | --- | --- | --- | --- | -| B1 | Emit the per-user rate as a named field rather than leaving every consumer to derive it. | 2 | n/a | XS | -| B2 | Make the reported latency percentile explicit (P90, per §4.1). | n/a | B1 | XS | - -### C. Duration support - -| # | Task | Fixes | Depends on | Size | -| --- | --- | --- | --- | --- | -| C1 | Duration floor: gate the count branch on minimum elapsed time (AND semantics), and accept the setting for `concurrency`. Must land as one change, because relaxing the validator alone makes `total_samples_to_issue()` raise. Prefer a new setting name over widening `min_issue_duration_ms`. | 3, 4 | n/a | S | - -### D. Multi-point support - -| # | Task | Fixes | Depends on | Size | -| --- | --- | --- | --- | --- | -| D1 | Decide whether a multi-point sweep driver belongs here or in submitter tooling, then build or document accordingly. | 5 | n/a | M | -| D2 | Multi-point publish layout for `publish_submission.py`. | 5 | D1 | M | - -### E. Validation support - -| # | Task | Fixes | Depends on | Size | -| --- | --- | --- | --- | --- | -| E1 | Validate multi-point curves. Redesigns what `check_submission` covers, since the config lock is built around a single single-stream point. | 6 | C1 | L | - -### F. Tokenless support +### B. Tokenless support | # | Task | Fixes | Depends on | Size | | --- | --- | --- | --- | --- | -| F1 | Modality-aware steady-state gating: gate on an alternative metric where the gated one has no samples (end-to-end latency exists for every workload), or report un-gated with a reason instead of returning `None`. Requires a way for a workload to declare its gating metric. | 7 | n/a | M | -| F2 | Remove the tokenizer footgun: stop routing a path through `response_output`, or mark the field non-tokenizable for artifact-output adapters. | 8 | n/a | S | +| B1 | Modality-aware steady-state gating: gate on an alternative metric where the gated one has no samples (end-to-end latency exists for every workload), or report un-gated with a reason instead of returning `None`. Requires a way for a workload to declare its gating metric. | 2 | n/a | M | +| B2 | Remove the tokenizer footgun: stop routing a path through `response_output`, or mark the field non-tokenizable for artifact-output adapters. | 3 | n/a | S | -### G. Example configs +### C. Example configs | # | Task | Fixes | Depends on | Size | | --- | --- | --- | --- | --- | -| G1 | Add concurrency-region example configs sized per §6.4. The shipped video configs cover only `max_throughput` (the Offline point) and `concurrency: 1`. | 5 | C1 | XS | +| C1 | Add concurrency-region example configs sized per §6.4, using the §4 recipe. The shipped video configs cover only `max_throughput` (the Offline point) and `concurrency: 1`. | n/a | n/a | XS | -### H. Documentation +### D. Documentation | # | Task | Depends on | | --- | --- | --- | -| H1 | Update this document as items land; update `AGENTS.md` if any module moves or is added. | any | -| H2 | Note the modality constraint in `steady_state_diagnostics.md` once F1 settles. | F1 | +| D1 | Update this document as items land; update `AGENTS.md` if any module moves or is added. | any | +| D2 | Note the modality constraint in `steady_state_diagnostics.md` once B1 settles. | B1 | -**Critical path:** C1 → G1 → (E1, D1). C1 unblocks a legitimate multi-point run for every model. -E1 is the only item that is unavoidable rather than convenient. +**B1 is the critical item.** Under v1.0 §4.4 the steady-state verdict is the official reporting +basis, so without it a T2V run has no official result. A1 and C1 are small and independent. ## 6. Out of scope for this repository From 976988861e1fcbf00a645c850a254f4e70f357e6 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 15:57:23 -0700 Subject: [PATCH 18/23] docs: drop items unrelated to T2V Section 2 listed three pieces of generic client behaviour with no bearing on a video workload: the ConcurrencyScheduler semaphore, performance-phase isolation, and target_concurrency landing in the run config. The last was also stale, since the per-user rate is now derived from latency rather than from concurrency. What remains in section 2 is there for a reason: throughput and latency because they are the video metrics, the infinite sample order because the section 4 recipe depends on it, TEST04 because the shipped video configs run it, and graceful tokenizer absence because video has no tokenizer. Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index 0fb33c49..1b60de44 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -90,12 +90,7 @@ interest, and latency metrics explicitly do not apply. (`commands/benchmark/pipeline.py:222-225`), so concurrency uses the native window. - **Per-request latency percentile.** `sample_latency_ns` maps to `latency` (`metrics/report.py:50`, `:234`), with P90 and P95 both in the default grid. -- **`ConcurrencyScheduler`.** Semaphore released on every terminal result, errors included - (`load_generator/strategy.py`). -- **Phase isolation.** `max_issue_duration_ms` bounds only the performance phase - (`commands/benchmark/watchdog.py`). - **Sample order is infinite**, never raising `StopIteration` (`load_generator/sample_order.py`). -- **Point identity.** `target_concurrency` lands in `result_summary.json` via `run_config`. - **Audit.** TEST04 already accepts `concurrency` (`compliance/audit_test/output_caching_test.py`). - **Missing tokenizer is handled gracefully**, not fatally (`commands/benchmark/execute.py`). From 7db00160da0bdb86c4ab3cc044222cbbc2f68442 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Mon, 28 Sep 2026 16:25:31 -0700 Subject: [PATCH 19/23] docs: say valid result rather than verdict Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index 1b60de44..ee5e9d53 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -121,15 +121,15 @@ why it does not apply. has a resolved tokenizer *and* streaming, on top of `settings.steady_state.enabled` being set at all (it is off by default). A tokenless run fails both conditions by construction. - The reasoning is sound for its purpose: without streaming there is no `TpotTrigger`, so every - plateau gate would see an empty TPOT series and every verdict would be `found: false`. -- Outcome is a `None` verdict, not an error: `Report.steady_state` is `SteadyState | None` + plateau gate would see an empty TPOT series and nothing would ever validate. +- Outcome is a `None` result, not an error: `Report.steady_state` is `SteadyState | None` (`metrics/report.py:289`, `:456`). - There is no way for a workload to declare a different gating metric. Eligibility comes from `Profile.supported` (`metrics/steady_state_diagnostics.py:311`), and `SteadyStateConfig` (`config/schema.py:947`, `:994`) carries no model allowlist (`:958`). The tension is between a defensible client design and the v1.0 rule. The client correctly declines -to emit a verdict it cannot compute, while §4.4 makes that verdict the basis of the official +to emit a result it cannot compute, while §4.4 makes that result the basis of the official result. Closing it needs a gating metric that exists for tokenless workloads, not a change on either side alone. @@ -222,7 +222,7 @@ requires an explicit marker. | D1 | Update this document as items land; update `AGENTS.md` if any module moves or is added. | any | | D2 | Note the modality constraint in `steady_state_diagnostics.md` once B1 settles. | B1 | -**B1 is the critical item.** Under v1.0 §4.4 the steady-state verdict is the official reporting +**B1 is the critical item.** Under v1.0 §4.4 the steady-state result is the official reporting basis, so without it a T2V run has no official result. A1 and C1 are small and independent. ## 6. Out of scope for this repository From bc213f7b71b4a108b7abac9b2f683f1eeb9abe05 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Wed, 30 Sep 2026 11:45:51 -0700 Subject: [PATCH 20/23] docs: correct the sizing recipe and restore a missing gap An independent review found the section 4 recipe sized against the wrong constraint, and verification against the rules confirmed it. Section 4.4 requires the steady window to span more than 4 super-passes (MIN_TREND_N = 4). Sizing on the duration floor alone gives N = 1 at low concurrency, which 4.4 classes insufficient_passes and does not count as an official result. The rule is now N = max(5, ceil(duration x throughput / dataset_size)). Section 4 also claimed to satisfy 6.2. It does not: 6.2 is measured over the steady window's issue-time span, which section 3.1 says is uncomputable for this workload. The recipe now distinguishes what it satisfies outright (6.4's issued count) from what it only makes possible (6.2 and 4.4, uncertifiable until the gating gap closes). Section 1's claim that neither rule is expressible is corrected: 6.4 is, and section 4 is the proof. Restores the stream_all_chunks conflict as gap 4, dropped when the list was cut to three. Section 6.5 mandates it true for all performance runs and a non-streaming workload cannot comply, so it needs a rules amendment rather than a client change. Adds two caveats the recipe was missing: termination counts issued while 6.4's minimum counts completed, so failures can end a point short; and 6.5 mandates WithReplacementSampleOrder, which the example now sets. Fixes the per-user-rate citation, which pointed at qps rather than the latency P90 it is actually the reciprocal of, and re-baselines the rules citations from 6b0b1ef to cdb203c. Co-Authored-By: Claude Sonnet 4.6 --- docs/t2v_benchmark_gaps.md | 61 +++++++++++++++++++++++++++----------- 1 file changed, 43 insertions(+), 18 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index ee5e9d53..583d00e5 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -3,7 +3,7 @@ Status: proposal · Baseline: `4235a9c` · Scope: this repository only. > **Rules basis.** Requirements below are cited against `endpoints_policies` branch -> `v1.0_rules_dev` @ `6b0b1ef` (2026-09-22). That branch is **unmerged and still moving**; every +> `v1.0_rules_dev` @ `cdb203c` (2026-09-30). That branch is **unmerged and still moving**; every > rule citation here is as-of that commit and should be re-checked before being relied on. An Endpoints submission needs a **pareto curve**: several runs at different concurrency levels, @@ -24,8 +24,10 @@ Everything around them is what is missing, and §5 lists that work. **The gaps:** -- 3 gaps (§3), plus example configs. -- Only 1 is a hard blocker: steady state cannot certify a window without tokens (§3.1). +- 4 gaps (§3), plus example configs. +- 1 is a hard blocker: steady state cannot certify a window without tokens (§3.1). +- 1 is a **rules** conflict, not a client change: §6.5 mandates `stream_all_chunks = true`, which + a non-streaming workload cannot set. It needs an amendment. - The client also has limits that affect every model, not just video. They do **not** block a T2V benchmark, because the same workaround text curves already use works here too. See §3.3. @@ -38,6 +40,7 @@ Everything around them is what is missing, and §5 lists that work. | [Naming](#a-naming) | A ruleset model and dataset entry, so the tooling recognises the benchmark at all | | [Tokenless support](#b-tokenless-support) | Steady-state gating without TPOT, and artifact-safe responses | | [Example configs](#c-example-configs) | Concurrency-region configs sized to whole dataset passes | +| Rules amendment | An exemption from §6.5's `stream_all_chunks = true`. Not a client task; recorded so the gap list is complete | The adapter and the example workload already run. What is missing is the description of the benchmark to the tooling, and the handling of a reply that carries no tokens. @@ -51,11 +54,17 @@ runs. 2 rules govern how a single point must run: - **§6.4**: total samples issued at a point MUST be a positive integer multiple of the dataset - size. A point ends on a whole pass, not on a clock. + size. A point ends on a whole pass, not on a clock. §6.4 separately requires a minimum number + of *completed* queries, one pass over the dataset. - **§6.2**: minimum steady-state duration is 600 s in the Ultra Low Concurrency region and **1200 s** in the Low, Medium, and High regions. +- **§4.4**: the steady window must span **more than 4 super-passes** (`MIN_TREND_N = 4`), and + §6.2's minimum is measured over that window's **issue-time span**, not wall-clock. The effective + floor is `max(4 super-passes, §6.2 minimum)`. -Neither is expressible today for a concurrency-scheduled run: see §3.3. +§6.4's issued-count rule **is** expressible today, and §4 shows how. What cannot be expressed is a +duration floor (§3.3), and what cannot be *certified* is §6.2 or §4.4, because neither is +computable without a steady window (§3.1). ### 1.1 Metrics for video generation @@ -104,6 +113,7 @@ Ordered by the categories in §0. | 1 | No ruleset entry exists for the benchmark, so `_resolve_model` raises `KeyError` before any other check runs. Without one there is also no golden accuracy and no validity thresholds. | `compliance/checker.py:156-164` | code | | 2 | Steady state cannot certify a window with no tokens. See §3.1. **This is the only hard blocker.** | `metrics/steady_state_diagnostics.py` | code | | 3 | `VideoGenAdapter` mirrors the video *path string* into `response_output`, which the OSL trigger tokenizes. Harmless only because the model name resolves to no tokenizer; supplying one yields meaningless OSL and TPS. See §3.2. | `videogen/adapter.py` | latent | +| 4 | §6.5 requires `stream_all_chunks = true` for **all** performance runs. Every video config sets `streaming: "off"`, and `VideoGenAdapter.decode_sse_message` raises by design. The rule text forbids the only configuration the workload can run, so this needs a rules amendment, not a client change. | `videogen/adapter.py`, §6.5 | rules | ### 3.1 Why steady state cannot gate a tokenless run @@ -152,8 +162,9 @@ benchmark, and they are recorded here only so the §4 recipe makes sense. - One invocation measures one point. There is no sweep driver and no multi-point publish layout. Existing text curves are built by running each concurrency on its own and joining the results afterwards; the same applies here. -- The per-user rate is not emitted as a named field, but it is the reciprocal of a value already - reported (`metrics/report.py:250`), so it is one division in post-processing. +- The per-user rate is not emitted as a named field, but it is the reciprocal of the latency P90 + already reported inside the `latency` rollup (`metrics/report.py:234`), so it is one division in + post-processing. It is *not* derived from `qps` (`:250`); §1.1 defines it from tail latency. - `compliance/checker.py` requires `target_concurrency == 1`, but it is scoped to Edge-Agentic (BFCL v4) submissions and is reachable only through `scripts/check_compliance.py`. It is not the validator for an Endpoints pareto submission, so it is not on this path. @@ -163,33 +174,47 @@ not a prerequisite here. ## 4. Workaround available today -Since a point must end on a whole dataset pass (§6.4) *and* meet a duration floor (§6.2), and no -floor setting exists (§3.3), size the count to the duration instead of capping the clock: +A point must end on a whole dataset pass (§6.4), span more than 4 super-passes (§4.4), and meet a +duration floor (§6.2). No floor setting exists (§3.3), so size the count instead of capping the +clock: 1. Calibrate. Run the point briefly to estimate sustained throughput at that concurrency. -2. Choose the smallest integer `N` where `N × dataset_size / throughput` comfortably exceeds the - region's minimum duration. +2. Choose `N = max(5, ceil(min_duration x throughput / dataset_size))`. The **5** is §4.4's + trend-test floor: a window needs more than 4 super-passes, and one super-pass is one dataset + pass by default. Sizing on duration alone gives `N = 1` at low concurrency, which §4.4 classes + as `insufficient_passes` and does **not** count as an official result. 3. Issue exactly that many samples, with no clock cap: ```yaml settings: runtime: - n_samples_to_issue: 1240 # N x dataset_size (e.g. 5 x 248), an exact multiple + n_samples_to_issue: 1240 # N x dataset_size (5 x 248), an exact multiple + sample_order: with_replacement # sec 6.5 requires WithReplacementSampleOrder for perf runs load_pattern: type: "concurrency" target_concurrency: 1 ``` -The phase then ends on the count, at a pass boundary, having run at least the required duration, -satisfying both rules without a client patch. `n_samples_to_issue` is returned verbatim and the +The phase ends on the count, at a pass boundary. `n_samples_to_issue` is returned verbatim and the sample order is infinite, so nothing clamps it. +**What this does and does not satisfy.** It satisfies §6.4's issued-count rule outright. It makes +the run long enough in *wall-clock* to clear §6.2, which is necessary but **not sufficient**: §6.2 +is measured over the steady window's issue-time span, after warmup and ramp-crop and before the +drain, and §4.4 gates on that window spanning more than 4 super-passes. Neither can be certified +for this workload until gap 2 is closed, because there is no window to measure (§3.1). Size for +them anyway, so the data qualifies once it can be checked. + - **The duration is achieved, not enforced.** If throughput is lower than calibrated the point - simply runs longer, which is safe. If higher, it may undershoot the floor, so size `N` with - margin and check the achieved duration afterwards. + runs longer, which is safe. If higher, it may undershoot, so size `N` with margin and check the + achieved duration afterwards. +- **Termination counts issued, not completed** (`load_generator/session.py:868-873`), while §6.4's + minimum-query rule counts *completed*. A point with request failures issues exactly + `N x dataset_size` yet can complete fewer, and finishes early because failures return fast. + Check `n_samples_completed`, not just elapsed time. - Do **not** use `min_issue_duration_ms` for this (§3.3). -- Capping with `max_issue_duration_ms` instead would end the run mid-pass and violate §6.4. It - remains useful only as a runaway guard set well above the expected finish. +- Capping with `max_issue_duration_ms` would end the run mid-pass and violate §6.4. It is useful + only as a runaway guard set well above the expected finish. ## 5. Pending work From 58ea98c109825eb198b22e23f0bd6f7fe57182b2 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Wed, 30 Sep 2026 16:35:16 -0700 Subject: [PATCH 21/23] docs: add the perf-run sample order gap The sizing recipe set settings.runtime.sample_order, which RuntimeConfig does not declare and rejects as an extra key. RuntimeSettings.from_config always builds the without-replacement default, so no perf run can meet the v1.0 section 6.5 requirement for WithReplacementSampleOrder. Record it as gap 4 under a new Run settings category with task C1, drop the invalid key from the recipe, and renumber the following sections. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/t2v_benchmark_gaps.md | 65 ++++++++++++++++++++++++++++---------- 1 file changed, 48 insertions(+), 17 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index 583d00e5..53de8ea2 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -24,12 +24,15 @@ Everything around them is what is missing, and §5 lists that work. **The gaps:** -- 4 gaps (§3), plus example configs. -- 1 is a hard blocker: steady state cannot certify a window without tokens (§3.1). +- 5 gaps (§3), plus example configs. +- 2 block a valid result: + - Steady state cannot certify a window without tokens (§3.1). This is the large one. + - Perf runs cannot select with-replacement sample order, which §6.5 requires (§3.3). This one + is small. - 1 is a **rules** conflict, not a client change: §6.5 mandates `stream_all_chunks = true`, which a non-streaming workload cannot set. It needs an amendment. - The client also has limits that affect every model, not just video. They do **not** block a - T2V benchmark, because the same workaround text curves already use works here too. See §3.3. + T2V benchmark, because the same workaround text curves already use works here too. See §3.4. **Scope:** the client only. §6 lists what is left out. @@ -39,7 +42,8 @@ Everything around them is what is missing, and §5 lists that work. | --- | --- | | [Naming](#a-naming) | A ruleset model and dataset entry, so the tooling recognises the benchmark at all | | [Tokenless support](#b-tokenless-support) | Steady-state gating without TPOT, and artifact-safe responses | -| [Example configs](#c-example-configs) | Concurrency-region configs sized to whole dataset passes | +| [Run settings](#c-run-settings) | A config setting for with-replacement sample order on perf runs | +| [Example configs](#d-example-configs) | Concurrency-region configs sized to whole dataset passes | | Rules amendment | An exemption from §6.5's `stream_all_chunks = true`. Not a client task; recorded so the gap list is complete | The adapter and the example workload already run. What is missing is the description of the @@ -63,7 +67,7 @@ runs. floor is `max(4 super-passes, §6.2 minimum)`. §6.4's issued-count rule **is** expressible today, and §4 shows how. What cannot be expressed is a -duration floor (§3.3), and what cannot be *certified* is §6.2 or §4.4, because neither is +duration floor (§3.4), and what cannot be *certified* is §6.2 or §4.4, because neither is computable without a steady window (§3.1). ### 1.1 Metrics for video generation @@ -111,9 +115,10 @@ Ordered by the categories in §0. | # | Gap | Evidence | Kind | | --- | --- | --- | --- | | 1 | No ruleset entry exists for the benchmark, so `_resolve_model` raises `KeyError` before any other check runs. Without one there is also no golden accuracy and no validity thresholds. | `compliance/checker.py:156-164` | code | -| 2 | Steady state cannot certify a window with no tokens. See §3.1. **This is the only hard blocker.** | `metrics/steady_state_diagnostics.py` | code | +| 2 | Steady state cannot certify a window with no tokens. See §3.1. **This is the largest blocker.** | `metrics/steady_state_diagnostics.py` | code | | 3 | `VideoGenAdapter` mirrors the video *path string* into `response_output`, which the OSL trigger tokenizes. Harmless only because the model name resolves to no tokenizer; supplying one yields meaningless OSL and TPS. See §3.2. | `videogen/adapter.py` | latent | -| 4 | §6.5 requires `stream_all_chunks = true` for **all** performance runs. Every video config sets `streaming: "off"`, and `VideoGenAdapter.decode_sse_message` raises by design. The rule text forbids the only configuration the workload can run, so this needs a rules amendment, not a client change. | `videogen/adapter.py`, §6.5 | rules | +| 4 | §6.5 requires `WithReplacementSampleOrder` for performance runs, but no config field selects it: `RuntimeConfig` has no `sample_order` and forbids extra keys, and `RuntimeSettings.from_config` always builds the without-replacement default. Only the TEST04 audit overrides it. Affects every benchmark, not just video. See §3.3. | `config/runtime_settings.py:174`, `config/schema.py:593-600` | code | +| 5 | §6.5 requires `stream_all_chunks = true` for **all** performance runs. Every video config sets `streaming: "off"`, and `VideoGenAdapter.decode_sse_message` raises by design. The rule text forbids the only configuration the workload can run, so this needs a rules amendment, not a client change. | `videogen/adapter.py`, §6.5 | rules | ### 3.1 Why steady state cannot gate a tokenless run @@ -150,7 +155,24 @@ output-sequence-length trigger tokenizes. Nothing breaks today only because the not resolve to a tokenizer, so token metrics stay disabled. Supplying `--tokenizer` would silently produce OSL and TPS figures computed from a filesystem path. -### 3.3 Client limits that are not T2V problems +### 3.3 Sample order cannot be set for perf runs + +§6.5 (`endpoints_rules.md:1228-1229`) requires `WithReplacementSampleOrder` for performance runs +and `WithoutReplacementSampleOrder` for accuracy runs. The client implements both orders +(`load_generator/sample_order.py`), but the user-facing config cannot choose between them: + +- `RuntimeConfig` (`config/schema.py:593`) has no `sample_order` field and sets + `extra="forbid"` (`:600`), so a `sample_order:` key in YAML is a validation error. +- `RuntimeSettings.from_config` sets `"sample_order": SampleOrderSpec()` + (`config/runtime_settings.py:174`), which is `WITHOUT_REPLACEMENT`. +- The only override is the audit path (`commands/benchmark/execute.py`), which TEST04 uses for + its own phases. + +So every perf run today samples without replacement. This is not specific to video, but a T2V +perf point cannot be a valid result until it is fixed. The fix is small: expose the order in +`RuntimeConfig`, or have a ruleset set it per test mode. + +### 3.4 Client limits that are not T2V problems Working out how to run a sweep surfaced several limits in the client. None of them block a T2V benchmark, and they are recorded here only so the §4 recipe makes sense. @@ -175,7 +197,7 @@ not a prerequisite here. ## 4. Workaround available today A point must end on a whole dataset pass (§6.4), span more than 4 super-passes (§4.4), and meet a -duration floor (§6.2). No floor setting exists (§3.3), so size the count instead of capping the +duration floor (§6.2). No floor setting exists (§3.4), so size the count instead of capping the clock: 1. Calibrate. Run the point briefly to estimate sustained throughput at that concurrency. @@ -189,12 +211,14 @@ clock: settings: runtime: n_samples_to_issue: 1240 # N x dataset_size (5 x 248), an exact multiple - sample_order: with_replacement # sec 6.5 requires WithReplacementSampleOrder for perf runs load_pattern: type: "concurrency" target_concurrency: 1 ``` +There is no setting for sample order yet, so this run samples without replacement and does not +meet §6.5 until gap 4 is closed (§3.3). + The phase ends on the count, at a pass boundary. `n_samples_to_issue` is returned verbatim and the sample order is infinite, so nothing clamps it. @@ -212,7 +236,7 @@ them anyway, so the data qualifies once it can be checked. minimum-query rule counts *completed*. A point with request failures issues exactly `N x dataset_size` yet can complete fewer, and finishes early because failures return fast. Check `n_samples_completed`, not just elapsed time. -- Do **not** use `min_issue_duration_ms` for this (§3.3). +- Do **not** use `min_issue_duration_ms` for this (§3.4). - Capping with `max_issue_duration_ms` would end the run mid-pass and violate §6.4. It is useful only as a runaway guard set well above the expected finish. @@ -234,21 +258,28 @@ requires an explicit marker. | B1 | Modality-aware steady-state gating: gate on an alternative metric where the gated one has no samples (end-to-end latency exists for every workload), or report un-gated with a reason instead of returning `None`. Requires a way for a workload to declare its gating metric. | 2 | n/a | M | | B2 | Remove the tokenizer footgun: stop routing a path through `response_output`, or mark the field non-tokenizable for artifact-output adapters. | 3 | n/a | S | -### C. Example configs +### C. Run settings + +| # | Task | Fixes | Depends on | Size | +| --- | --- | --- | --- | --- | +| C1 | Let a perf run select `WithReplacementSampleOrder`: add `sample_order` to `RuntimeConfig`, or have the ruleset pick it per test mode (with replacement for perf, without for accuracy). | 4 | n/a | XS | + +### D. Example configs | # | Task | Fixes | Depends on | Size | | --- | --- | --- | --- | --- | -| C1 | Add concurrency-region example configs sized per §6.4, using the §4 recipe. The shipped video configs cover only `max_throughput` (the Offline point) and `concurrency: 1`. | n/a | n/a | XS | +| D1 | Add concurrency-region example configs sized per §6.4, using the §4 recipe. The shipped video configs cover only `max_throughput` (the Offline point) and `concurrency: 1`. | n/a | C1 | XS | -### D. Documentation +### E. Documentation | # | Task | Depends on | | --- | --- | --- | -| D1 | Update this document as items land; update `AGENTS.md` if any module moves or is added. | any | -| D2 | Note the modality constraint in `steady_state_diagnostics.md` once B1 settles. | B1 | +| E1 | Update this document as items land; update `AGENTS.md` if any module moves or is added. | any | +| E2 | Note the modality constraint in `steady_state_diagnostics.md` once B1 settles. | B1 | **B1 is the critical item.** Under v1.0 §4.4 the steady-state result is the official reporting -basis, so without it a T2V run has no official result. A1 and C1 are small and independent. +basis, so without it a T2V run has no official result. C1 is small but also blocks a valid perf +result. A1 is small and independent; D1 follows C1. ## 6. Out of scope for this repository From 4179d1ea7f27256d86e46049a0a981f276c97b66 Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Thu, 1 Oct 2026 15:48:14 -0700 Subject: [PATCH 22/23] docs: correct the T2V client plan after review Review against the client at 4235a9c and the v1.0 rules at cdb203c. - Steady state: a tokenless point falls back to the whole-run total, which section 4.4 names the official fallback. It blocks a valid point only if the WG rules a not-found run invalid. CL-B1 is now conditional and sized L, with its real touch points listed. - stream_all_chunks is a plain client flag a video config can set; the rules-conflict gap is removed. - New gap: the example configs are not submission-shaped (144-prompt subset, 10-minute caps, 20-sample SingleStream, no submission_ref). - Recipe: N = 1 per point, seeds bound through submission_ref, check n_samples_failed because failures count as completed. Super-pass wording fixed: window of at least 4, run of more than 4. - Target shape follows the elected-Offline decision: 7 runs, 4 accuracy runs, C_min 1, C_max 72. - CL-A1 re-scoped to a dataset entry and the shared slug; CL-B2 keeps the path VBench needs. Task IDs prefixed CL-. - Adds a point timeline and an official-basis decision chart. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/t2v_benchmark_gaps.md | 364 +++++++++++++++++-------------------- 1 file changed, 170 insertions(+), 194 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index 53de8ea2..9db6c21b 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -2,111 +2,114 @@ Status: proposal · Baseline: `4235a9c` · Scope: this repository only. -> **Rules basis.** Requirements below are cited against `endpoints_policies` branch -> `v1.0_rules_dev` @ `cdb203c` (2026-09-30). That branch is **unmerged and still moving**; every -> rule citation here is as-of that commit and should be re-checked before being relied on. +> **Rules basis.** Requirements below cite the MLPerf Endpoints rules on the `endpoints_policies` +> branch `v1.0_rules_dev` @ `cdb203c` (2026-09-30): +> [`endpoints_rules.md`](https://github.com/mlcommons/endpoints_policies/blob/cdb203c/endpoints_rules.md) +> and +> [`endpoints_submission_rules.md`](https://github.com/mlcommons/endpoints_policies/blob/cdb203c/endpoints_submission_rules.md). +> That branch is unmerged; re-check every rule citation before relying on it. -An Endpoints submission needs a **pareto curve**: several runs at different concurrency levels, -plus 1 Offline run and a set of accuracy runs. Each run has to hold steady for a minimum time. - -**Blockers:** - -- The client measures **1 point per run**. There is no way to sweep several. -- A video reply is **1 file, not a stream of tokens**. The metric and steady-state code assume - tokens everywhere. +An Endpoints submission is a **pareto curve**: runs at several concurrency levels, an Offline +result, and accuracy runs. This document covers what the client must change so a text-to-video +(T2V) workload, Wan2.2 with 248 prompts and 1 video file per response, can produce valid points. **What already exists:** -- The adapter, `videogen/`. -- The example workload, `examples/09_Wan22_VideoGen_Example/`. +- The adapter, `videogen/`, and the example workload, `examples/09_Wan22_VideoGen_Example/`. +- System throughput and a P90 latency percentile, both emitted natively (§2). -Everything around them is what is missing, and §5 lists that work. +**What blocks a valid point:** -**The gaps:** +- Perf runs cannot select with-replacement sample order, which §6.5 requires (gap 4). Small. +- The shipped example configs are not submission-shaped (gap 5). Small. -- 5 gaps (§3), plus example configs. -- 2 block a valid result: - - Steady state cannot certify a window without tokens (§3.1). This is the large one. - - Perf runs cannot select with-replacement sample order, which §6.5 requires (§3.3). This one - is small. -- 1 is a **rules** conflict, not a client change: §6.5 mandates `stream_all_chunks = true`, which - a non-streaming workload cannot set. It needs an amendment. -- The client also has limits that affect every model, not just video. They do **not** block a - T2V benchmark, because the same workaround text curves already use works here too. See §3.4. +**What does not block a valid point under the rules as written:** -**Scope:** the client only. §6 lists what is left out. +- Steady state cannot gate a tokenless run (gap 2). The point falls back to the whole-run `total` + metrics, which §4.4 names "the official fallback where no steady state holds". This becomes a + blocker only if the working group (WG) rules a `not found` run invalid, which §4.4 lists as + pending ratification. +- `stream_all_chunks = true` (§6.5) is a plain client flag that a video config can set (§3.4). + +**Not client work:** the §9.1 metric consistency check requires a TPOT distribution and rejects a +submission at Week 0. That needs WG rule text, tracked in the NVIDIA readiness plan. ## 0. What is missing | Category | Missing | | --- | --- | -| [Naming](#a-naming) | A ruleset model and dataset entry, so the tooling recognises the benchmark at all | -| [Tokenless support](#b-tokenless-support) | Steady-state gating without TPOT, and artifact-safe responses | +| [Naming](#a-naming) | A dataset entry and 1 model slug shared with the results repository | +| [Tokenless support](#b-tokenless-support) | Artifact-safe tokenizer handling; steady-state gating without TPOT, if the WG requires it | | [Run settings](#c-run-settings) | A config setting for with-replacement sample order on perf runs | -| [Example configs](#d-example-configs) | Concurrency-region configs sized to whole dataset passes | -| Rules amendment | An exemption from §6.5's `stream_all_chunks = true`. Not a client task; recorded so the gap list is complete | - -The adapter and the example workload already run. What is missing is the description of the -benchmark to the tooling, and the handling of a reply that carries no tokens. +| [Example configs](#d-example-configs) | Point configs sized to whole passes, with seeds bound and no binding time cap | +| Rules amendments | Video forms of §9.1 metric consistency, §4.1 metrics, and §4.4 gating. Not client tasks | ## 1. Target shape -Per v1.0 §5.3 and §5.7, a non-agentic submission is 4 mandatory concurrency points (Ultra Low -1-32, Low, Medium, High), 3 submitter's-choice points, 1 Offline point, and 5 accuracy -runs. - -2 rules govern how a single point must run: - -- **§6.4**: total samples issued at a point MUST be a positive integer multiple of the dataset - size. A point ends on a whole pass, not on a clock. §6.4 separately requires a minimum number - of *completed* queries, one pass over the dataset. -- **§6.2**: minimum steady-state duration is 600 s in the Ultra Low Concurrency region and - **1200 s** in the Low, Medium, and High regions. -- **§4.4**: the steady window must span **more than 4 super-passes** (`MIN_TREND_N = 4`), and - §6.2's minimum is measured over that window's **issue-time span**, not wall-clock. The effective - floor is `max(4 super-passes, §6.2 minimum)`. +Per v1.0 §5.3 and §5.7.2, a non-agentic submission with an **elected** Offline result is **7 +runs**: 1 Ultra Low point, 3 mandatory points (Low, Medium, High), and 3 submitter's-choice points, +with the `C_max` point also reported as the Offline result. Accuracy is required at the 4 mandatory +points plus Offline; the elected `C_max` point counts for both, so **4 accuracy runs**. + +The planned point set, from the readiness plan: `C_min = 1`, `C_max = 72`, 1 x 72-GPU layout, +points at concurrency 1, 2, 4, 8, 12, 36, 72; accuracy at 1, 4, 12, 72. + +3 rules govern how a single point runs: + +- **§6.4**: total samples issued MUST be a positive integer multiple of the dataset size, and at + least 1 full pass must *complete*. +- **§6.2**: minimum duration is 600 s in Ultra Low and 1200 s in Low, Medium, and High, measured + on the steady window's issue-time span when a window exists. +- **§4.4**: a steady-state result needs a window of **≥ 4 super-passes**, so a run of **more than + 4**, and a gating metric that is a Plateau. Otherwise the point reports `total`. + +```mermaid +flowchart LR + W["declared warmup
not in data"] --> R["SP1: ramp crop
at least 1"] + R --> S1[SP2] --> S2[SP3] --> S3[SP4] --> S4[SP5] + S4 --> D["drain
excluded on issue time"] + subgraph window["steady window: at least 4 SPs AND issue span at least 600 or 1200 s"] + S1 & S2 & S3 & S4 + end +``` -§6.4's issued-count rule **is** expressible today, and §4 shows how. What cannot be expressed is a -duration floor (§3.4), and what cannot be *certified* is §6.2 or §4.4, because neither is -computable without a steady window (§3.1). +```mermaid +flowchart TD + A{"gating metric
has samples?"} -- no --> T["total is the official result
(not found: validity pending ratification)"] + A -- yes --> B{"window at least
4 super-passes?"} -- no --> T + B -- yes --> C{"window issue span
meets §6.2?"} -- no --> T + C -- yes --> S["steady-state metrics official
total supplementary"] +``` ### 1.1 Metrics for video generation -For video generation the per-point metrics cannot be token-derived, because a response is 1 -artifact rather than a token stream: - -| Token metric | Substitute | -| --- | --- | -| `system_tps` | `completed_videos / elapsed_seconds` | -| `tps_per_user` (v1.0: `1000 / tpot_p90_ms`) | `1000 / latency_p90_ms`, the same latency-reciprocal shape | -| `ttft_*` | end-to-end issue to complete-response percentile | +Proposed, pending rule text; names match the results-repository plan: -The per-user metric mirrors v1.0's definition rather than dividing throughput by concurrency, -because v1.0 derives per-user rate from the tail latency a user actually experiences. +| Token metric | Video metric | From the client report | +| --- | --- | --- | +| `system_tps` | `system_qps` | `qps` | +| `tps_per_user` (v1.0: `1000 / tpot_p90_ms`) | `qps_per_user` = `1000 / latency_p90_ms` | derived | +| `ttft_p90_ms` | `latency_p90_ms`, issue to complete response | `latency.percentiles["90.0"] / 1e6` (reported in ns) | -**Percentile: P90.** This *matches* v1.0 rather than diverging from it, since §4.1 moved TTFT from -P95 to P90 with an explicit versioning note. The independent reason also holds: a stable P95 needs -roughly twice the completed queries of a P90, and 1 video takes tens of seconds to minutes, so a -P95 at concurrency 1 would need on the order of 100+ videos per point. `DEFAULT_PERCENTILES` -already carries both (`async_utils/services/metrics_aggregator/registry.py:411-423`). +The per-user metric mirrors v1.0's latency-reciprocal definition rather than dividing throughput by +concurrency. **P90** aligns with v1.0 §4.1, which moved TTFT from P95 to P90. `DEFAULT_PERCENTILES` +carries both (`async_utils/services/metrics_aggregator/registry.py:411-423`). -**Offline point.** `max_throughput` is the Offline load pattern, so the existing -`examples/09_Wan22_VideoGen_Example/offline_wan22*.yaml` configs are already the right shape. Per -§5.7 the reported concurrency is the dataset cardinality, throughput is the only metric of -interest, and latency metrics explicitly do not apply. +**Offline.** The elected `C_max` point keeps its latency metrics (§5.7.2). No `max_throughput` run +is needed. ## 2. Already works, no change needed -- **System throughput.** `Report.qps` = `n_completed / duration_s`, duration from the - `tracked_duration_ns` counter (`metrics/report.py:360-361`, `:390-396`). For a video workload - that *is* videos/second. The legacy LoadGen window is poisson-only - (`commands/benchmark/pipeline.py:222-225`), so concurrency uses the native window. -- **Per-request latency percentile.** `sample_latency_ns` maps to `latency` - (`metrics/report.py:50`, `:234`), with P90 and P95 both in the default grid. +- **System throughput.** `Report.qps` = `n_completed / duration_s` (`metrics/report.py:360-361`, + `:390-396`). For a video workload that is videos/second on the `total` basis. A steady-window + videos/s would need CL-B1. +- **Per-request latency percentile.** `sample_latency_ns` maps to `latency` (`metrics/report.py:50`, + `:234`), with P90 and P95 in the default grid. - **Sample order is infinite**, never raising `StopIteration` (`load_generator/sample_order.py`). -- **Audit.** TEST04 already accepts `concurrency` - (`compliance/audit_test/output_caching_test.py`). -- **Missing tokenizer is handled gracefully**, not fatally (`commands/benchmark/execute.py`). +- **Audit.** TEST04 accepts `concurrency` (`compliance/audit_test/output_caching_test.py`). +- **Missing tokenizer is handled.** `_check_tokenizer_exists` returns `False` with a warning for a + name that is not a tokenizer (`commands/benchmark/execute.py:215-259`), so token metrics stay off. +- **`stream_all_chunks`** is settable for any workload (§3.4). ## 3. Gaps @@ -114,177 +117,150 @@ Ordered by the categories in §0. | # | Gap | Evidence | Kind | | --- | --- | --- | --- | -| 1 | No ruleset entry exists for the benchmark, so `_resolve_model` raises `KeyError` before any other check runs. Without one there is also no golden accuracy and no validity thresholds. | `compliance/checker.py:156-164` | code | -| 2 | Steady state cannot certify a window with no tokens. See §3.1. **This is the largest blocker.** | `metrics/steady_state_diagnostics.py` | code | -| 3 | `VideoGenAdapter` mirrors the video *path string* into `response_output`, which the OSL trigger tokenizes. Harmless only because the model name resolves to no tokenizer; supplying one yields meaningless OSL and TPS. See §3.2. | `videogen/adapter.py` | latent | -| 4 | §6.5 requires `WithReplacementSampleOrder` for performance runs, but no config field selects it: `RuntimeConfig` has no `sample_order` and forbids extra keys, and `RuntimeSettings.from_config` always builds the without-replacement default. Only the TEST04 audit overrides it. Affects every benchmark, not just video. See §3.3. | `config/runtime_settings.py:174`, `config/schema.py:593-600` | code | -| 5 | §6.5 requires `stream_all_chunks = true` for **all** performance runs. Every video config sets `streaming: "off"`, and `VideoGenAdapter.decode_sse_message` raises by design. The rule text forbids the only configuration the workload can run, so this needs a rules amendment, not a client change. | `videogen/adapter.py`, §6.5 | rules | - -### 3.1 Why steady state cannot gate a tokenless run - -This is the most consequential of the tokenless gaps. Under v1.0 §4.4 the detected steady-state -window is not a diagnostic. It is **the official reporting basis**: a point's metrics are computed -over that window, with whole-run values kept only as supplementary. §4.4 names the gating metric -as TPOT at P50 and P90. A workload with no TPOT therefore has no gating metric, and so no official -result. - -See [`steady_state_diagnostics.md`](steady_state_diagnostics.md) for the algorithm; this is only -why it does not apply. - -- **The exclusion is explicit, not accidental.** `steady_state_profile` - (`commands/benchmark/pipeline.py:96-118`) returns `None`, meaning no collection, unless the run - has a resolved tokenizer *and* streaming, on top of `settings.steady_state.enabled` being set at - all (it is off by default). A tokenless run fails both conditions by construction. -- The reasoning is sound for its purpose: without streaming there is no `TpotTrigger`, so every - plateau gate would see an empty TPOT series and nothing would ever validate. -- Outcome is a `None` result, not an error: `Report.steady_state` is `SteadyState | None` - (`metrics/report.py:289`, `:456`). -- There is no way for a workload to declare a different gating metric. Eligibility comes from - `Profile.supported` (`metrics/steady_state_diagnostics.py:311`), and `SteadyStateConfig` - (`config/schema.py:947`, `:994`) carries no model allowlist (`:958`). - -The tension is between a defensible client design and the v1.0 rule. The client correctly declines -to emit a result it cannot compute, while §4.4 makes that result the basis of the official -result. Closing it needs a gating metric that exists for tokenless workloads, not a change on -either side alone. +| 1 | No dataset entry or agreed slug. The Endpoints round ruleset deliberately has no per-model rulesets (`benchmark_rulesets={}`), so a model entry has no consumer; the canonical dataset and the slug do. | `config/rulesets/mlcommons/rules.py:364-367`, `datasets.py` | naming | +| 2 | Steady state cannot gate a tokenless run, so the point reports `total`. Blocks a valid point only if the WG rules `not found` invalid. See §3.1. | `commands/benchmark/pipeline.py:96-138`, `metrics/steady_state_diagnostics.py` | code, conditional | +| 3 | `VideoGenAdapter` mirrors the video path into `response_output`, which the OSL trigger tokenizes if a tokenizer is supplied. See §3.2. | `videogen/adapter.py:123-127` | latent | +| 4 | §6.5 requires `WithReplacementSampleOrder` for perf runs, but no config field selects it. See §3.3. | `config/runtime_settings.py:174`, `config/schema.py:593-600` | code | +| 5 | The example configs are not submission-shaped: `offline_wan22_submission.yaml` issues a 144-prompt subset; `offline_wan22.yaml` and `offline_wan22_accuracy.yaml` cap at 10 min where 1 pass takes about 24 min; `single_stream_wan22_submission.yaml` issues 20; none sets `submission_ref`, so seeds stay at the defaults. | `examples/09_Wan22_VideoGen_Example/` | config | + +### 3.1 Steady state for a tokenless run + +- **The exclusion is explicit.** `steady_state_profile` (`commands/benchmark/pipeline.py:96-138`) + returns `None` unless the run has a resolved tokenizer *and* streaming, on top of + `settings.steady_state.enabled` (off by default). +- **The rules fall back, they do not void.** §4.4 gives `total` as the official result for + `insufficient_passes` and `not found`. Pending ratification: whether `not found` is *invalid* or + *reported-with-flags*. +- **What CL-B1 would take**, if the WG requires a window: + 1. A latency-gated profile in `PROFILES` (`steady_state_diagnostics.py:303-315`) and a way for + the workload to select it; the profile crosses to the aggregator subprocess as + `--steady-state-profile`. + 2. Relax the tokenizer and streaming refusals in `pipeline.steady_state_profile` for that profile. + The collector already records latency without a first chunk. + 3. Generalise TPOT-only logic: anomaly detection, the minimum-duration precision term, drift + watch, and the token-based `TpsBlock` (`:163`). Add a window videos/s. + 4. The standalone post-processing CLI that §4.4 names requires a tokenizer (`:2172`). + 5. It depends on WG text naming the tokenless gating metric. The rules disagree with themselves + today: the definitions table says TPOT P50/P90, the operative paragraph says TTFT and TPOT. ### 3.2 Tokenizer footgun for artifact responses -`VideoGenAdapter` puts the response's file path into `response_output`, which is the field the -output-sequence-length trigger tokenizes. Nothing breaks today only because the model name does -not resolve to a tokenizer, so token metrics stay disabled. Supplying `--tokenizer` would silently -produce OSL and TPS figures computed from a filesystem path. +The path in `response_output` is what VBench scores: the adapter mirrors it "so the event log +carries it to the accuracy scorer" (`videogen/adapter.py:123-124`). Removing it breaks accuracy. +The safe fix is to never tokenize an artifact-output adapter: force `tokenizer_name=None` for it in +`setup_benchmark` (`commands/benchmark/execute.py:516-525`), or reject `--tokenizer`. ### 3.3 Sample order cannot be set for perf runs §6.5 (`endpoints_rules.md:1228-1229`) requires `WithReplacementSampleOrder` for performance runs -and `WithoutReplacementSampleOrder` for accuracy runs. The client implements both orders -(`load_generator/sample_order.py`), but the user-facing config cannot choose between them: +and `WithoutReplacementSampleOrder` for accuracy runs. The client implements both +(`load_generator/sample_order.py`), but the config cannot choose: -- `RuntimeConfig` (`config/schema.py:593`) has no `sample_order` field and sets - `extra="forbid"` (`:600`), so a `sample_order:` key in YAML is a validation error. +- `RuntimeConfig` (`config/schema.py:593`) has no `sample_order` field and sets `extra="forbid"` + (`:600`), so a `sample_order:` key in YAML is a validation error. - `RuntimeSettings.from_config` sets `"sample_order": SampleOrderSpec()` (`config/runtime_settings.py:174`), which is `WITHOUT_REPLACEMENT`. -- The only override is the audit path (`commands/benchmark/execute.py`), which TEST04 uses for - its own phases. +- Only the TEST04 audit overrides it. -So every perf run today samples without replacement. This is not specific to video, but a T2V -perf point cannot be a valid result until it is fixed. The fix is small: expose the order in -`RuntimeConfig`, or have a ruleset set it per test mode. +Every perf run today samples without replacement. This affects every benchmark, but a T2V perf +point cannot be valid until it is fixed. With replacement, a super-pass is 248 random draws rather +than every prompt once; for fixed-shape video output that changes little. ### 3.4 Client limits that are not T2V problems -Working out how to run a sweep surfaced several limits in the client. None of them block a T2V -benchmark, and they are recorded here only so the §4 recipe makes sense. - -- There is no minimum-duration floor in the run loop, and `min_issue_duration_ms` is a poisson - sample-count sizer rather than a floor, rejected for `concurrency` - (`load_generator/session.py:866-880`, `config/schema.py:1021-1027`, - `config/runtime_settings.py:249`). The §4 recipe meets the floor without either. -- One invocation measures one point. There is no sweep driver and no multi-point publish layout. - Existing text curves are built by running each concurrency on its own and joining the results - afterwards; the same applies here. -- The per-user rate is not emitted as a named field, but it is the reciprocal of the latency P90 - already reported inside the `latency` rollup (`metrics/report.py:234`), so it is one division in - post-processing. It is *not* derived from `qps` (`:250`); §1.1 defines it from tail latency. -- `compliance/checker.py` requires `target_concurrency == 1`, but it is scoped to Edge-Agentic - (BFCL v4) submissions and is reachable only through `scripts/check_compliance.py`. It is not the - validator for an Endpoints pareto submission, so it is not on this path. - -Fixing any of these would help every benchmark, not just video. That is a separate argument and -not a prerequisite here. - -## 4. Workaround available today - -A point must end on a whole dataset pass (§6.4), span more than 4 super-passes (§4.4), and meet a -duration floor (§6.2). No floor setting exists (§3.4), so size the count instead of capping the -clock: - -1. Calibrate. Run the point briefly to estimate sustained throughput at that concurrency. -2. Choose `N = max(5, ceil(min_duration x throughput / dataset_size))`. The **5** is §4.4's - trend-test floor: a window needs more than 4 super-passes, and one super-pass is one dataset - pass by default. Sizing on duration alone gives `N = 1` at low concurrency, which §4.4 classes - as `insufficient_passes` and does **not** count as an official result. -3. Issue exactly that many samples, with no clock cap: +None of these block a T2V benchmark; they explain the §4 recipe. + +- **No duration floor.** `min_issue_duration_ms` sizes poisson runs and is rejected for + `concurrency` (`load_generator/session.py:866-880`, `config/schema.py:1021-1027`). The recipe + meets the floor by count. +- **1 invocation measures 1 point.** Curves are built by running each concurrency separately. +- **Failed requests count as completed.** `n_samples_completed` includes failures + (`metrics/report.py:215`), and `qps` divides by it (`:396`), so a point with fast failures + reports inflated `qps` and deflated latency. Check `n_samples_failed`. +- **`stream_all_chunks`** is a plain `settings.client` flag (`endpoint_client/config.py:186`), read + only on the SSE path (`endpoint_client/worker.py:429`); nothing ties it to `streaming`. Setting + it in a video config meets §6.5 and the §9.1 streaming check as written. +- **`compliance/checker.py`** is the Edge-Agentic (BFCL v4) checker, reachable only through + `scripts/check_compliance.py`. It is not the Endpoints validator. + +## 4. Running a point today + +1. **Size `N`, the number of dataset passes.** `N = 1` meets §6.4 and, at the measured 5.7-7.6 s + per video, §6.2 (1 pass is 1,414 s at 5.7 s; the 1200 s floor binds only below 4.84 s). The + point reports `total`. +2. **If a steady window is required** (after CL-B1), size `N = w + max(4, ceil(T_min x X / D))`, + where `w ≥ 1` is the ramp crop, `T_min` the §6.2 floor, `X` throughput, `D` = 248. `N = 5` has + no margin: the adaptive crop can take more than 1 super-pass. +3. **Configure the point:** ```yaml +submission_ref: + ruleset: "endpoints-v1.0-2026-10-C1-A" # binds the seeds (§9.1 seed-set validity) + model: "wan-2.2-t2v-a14b" settings: runtime: - n_samples_to_issue: 1240 # N x dataset_size (5 x 248), an exact multiple + n_samples_to_issue: 248 # N x 248, an exact multiple load_pattern: type: "concurrency" target_concurrency: 1 + client: + stream_all_chunks: true # sec 6.5; no effect on a non-SSE adapter ``` There is no setting for sample order yet, so this run samples without replacement and does not -meet §6.5 until gap 4 is closed (§3.3). - -The phase ends on the count, at a pass boundary. `n_samples_to_issue` is returned verbatim and the -sample order is infinite, so nothing clamps it. - -**What this does and does not satisfy.** It satisfies §6.4's issued-count rule outright. It makes -the run long enough in *wall-clock* to clear §6.2, which is necessary but **not sufficient**: §6.2 -is measured over the steady window's issue-time span, after warmup and ramp-crop and before the -drain, and §4.4 gates on that window spanning more than 4 super-passes. Neither can be certified -for this workload until gap 2 is closed, because there is no window to measure (§3.1). Size for -them anyway, so the data qualifies once it can be checked. - -- **The duration is achieved, not enforced.** If throughput is lower than calibrated the point - runs longer, which is safe. If higher, it may undershoot, so size `N` with margin and check the - achieved duration afterwards. -- **Termination counts issued, not completed** (`load_generator/session.py:868-873`), while §6.4's - minimum-query rule counts *completed*. A point with request failures issues exactly - `N x dataset_size` yet can complete fewer, and finishes early because failures return fast. - Check `n_samples_completed`, not just elapsed time. -- Do **not** use `min_issue_duration_ms` for this (§3.4). -- Capping with `max_issue_duration_ms` would end the run mid-pass and violate §6.4. It is useful - only as a runaway guard set well above the expected finish. +meet §6.5 until CL-C1 lands. The `submission_ref` model name must match the slug CL-A1 registers. +Keep `model_params.name` set to the served model name: when it is unset, the client fills it from +`submission_ref.model` (`config/schema.py:1179-1180`) and would send the slug to the server. + +**Checks after the run:** + +- **`n_samples_failed == 0`.** Termination counts issued, not completed + (`load_generator/session.py:868-873`), and completed includes failures (§3.4). +- **Achieved duration.** The duration is achieved by count, not enforced. +- Do **not** use `min_issue_duration_ms`. Use `max_issue_duration_ms` only as a runaway guard set + well above the expected finish; a binding cap ends the point mid-pass and breaks §6.4. ## 5. Pending work -Grouped by the categories in §0. Every code task carries tests; `AGENTS.md` sets >90% coverage and -requires an explicit marker. +Grouped by the categories in §0. Every code task carries tests; `AGENTS.md` sets > 90% coverage. ### A. Naming | # | Task | Fixes | Depends on | Size | | --- | --- | --- | --- | --- | -| A1 | Register the benchmark as a ruleset model and dataset (`config/rulesets/mlcommons/`). Not needed to *run*, but without it there is no golden accuracy, no validity thresholds, and `_resolve_model` raises. | 1 | n/a | S | +| CL-A1 | Register the 248-prompt dataset in `config/rulesets/mlcommons/datasets.py` with its MLPerf Inference provenance, under the slug `wan-2.2-t2v-a14b`. No per-model ruleset: the Endpoints round has none by design. | 1 | n/a | XS | ### B. Tokenless support | # | Task | Fixes | Depends on | Size | | --- | --- | --- | --- | --- | -| B1 | Modality-aware steady-state gating: gate on an alternative metric where the gated one has no samples (end-to-end latency exists for every workload), or report un-gated with a reason instead of returning `None`. Requires a way for a workload to declare its gating metric. | 2 | n/a | M | -| B2 | Remove the tokenizer footgun: stop routing a path through `response_output`, or mark the field non-tokenizable for artifact-output adapters. | 3 | n/a | S | +| CL-B1 | Latency-gated steady state for tokenless workloads (§3.1, items 1-4). **Only if** the WG rules `not found` invalid or a steady-state basis is wanted. | 2 | WG §4.4 text | L | +| CL-B2 | Never tokenize artifact-output adapters (§3.2). | 3 | n/a | XS | ### C. Run settings | # | Task | Fixes | Depends on | Size | | --- | --- | --- | --- | --- | -| C1 | Let a perf run select `WithReplacementSampleOrder`: add `sample_order` to `RuntimeConfig`, or have the ruleset pick it per test mode (with replacement for perf, without for accuracy). | 4 | n/a | XS | +| CL-C1 | Add `sample_order` to `RuntimeConfig` and map it in `from_config`. Accuracy phases stay safe: they run sequentially, and the audit override still wins. The `regenerate-templates` pre-commit hook updates the YAML templates. | 4 | n/a | XS | ### D. Example configs | # | Task | Fixes | Depends on | Size | | --- | --- | --- | --- | --- | -| D1 | Add concurrency-region example configs sized per §6.4, using the §4 recipe. The shipped video configs cover only `max_throughput` (the Offline point) and `concurrency: 1`. | n/a | C1 | XS | +| CL-D1 | Point configs for the planned set (§1): full 248-prompt dataset, `n_samples_to_issue = N x 248`, `submission_ref`, `stream_all_chunks: true`, `sample_order: with_replacement`, no binding cap. Retire or relabel the 144-prompt and 20-sample configs. | 5 | CL-A1, CL-C1 | S | ### E. Documentation | # | Task | Depends on | | --- | --- | --- | -| E1 | Update this document as items land; update `AGENTS.md` if any module moves or is added. | any | -| E2 | Note the modality constraint in `steady_state_diagnostics.md` once B1 settles. | B1 | +| CL-E1 | Update this document as items land; update `AGENTS.md` if a module moves or is added. | any | +| CL-E2 | Note the modality constraint in `steady_state_diagnostics.md` once CL-B1 settles. | CL-B1 | -**B1 is the critical item.** Under v1.0 §4.4 the steady-state result is the official reporting -basis, so without it a T2V run has no official result. C1 is small but also blocks a valid perf -result. A1 is small and independent; D1 follows C1. +**Critical path:** CL-C1 → CL-D1, with CL-A1 alongside. CL-B1 is conditional on the WG. ## 6. Out of scope for this repository -- The accuracy quality target. The scorer exists; the threshold is a benchmark-definition - decision, not a client one. -- Submitter-side result layout, aggregation, and visualisation of a finished curve. +- The accuracy quality target (the scorer exists; the threshold is a benchmark-definition item). +- Rule text: §9.1 metric consistency, §4.1 metrics, §4.4 gating. +- Result layout, aggregation, and visualisation of a finished curve. - The model server and serving stack under test. -- Executing sweeps and producing results. From 3ecbfdc0179d4602b48df2cc98b572cc400065cb Mon Sep 17 00:00:00 2001 From: Tin-Yin Lai Date: Thu, 1 Oct 2026 16:24:16 -0700 Subject: [PATCH 23/23] docs: note pending rules changes and drop the section 1 charts A pending streaming-check change (d9e1ea6) would block every T2V point; a pending Rules TF patch (b36c220) makes the accuracy count exact without saying how an elected Offline point counts. Remove the point timeline and official-result charts. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/t2v_benchmark_gaps.md | 32 +++++++++++--------------------- 1 file changed, 11 insertions(+), 21 deletions(-) diff --git a/docs/t2v_benchmark_gaps.md b/docs/t2v_benchmark_gaps.md index 9db6c21b..fa6dddbf 100644 --- a/docs/t2v_benchmark_gaps.md +++ b/docs/t2v_benchmark_gaps.md @@ -30,6 +30,10 @@ result, and accuracy runs. This document covers what the client must change so a blocker only if the working group (WG) rules a `not found` run invalid, which §4.4 lists as pending ratification. - `stream_all_chunks = true` (§6.5) is a plain client flag that a video config can set (§3.4). + A pending rules change (`arekay/streaming_chunks_client_false`, `d9e1ea6`) would instead require + `model_params.streaming` to resolve to `on` for every fixed-concurrency perf run. The video + adapter cannot stream, so that would block every T2V point until the rule exempts + non-streaming classes. **Not client work:** the §9.1 metric consistency check requires a TPOT distribution and rejects a submission at Week 0. That needs WG rule text, tracked in the NVIDIA readiness plan. @@ -42,14 +46,17 @@ submission at Week 0. That needs WG rule text, tracked in the NVIDIA readiness p | [Tokenless support](#b-tokenless-support) | Artifact-safe tokenizer handling; steady-state gating without TPOT, if the WG requires it | | [Run settings](#c-run-settings) | A config setting for with-replacement sample order on perf runs | | [Example configs](#d-example-configs) | Point configs sized to whole passes, with seeds bound and no binding time cap | -| Rules amendments | Video forms of §9.1 metric consistency, §4.1 metrics, and §4.4 gating. Not client tasks | +| Rules amendments | Video forms of §9.1 metric consistency, §4.1 metrics, and §4.4 gating; a non-streaming exemption if the pending streaming change merges. Not client tasks | ## 1. Target shape Per v1.0 §5.3 and §5.7.2, a non-agentic submission with an **elected** Offline result is **7 runs**: 1 Ultra Low point, 3 mandatory points (Low, Medium, High), and 3 submitter's-choice points, with the `C_max` point also reported as the Offline result. Accuracy is required at the 4 mandatory -points plus Offline; the elected `C_max` point counts for both, so **4 accuracy runs**. +points plus Offline; the elected `C_max` point fills both roles, so the plan uses **4 accuracy +runs**. A pending Rules TF patch (`nvashutoshd_v1.0_rules_patch`, `b36c220`) makes `N` = 5 an +exact count checked automatically, without saying how an elected point counts; if it needs 2 +results, the plan needs 5 runs. Choice points must not carry accuracy results under that patch. The planned point set, from the readiness plan: `C_min = 1`, `C_max = 72`, 1 x 72-GPU layout, points at concurrency 1, 2, 4, 8, 12, 36, 72; accuracy at 1, 4, 12, 72. @@ -63,24 +70,6 @@ points at concurrency 1, 2, 4, 8, 12, 36, 72; accuracy at 1, 4, 12, 72. - **§4.4**: a steady-state result needs a window of **≥ 4 super-passes**, so a run of **more than 4**, and a gating metric that is a Plateau. Otherwise the point reports `total`. -```mermaid -flowchart LR - W["declared warmup
not in data"] --> R["SP1: ramp crop
at least 1"] - R --> S1[SP2] --> S2[SP3] --> S3[SP4] --> S4[SP5] - S4 --> D["drain
excluded on issue time"] - subgraph window["steady window: at least 4 SPs AND issue span at least 600 or 1200 s"] - S1 & S2 & S3 & S4 - end -``` - -```mermaid -flowchart TD - A{"gating metric
has samples?"} -- no --> T["total is the official result
(not found: validity pending ratification)"] - A -- yes --> B{"window at least
4 super-passes?"} -- no --> T - B -- yes --> C{"window issue span
meets §6.2?"} -- no --> T - C -- yes --> S["steady-state metrics official
total supplementary"] -``` - ### 1.1 Metrics for video generation Proposed, pending rule text; names match the results-repository plan: @@ -179,7 +168,8 @@ None of these block a T2V benchmark; they explain the §4 recipe. reports inflated `qps` and deflated latency. Check `n_samples_failed`. - **`stream_all_chunks`** is a plain `settings.client` flag (`endpoint_client/config.py:186`), read only on the SSE path (`endpoint_client/worker.py:429`); nothing ties it to `streaming`. Setting - it in a video config meets §6.5 and the §9.1 streaming check as written. + it in a video config meets §6.5 and the §9.1 streaming check as written at `cdb203c`; see the + pending change noted at the top of this document. - **`compliance/checker.py`** is the Edge-Agentic (BFCL v4) checker, reachable only through `scripts/check_compliance.py`. It is not the Endpoints validator.