Skip to content

Commit 3ecbfdc

Browse files
Tin-Yin Laiclaude
andcommitted
docs: note pending rules changes and drop the section 1 charts
A pending streaming-check change (d9e1ea6) would block every T2V point; a pending Rules TF patch (b36c220) makes the accuracy count exact without saying how an elected Offline point counts. Remove the point timeline and official-result charts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent 4179d1e commit 3ecbfdc

1 file changed

Lines changed: 11 additions & 21 deletions

File tree

‎docs/t2v_benchmark_gaps.md‎

Lines changed: 11 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,10 @@ result, and accuracy runs. This document covers what the client must change so a
3030
blocker only if the working group (WG) rules a `not found` run invalid, which §4.4 lists as
3131
pending ratification.
3232
- `stream_all_chunks = true` (§6.5) is a plain client flag that a video config can set (§3.4).
33+
A pending rules change (`arekay/streaming_chunks_client_false`, `d9e1ea6`) would instead require
34+
`model_params.streaming` to resolve to `on` for every fixed-concurrency perf run. The video
35+
adapter cannot stream, so that would block every T2V point until the rule exempts
36+
non-streaming classes.
3337

3438
**Not client work:** the §9.1 metric consistency check requires a TPOT distribution and rejects a
3539
submission at Week 0. That needs WG rule text, tracked in the NVIDIA readiness plan.
@@ -42,14 +46,17 @@ submission at Week 0. That needs WG rule text, tracked in the NVIDIA readiness p
4246
| [Tokenless support](#b-tokenless-support) | Artifact-safe tokenizer handling; steady-state gating without TPOT, if the WG requires it |
4347
| [Run settings](#c-run-settings) | A config setting for with-replacement sample order on perf runs |
4448
| [Example configs](#d-example-configs) | Point configs sized to whole passes, with seeds bound and no binding time cap |
45-
| Rules amendments | Video forms of §9.1 metric consistency, §4.1 metrics, and §4.4 gating. Not client tasks |
49+
| Rules amendments | Video forms of §9.1 metric consistency, §4.1 metrics, and §4.4 gating; a non-streaming exemption if the pending streaming change merges. Not client tasks |
4650

4751
## 1. Target shape
4852

4953
Per v1.0 §5.3 and §5.7.2, a non-agentic submission with an **elected** Offline result is **7
5054
runs**: 1 Ultra Low point, 3 mandatory points (Low, Medium, High), and 3 submitter's-choice points,
5155
with the `C_max` point also reported as the Offline result. Accuracy is required at the 4 mandatory
52-
points plus Offline; the elected `C_max` point counts for both, so **4 accuracy runs**.
56+
points plus Offline; the elected `C_max` point fills both roles, so the plan uses **4 accuracy
57+
runs**. A pending Rules TF patch (`nvashutoshd_v1.0_rules_patch`, `b36c220`) makes `N` = 5 an
58+
exact count checked automatically, without saying how an elected point counts; if it needs 2
59+
results, the plan needs 5 runs. Choice points must not carry accuracy results under that patch.
5360

5461
The planned point set, from the readiness plan: `C_min = 1`, `C_max = 72`, 1 x 72-GPU layout,
5562
points at concurrency 1, 2, 4, 8, 12, 36, 72; accuracy at 1, 4, 12, 72.
@@ -63,24 +70,6 @@ points at concurrency 1, 2, 4, 8, 12, 36, 72; accuracy at 1, 4, 12, 72.
6370
- **§4.4**: a steady-state result needs a window of **≥ 4 super-passes**, so a run of **more than
6471
4**, and a gating metric that is a Plateau. Otherwise the point reports `total`.
6572

66-
```mermaid
67-
flowchart LR
68-
W["declared warmup<br/>not in data"] --> R["SP1: ramp crop<br/>at least 1"]
69-
R --> S1[SP2] --> S2[SP3] --> S3[SP4] --> S4[SP5]
70-
S4 --> D["drain<br/>excluded on issue time"]
71-
subgraph window["steady window: at least 4 SPs AND issue span at least 600 or 1200 s"]
72-
S1 & S2 & S3 & S4
73-
end
74-
```
75-
76-
```mermaid
77-
flowchart TD
78-
A{"gating metric<br/>has samples?"} -- no --> T["total is the official result<br/>(not found: validity pending ratification)"]
79-
A -- yes --> B{"window at least<br/>4 super-passes?"} -- no --> T
80-
B -- yes --> C{"window issue span<br/>meets §6.2?"} -- no --> T
81-
C -- yes --> S["steady-state metrics official<br/>total supplementary"]
82-
```
83-
8473
### 1.1 Metrics for video generation
8574

8675
Proposed, pending rule text; names match the results-repository plan:
@@ -179,7 +168,8 @@ None of these block a T2V benchmark; they explain the §4 recipe.
179168
reports inflated `qps` and deflated latency. Check `n_samples_failed`.
180169
- **`stream_all_chunks`** is a plain `settings.client` flag (`endpoint_client/config.py:186`), read
181170
only on the SSE path (`endpoint_client/worker.py:429`); nothing ties it to `streaming`. Setting
182-
it in a video config meets §6.5 and the §9.1 streaming check as written.
171+
it in a video config meets §6.5 and the §9.1 streaming check as written at `cdb203c`; see the
172+
pending change noted at the top of this document.
183173
- **`compliance/checker.py`** is the Edge-Agentic (BFCL v4) checker, reachable only through
184174
`scripts/check_compliance.py`. It is not the Endpoints validator.
185175

0 commit comments

Comments
 (0)