You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3ecbfdc
Browse filesBrowse the repository at this point in the historyBrowse files
docs: note pending rules changes and drop the section 1 charts
A pending streaming-check change (d9e1ea6) would block every T2V point; a pending Rules TF patch (b36c220) makes the accuracy count exact without saying how an elected Offline point counts. Remove the point timeline and official-result charts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/t2v_benchmark_gaps.md
+11-21Lines changed: 11 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -30,6 +30,10 @@ result, and accuracy runs. This document covers what the client must change so a
30
30
blocker only if the working group (WG) rules a `not found` run invalid, which §4.4 lists as
31
31
pending ratification.
32
32
-`stream_all_chunks = true` (§6.5) is a plain client flag that a video config can set (§3.4).
33
+
A pending rules change (`arekay/streaming_chunks_client_false`, `d9e1ea6`) would instead require
34
+
`model_params.streaming` to resolve to `on` for every fixed-concurrency perf run. The video
35
+
adapter cannot stream, so that would block every T2V point until the rule exempts
36
+
non-streaming classes.
33
37
34
38
**Not client work:** the §9.1 metric consistency check requires a TPOT distribution and rejects a
35
39
submission at Week 0. That needs WG rule text, tracked in the NVIDIA readiness plan.
@@ -42,14 +46,17 @@ submission at Week 0. That needs WG rule text, tracked in the NVIDIA readiness p
42
46
|[Tokenless support](#b-tokenless-support)| Artifact-safe tokenizer handling; steady-state gating without TPOT, if the WG requires it |
43
47
|[Run settings](#c-run-settings)| A config setting for with-replacement sample order on perf runs |
44
48
|[Example configs](#d-example-configs)| Point configs sized to whole passes, with seeds bound and no binding time cap |
45
-
| Rules amendments | Video forms of §9.1 metric consistency, §4.1 metrics, and §4.4 gating. Not client tasks |
49
+
| Rules amendments | Video forms of §9.1 metric consistency, §4.1 metrics, and §4.4 gating; a non-streaming exemption if the pending streaming change merges. Not client tasks |
46
50
47
51
## 1. Target shape
48
52
49
53
Per v1.0 §5.3 and §5.7.2, a non-agentic submission with an **elected** Offline result is **7
50
54
runs**: 1 Ultra Low point, 3 mandatory points (Low, Medium, High), and 3 submitter's-choice points,
51
55
with the `C_max` point also reported as the Offline result. Accuracy is required at the 4 mandatory
52
-
points plus Offline; the elected `C_max` point counts for both, so **4 accuracy runs**.
56
+
points plus Offline; the elected `C_max` point fills both roles, so the plan uses **4 accuracy
57
+
runs**. A pending Rules TF patch (`nvashutoshd_v1.0_rules_patch`, `b36c220`) makes `N` = 5 an
58
+
exact count checked automatically, without saying how an elected point counts; if it needs 2
59
+
results, the plan needs 5 runs. Choice points must not carry accuracy results under that patch.
53
60
54
61
The planned point set, from the readiness plan: `C_min = 1`, `C_max = 72`, 1 x 72-GPU layout,
55
62
points at concurrency 1, 2, 4, 8, 12, 36, 72; accuracy at 1, 4, 12, 72.
@@ -63,24 +70,6 @@ points at concurrency 1, 2, 4, 8, 12, 36, 72; accuracy at 1, 4, 12, 72.
63
70
-**§4.4**: a steady-state result needs a window of **≥ 4 super-passes**, so a run of **more than
64
71
4**, and a gating metric that is a Plateau. Otherwise the point reports `total`.
65
72
66
-
```mermaid
67
-
flowchart LR
68
-
W["declared warmup<br/>not in data"] --> R["SP1: ramp crop<br/>at least 1"]
69
-
R --> S1[SP2] --> S2[SP3] --> S3[SP4] --> S4[SP5]
70
-
S4 --> D["drain<br/>excluded on issue time"]
71
-
subgraph window["steady window: at least 4 SPs AND issue span at least 600 or 1200 s"]
72
-
S1 & S2 & S3 & S4
73
-
end
74
-
```
75
-
76
-
```mermaid
77
-
flowchart TD
78
-
A{"gating metric<br/>has samples?"} -- no --> T["total is the official result<br/>(not found: validity pending ratification)"]
79
-
A -- yes --> B{"window at least<br/>4 super-passes?"} -- no --> T
80
-
B -- yes --> C{"window issue span<br/>meets §6.2?"} -- no --> T
81
-
C -- yes --> S["steady-state metrics official<br/>total supplementary"]
82
-
```
83
-
84
73
### 1.1 Metrics for video generation
85
74
86
75
Proposed, pending rule text; names match the results-repository plan:
@@ -179,7 +168,8 @@ None of these block a T2V benchmark; they explain the §4 recipe.
179
168
reports inflated `qps` and deflated latency. Check `n_samples_failed`.
180
169
-**`stream_all_chunks`** is a plain `settings.client` flag (`endpoint_client/config.py:186`), read
181
170
only on the SSE path (`endpoint_client/worker.py:429`); nothing ties it to `streaming`. Setting
182
-
it in a video config meets §6.5 and the §9.1 streaming check as written.
171
+
it in a video config meets §6.5 and the §9.1 streaming check as written at `cdb203c`; see the
172
+
pending change noted at the top of this document.
183
173
-**`compliance/checker.py`** is the Edge-Agentic (BFCL v4) checker, reachable only through
184
174
`scripts/check_compliance.py`. It is not the Endpoints validator.
0 commit comments