Publish Gemma 4 31B Q4 verified Tower2 results

COMPARISON.md

@@ -1,16 +1,33 @@
-# Local-model head-to-head — Coder-Next vs 27B (thinking) vs 27B (no-think)
+# Local-model head-to-head — Gemma 4 31B, Coder-Next, and Qwen3.6-27B
 
 > **Five-minute decision doc.** The detail lives in [`SCORECARD.md`](SCORECARD.md) and the per-benchmark `findings*.md` docs; this page is the synthesis. Every claim links to its source so you can drill straight into the evidence.
 >
 > **Read [`KNOWN-LIMITATIONS.md`](KNOWN-LIMITATIONS.md) before quoting any cell.** Most caveats live there, not here.
 >
-> **Last updated**: 2026-05-02 — reflects [`microbench-phase-b-2026-05-02`](benchmarks/microbench-phase-b-2026-05-02/) (N=10 + 27B-no-think third arm). Pre-no-think readers: the picture has shifted.
+> **Last updated**: 2026-08-02 — adds the complete [`Gemma 4 31B Q4 campaign`](benchmarks/gemma4-31b-q4/) while preserving the original three-AWQ-arm tables below.
 
-> **Operating point**: All arms are **Cyankiwi 4-bit AWQ** on **2× RTX PRO 6000 Blackwell at 500 W cap**. Other quants, VRAM tiers, hardware classes, and languages are **not characterized** — see [What this benchmark doesn't characterize](#what-this-benchmark-doesnt-characterize) below. The within-quant comparison here is informative; absolute model capability at higher precisions is a separate question.
+> **Operating point**: The original three arms are **Cyankiwi 4-bit AWQ** on **2× RTX PRO 6000 Blackwell at 500 W cap**. Gemma is Google's official QAT Q4_0 GGUF under llama.cpp at model-card sampling and native 262,144-token context. Other quants, VRAM tiers, hardware classes, and languages are **not characterized** — see [What this benchmark doesn't characterize](#what-this-benchmark-doesnt-characterize) below.
+
+## 2026-08-02 Gemma addendum
+
+Gemma changes the bounded-quality recommendation, but not the marathon warning:
+
+| Question | Current evidence-based choice | Evidence |
+|---|---|---|
+| Highest bounded quality versus Qwen3.6-27B | **Gemma 4 31B Q4** | 29/36 raw and 32/36 corrected at N=3 versus Qwen thinking's 20/36 raw |
+| One interactive user | approximately tied | 70.3 tok/s Gemma versus 72.1 tok/s Qwen at short context and 500 W |
+| Many simultaneous users | **Qwen3.6-27B/vLLM** | Qwen 1,336.5 aggregate tok/s at C32 on one GPU; Gemma 290.3 at total C8 across two GPUs; shapes differ but the operational gap is large |
+| Reliable ordinary completion | approximately tied | Gemma 116/120 `done_signal`; Qwen no-think 113/118 published valid outcomes |
+| Unattended marathon work | **none** | Gemma and Qwen both score zero strict passes on their published frozen 75-PR evidence |
+
+Qwen's 113/118 is not a quality score: its published phase-B grader sweep was
+pending. Gemma's 99/120 corrected figure is a quality result, with the raw
+89/120 preserved separately. The full Gemma comparison and artifact audit are
+in [`benchmarks/gemma4-31b-q4/GEMMA4_31B_Q4_VERIFIED_RESULTS.md`](benchmarks/gemma4-31b-q4/GEMMA4_31B_Q4_VERIFIED_RESULTS.md).
 
 ## TL;DR
 
-**No model is overall best.** The three arms have orthogonal strengths and statistically indistinguishable headline ship rates (74–96%). Pick by task class:
+**No model is overall best.** Within the original three AWQ arms, the models have orthogonal strengths and statistically indistinguishable headline ship rates (74–96%). Pick by task class; include Gemma when bounded answer quality matters more than dense batching:
 
 - **Default for most non-coding tasks**: **27B-no-think** — 95.8% ship rate across 12 cells × N=10, beats both originals on raw shipping
 - **Hallucination-sensitive or research-driven work**: **27B (either mode)** — 27B-thinking is the only arm that ships market-research at >70%; no-think 10/10 on adversarial-hallucination

MICROBENCH-INDEX.md

@@ -13,6 +13,7 @@
 
 | Entry | Tree | Models / arms | N | Headline |
 |---|---|---|---|---|
+| [`gemma4-31b-q4`](benchmarks/gemma4-31b-q4/) | benchmarks/ | **Gemma-4-31B-it-QAT-Q4_0** at model-card sampling | 3 and 10 | N=3 raw 29/36, corrected 32/36; N=10 raw 89/120, corrected 99/120. Extended strict audit 0/12. |
 | [`deepseek-v4-flash-0731`](benchmarks/deepseek-v4-flash-0731/) | benchmarks/ | **DeepSeek-V4-Flash-0731** at its model-card sampling point | 3 | Raw 23/36; corrected 35/36 after reproducible grader defects were repaired without rerunning the model. One genuine failure: 773 words against a 700-word cap. |
 | [`microbench-2026-04-28`](benchmarks/microbench-2026-04-28/) | benchmarks/ | Qwen3.6-**27B-AWQ** vs Qwen3-Coder-Next-**AWQ** | 3 | Aggregate-tied ~7/12 each; complementary task-class strengths; Coder-Next much faster/cheaper. |
 | [`microbench-phase-b-2026-05-02`](benchmarks/microbench-phase-b-2026-05-02/) | benchmarks/ | + **27B-AWQ no-think** third arm; 4 differential cells to N=10 | 10 | 27B ships 86.8% no-think vs 75% think (same `p3_doc` word-limit loop); Coder-Next market 0/10 (Wilson [0, 27.8%]). |
@@ -49,6 +50,16 @@ Qwen3.5-397B writing overlay moves its N=10 no-think result from 82/120 to
 regrade and DeepSeek N=10 expansion would be required for a statistically
 matched ranking.
 
+## Gemma adds a broader high-scoring local cohort
+
+Gemma's corrected **99/120 (82.5%)** at N=10 is the largest high-scoring
+canonical cohort in the current local set, and its N=3 corrected 32/36 is below
+DeepSeek's 35/36 but above Qwen3.6-27B and Coder-Next's 20/36 raw results. This
+still is not a global leaderboard: Gemma uses different sampling and a newer
+grader/campaign date. Its strict extended result is 0/12, which is why the
+canonical aggregate must not be read as proof of reliable complex-artifact or
+marathon execution.
+
 ## Qualitative comparison
 
 Most historical pass-rates tie; the qualitative layer is where those models actually differ. Provisional cross-model

README.md

@@ -15,6 +15,7 @@ but I'm making it public so that other people can use it too.
 | Where the benchmark folders start | [`benchmarks/README.md`](benchmarks/README.md) — agent-task benchmark landing page |
 | **"Coder-Next or 27B (or 27B-no-think) for my task?"** | [`COMPARISON.md`](COMPARISON.md) — head-to-head decision doc |
 | The full single-table comparison across all entries | [`SCORECARD.md`](SCORECARD.md) |
+| **Gemma 4 31B QAT Q4: complete verified campaign** | [`benchmarks/gemma4-31b-q4/`](benchmarks/gemma4-31b-q4/) — native-256K serving, canonical N=3/N=10, strict artifact audits, and Qwen3.6-27B comparison |
 | **DeepSeek V4 Flash 0731: complete verified campaign** | [`benchmarks/deepseek-v4-flash-0731/`](benchmarks/deepseek-v4-flash-0731/) — deployment, canonical N=3, extended suites, strict artifact audits, and full-context 75-PR outcomes |
 | **All 12-family microbench results (across both trees) + the four "27B"s** | [`MICROBENCH-INDEX.md`](MICROBENCH-INDEX.md) — cross-tree microbench index + quant disambiguation |
 | Cross-model **qualitative** spot-grades (provisional, not a ranking) | [`QUALITATIVE-SPOT-GRADES.md`](QUALITATIVE-SPOT-GRADES.md) + [`tooling/QUALITATIVE-GRADING-PROTOCOL.md`](tooling/QUALITATIVE-GRADING-PROTOCOL.md) — single-grader provisional scores + grader-independence rules |
@@ -24,7 +25,7 @@ but I'm making it public so that other people can use it too.
 
 ## Operating point (read before quoting)
 
-Most earlier agent-task benchmarks under [`benchmarks/`](benchmarks/) use **Cyankiwi 4-bit AWQ** quants on **2x RTX PRO 6000 Blackwell at 500 W cap**. The DeepSeek V4 Flash 0731 entry is an explicit exception: official FP4 weights, FP8 KV, model-card sampling, and a validated 1,048,576-token context. Every entry README pins its own operating point. Other quants, VRAM tiers, hardware classes, and languages other than Python are **not characterized** unless an entry says otherwise. See [`COMPARISON.md` section "What this benchmark doesn't characterize"](COMPARISON.md#what-this-benchmark-doesnt-characterize) for the model-benchmark validity boundaries, and [`ROADMAP.md`](ROADMAP.md) for what's queued to fill those gaps.
+Most earlier agent-task benchmarks under [`benchmarks/`](benchmarks/) use **Cyankiwi 4-bit AWQ** quants on **2x RTX PRO 6000 Blackwell at 500 W cap**. DeepSeek V4 Flash 0731 and Gemma 4 31B are explicit exceptions: DeepSeek uses official FP4 weights, FP8 KV, and a validated 1,048,576-token context; Gemma uses Google's official QAT Q4_0 GGUF, Q8 KV, llama.cpp, and a validated native 262,144-token context per slot. Every entry README pins its own operating point. Other quants, VRAM tiers, hardware classes, and languages other than Python are **not characterized** unless an entry says otherwise. See [`COMPARISON.md` section "What this benchmark doesn't characterize"](COMPARISON.md#what-this-benchmark-doesnt-characterize) for the model-benchmark validity boundaries, and [`ROADMAP.md`](ROADMAP.md) for what's queued to fill those gaps.
 
 Rig-characterisation studies under [`hardware-tests/`](hardware-tests/) have their own operating-point scope. Start with [`hardware-tests/README.md`](hardware-tests/README.md) before quoting hardware claims. In particular, [`hardware-tests/qwen3.6-q8-fleet-2026-05-17/`](hardware-tests/qwen3.6-q8-fleet-2026-05-17/) ranks four hardware classes on **Q8_0 GGUF** dense and MoE workloads under llama.cpp, with a Tower2 vLLM-FP8 appendix row for the MoE model the llama.cpp/CUDA path crashes on.
 
@@ -33,6 +34,7 @@ Rig-characterisation studies under [`hardware-tests/`](hardware-tests/) have the
 ```text
 benchmarks/
   README.md                    agent-task benchmark landing page and navigation map
+  gemma4-31b-q4/               complete Gemma campaign, N=3/N=10, strict audits, comparisons
   deepseek-v4-flash-0731/      cross-suite DeepSeek campaign, strict audits, and deployment evidence
   dreamserver-75-pr-audit/
     GPT-5.5/                   cloud, full audit
@@ -63,6 +65,7 @@ hardware-tests/
 
 | Benchmark | Prompt Shape | Model Entries |
 |---|---|---|
+| [`gemma4-31b-q4`](benchmarks/gemma4-31b-q4/) | Cross-suite publication: full 12-family N=3 and N=10, single-PR N=3, investment research, board presentation, and frozen 75-PR N=3. | `Gemma-4-31B-it-QAT-Q4_0`; includes immutable raw grades, a narrow corrected overlay, strict substantive audit, and Qwen3.6-27B comparison. |
 | [`deepseek-v4-flash-0731`](benchmarks/deepseek-v4-flash-0731/) | Cross-suite publication: full 12-family N=3, single-PR N=3, investment research, board presentation, and three valid full-context frozen 75-PR outcomes. | `DeepSeek-V4-Flash-0731`; includes corrected grader overlay and compact audit evidence. |
 | [`dreamserver-75-pr-audit`](benchmarks/dreamserver-75-pr-audit/) | Audit 75 open PRs in a live repository and produce a traceable maintainer triage repo. | `GPT-5.5`, `Opus-4.7`, `Qwen3.6-27B-AWQ`, `Qwen3-Coder-Next-AWQ` (failure-mode entry) |
 | [`dreamserver-1-pr-audit`](benchmarks/dreamserver-1-pr-audit/) | Same task spec, scaled to a single PR. Built as the floor of an escalation ladder (1 → 2 → 4 → 8 → 16 → 32) to find each model's complexity ceiling. | `Qwen3-Coder-Next-AWQ`, `Qwen3.6-27B-AWQ`, `Qwen3.6-35B-A3B-AWQ` (floor failure) |
@@ -93,6 +96,7 @@ For the benchmark landing page and per-folder navigation map, start with
 
 Two synthesis docs sit between this README and the per-entry detail:
 
+- [`benchmarks/gemma4-31b-q4/GEMMA4_31B_Q4_VERIFIED_RESULTS.md`](benchmarks/gemma4-31b-q4/GEMMA4_31B_Q4_VERIFIED_RESULTS.md) — full Gemma result, native-256K deployment, N=3/N=10 quality, strict artifact audit, and direct Qwen3.6-27B comparison.
 - [`benchmarks/deepseek-v4-flash-0731/DEEPSEEK_V4_FLASH_0731_VERIFIED_RESULTS.md`](benchmarks/deepseek-v4-flash-0731/DEEPSEEK_V4_FLASH_0731_VERIFIED_RESULTS.md) — full verified DeepSeek result, including corrected-vs-raw grading, strict artifact audits, production validation, and caveats.
 - [`COMPARISON.md`](COMPARISON.md) — **head-to-head decision doc** for the three local model arms (Coder-Next vs 27B-thinking vs 27B-no-think). Organized by task class with cell-level evidence. Read this if your question is "which one should I use?"
 - [`SCORECARD.md`](SCORECARD.md) — single-table summary across all entries (spec compliance, factual accuracy, fabricated-claim count, tests run, wall, cost upper bound, failure mode, "when to use which" guide). Read this if your question is "what's the full picture?"
@@ -128,6 +132,10 @@ The repo keeps the failures because the *kinds* of failure are themselves the co
 
 ## Current Entries
 
+**gemma4-31b-q4:**
+- [Verified campaign entry](benchmarks/gemma4-31b-q4/) — Gemma 4 31B QAT Q4_0 on two independent 500 W RTX PRO 6000 replicas, with native 262,144-token context per slot. Canonical N=10 is 89/120 raw and 99/120 corrected; extended strict result is 0/12. It beats Qwen3.6-27B on directly comparable bounded quality but not on batched serving.
+- [Completion audit](benchmarks/gemma4-31b-q4/GEMMA4_31B_Q4_COMPLETION_AUDIT.md) — requirement-to-evidence handoff, including the pending production restore gate.
+
 **deepseek-v4-flash-0731:**
 - [Verified campaign entry](benchmarks/deepseek-v4-flash-0731/) — DeepSeek V4 Flash 0731 on 2x RTX PRO 6000 at 500 W/GPU and 1,048,576 context. Canonical corrected result 35/36 (N=3); single-PR 2/3 expected verdicts with all three complete; investment workbooks 0/2 substantively valid; board deck shipped with material visual defects; frozen 75-PR strict result 0/3 (two scaffold-and-stop, one 815,279-token runaway-generation terminal failure).
 - [Completion audit](benchmarks/deepseek-v4-flash-0731/DEEPSEEK_V4_FLASH_0731_COMPLETION_AUDIT.md) — requirement-to-evidence handoff and artifact inventory.