benchmarks: Qwen3.6-27B vs Qwen3.8-27B head-to-head (802-cell freeze) — no-think delivery regression, grader-defect overlay, five retractions

[Lightheartdevs]

Summary

Head-to-head of Qwen3.6-27B and Qwen3.8-27B, both at Unsloth UD-Q4_K_XL, across the
12-family MMBT suite: 802 frozen cells (733 graded) over 9 sampler/mode arms, plus a
19-cell Q8_0 control. Every number derives from a single frozen extract
(data/mmbt-frozen-dataset-v2.csv, sha256 d2ed0bec…784cf018, stamped
2026-08-16T14:23:09Z) — the campaign is stopped, in-flight cells are quarantined, and
the corpus is static.

The one large, unambiguous effect is a delivery regression, not a capability
regression: 3.8's identical-call-loop rate in no-think mode is +28.0 pp over 3.6 at the
matched sampler (Fisher p = 4.9e-09) and +29.2 pp at the vendor points (p = 1.1e-11),
measured upstream of all grading. The regression disappears in think mode and reproduces
at Q8_0 (6/19, Wilson 95% CI excludes zero), so it is not an artifact of the 4-bit quant.
Conditional on delivering, and after grader corrections, the models are within noise at
the matched sampler (86.0% vs 86.2% no-think; 92.5% vs 89.6% think); the ~15 pp
vendor-point conditional gap survives correction and is left unresolved. Thinking costs
3.8 roughly 2.2–3.0x more tokens with no pass-rate gain.

The campaign also audited MMBT's own graders and found three defects that materially
distort raw numbers. Corrections ship as a non-destructive overlay (165 per-cell records,
digest e332cd2c…); no grade.json, task brief, or ground-truth file is modified.

What's in the entry (benchmarks/qwen36-vs-qwen38-27b-2026-08/)

• findings.md — the house-register entry: top-line, provenance, delivery/as-graded/
  corrected tables, cost, Q8_0 control, three qualitative exhibits, grader defects in one
  screen, retractions, what-we-did-not-run, reproduction instructions.
• grader-defects.md — full defect report: D1 word-gate tokenizer mismatch (62
  counter-dependent verdicts gate-invalidated), D2 p3_pm keyword literalism (36 verdict
  flips), D3 p2_triage brief/ground-truth contradiction (24 flips), with natural
  experiments, per-cell appendix, and prior-campaign exposure.
• results-tables.md + results.json — full statistical tables, rendered
  deterministically by tooling/build_results.sh.
• data/ — the frozen CSV + freeze stamp. .gitattributes pins the CSV binary so the
  stamped sha256 survives the repo-wide *.csv text eol=lf rule (the extract is CRLF by
  construction; the sha covers those bytes).
• overlay/ — manifest + 165 grade.corrected.json records mirroring the campaign log
  layout. D2/D3 records carry corrected_verdict; D1 records carry gate_invalidated
  only (counter-dependent, not a verified PASS).
• tooling/ — freeze extractor, overlay generator + verifier, results generators,
  validation harness. Stdlib-only Python.

Top-level integrations

• benchmarks/findings-index.md — new newest-first row.
• claims.yaml — 7 new claims (all provisional or held) and 5 new retractions; now 45
  claims / 7 retracted, ids unique file-wide. Retracted: the operator-label "abort rate",
  both defect-family results (p3_pm, p2_triage), the 9–3 family win split, and the
  "default vs default" thinking comparison.
• KNOWN-LIMITATIONS.md — 7 new subsections (freeze-extract provenance, replicate depth,
  shared-seed replicates, missing 3.8-vendor think arm, empty hand-rating slots,
  p3_market non-replayability, quantization specificity).

Campaign harness files (pre-existing work this investigation ran on)

• tooling/bench_autopilot.py — cleanup scoping fix: docker sandbox wipe + workspace
  cleanup now scoped to the running arm's label instead of a global wipe that killed
  concurrent campaigns' live cells.
• tooling/cfg-offspec.json, tooling/qwen3.8-27b-q4-t03-matched-mmbt.json, two fleet
  ensure-*.sh deployment scripts.

Pre-push verification (independent, from the committed tree)

• Reproducibility: results.json + results-tables.md rebuilt from the committed
  tree alone (git-archive extract; tooling + data + overlay) are byte-identical to the
  committed artifacts after normalizing exactly three absolute-path provenance strings.
  The defect_diagnostics block additionally reads raw grade.json files that stay
  outside the repo by convention (/logs/ is never committed).
• Number spot-check: 12 independent recomputations from the committed CSV + overlay
  (own Wilson/Fisher implementations) — group inventory, delivery, loop rates, as-graded
  and corrected tables, defect counts, Q8_0 block, vendor per-family delivery, overlay
  digest, saturation and label-disagreement cross-tabs — all match the published values
  at quoted precision.
• Conventions: claims.yaml parses, ids unique across the whole file; no gitignored
  path tracked; no grade.json / receipt.json / task brief / ground-truth file
  modified anywhere in the diff; committed CSV blob hashes to the stamped sha256.
• Size: 190 files, ~1.6 MB total committed content; largest file 195 KB.
• Hygiene: freeze-1 numbers (746 cells, 11:49:14Z) survive only in explicit
  "Supersedes" context; no Welch-based stat, no stale group sizes, no
  corrected_verdict on D1-only cells; all entry docs are LF-clean.
• verify_overlay.py: idempotence, write-guard, leniency invariant 0 violations over
  686 cells, 853 protected files unmutated, empty post-freeze-divergence ledger.
  validate_fixes.py: 174 checks, 0 failures.

Note: the audit document (AUDIT.md) was not present in staging at assembly time and is
not part of this PR.

🤖 Generated with Claude Code