methodology: preregistered corrective Qwen3.6-vs-3.8 protocol (clean base) - #48
Draft
Lightheartdevs wants to merge 7 commits into
Draft
methodology: preregistered corrective Qwen3.6-vs-3.8 protocol (clean base)#48Lightheartdevs wants to merge 7 commits into
Lightheartdevs wants to merge 7 commits into
Conversation
…b2ae6 onto clean main base Audit finding A1: PR 47 branch was based on the exploratory results branch (fb3fa85). This branch rebuilds the methodology/campaign-support content directly on origin/main (eaaa8ca) by path-copy from methodology/qwen-corrective-2026-08 @ 06b2ae6: every path changed vs origin/main EXCEPT the results-only content (benchmarks/qwen36-vs-qwen38-27b-2026-08/**, claims.yaml, benchmarks/findings-index.md, KNOWN-LIMITATIONS.md, .gitattributes CSV pin). 116 paths; harness.py, run_microbench.sh, graders/v2, tasks/v2, fixtures verified sha256-identical to the live campaign tree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…branch The corrective entry references benchmarks/qwen36-vs-qwen38-27b-2026-08/ by relative path (PREREGISTRATION section 9, pins covered_elsewhere); those targets are intentionally absent on this clean branch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t-enumeration inference Audit findings A6 + A7. Amendment 1a: the conditional BF16 arm is a preregistered exception to the section 2 no-new-arms rule with an exact mechanical decision rule (|paired delivery delta| >= 15 pp AND exact McNemar p < 0.05 within either model between Q4 and Q8), fixed N (12 families x 2 seeds x 2 models = 48 cells), identical analysis, and the claim-language constraint that a triggered BF16 arm reopens rather than settles the quant question until it completes. Amendment 1b: family-cluster randomization inference by EXACT enumeration of all 2^12 = 4096 family sign assignments (2^11 = 2048 if p3_market is demoted per section 3), superseding the 10,000-permutation wording in section 6; p-values exact, no sampling. Insertion-only: PREREGISTRATION.md is not edited; the amendment lists the superseded sentences by section. Committed before any quant-pilot observation: verified firsthand that zero quant-pilot cells exist in the campaign ledgers (arms present: official-nothink, official-think, diag-t03 only). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ving fixture policy) Audit finding A5. Fixture bytes are pinned by sha256 in manifest/pins-2026-08-16.json (fixtures_p3_market), so whitespace hygiene checks and text/EOL transformation must not apply to the snapshot corpus; any byte change must surface as a manifest sha mismatch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ontainerized cleanup (audit A4) - all three dryrun_* scripts now derive REPO from their own location instead of the hardcoded campaign-host checkout, so they operate on the tree they ship in (CI runners included) and can never touch the live campaign tree - dryrun_request_body.sh: scrubs its root-owned sandbox workspace output inside a container mounting only tooling/workspace (no sudo anywhere), pre-cleans stale evidence dirs so relocation cannot nest, and ends with an explicit assertion line (REQUEST_BODY_PLUMB_VERIFIED) reached only after a regression gate proves post-run cleanliness: throwaway dirs gone, git status --porcelain --ignored=matching empty for logs/ and tooling/workspace/, zero root-owned files left; evidence dir overridable via DRYRUN_OUT - dryrun_argv_capture.sh / dryrun_v2_wiring_capture.sh: same post-run cleanliness assertion (these are the hermetic CI dry-runs) - verified on the campaign host against this branch worktree: all three exit 0 ending in their final assertion lines; the pre-existing root-owned throwaway workspace left by the 2026-08-16 dry-run was removed the same container way and verified gone Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… A3) .github/workflows/verify-methodology.yml runs on push + pull_request for the methodology paths and gates: - the full grader-v2 + loop-terminator + delivery-validator + evidence-manifest unit suites (65 tests, floor asserted at the audited 64) - fixture-mirror determinism: serve_fixtures.py started in the runner, check_fixture_determinism.py must pass twice (36 snapshots + catalog pages, sha256s vs index.json) - syntax gates: py_compile over all tooling python, bash -n over all tooling shell - the hermetic A4 dry-runs (argv-capture + v2-wiring); dryrun_request_body.sh is excluded with reasons in the workflow header (needs the campaign-host bench-sandbox image + docker socket) - hygiene: git diff --check against the merge-base with main under the .gitattributes fixture-pin policy, plus a guard failing the build if the held exploratory results entry benchmarks/qwen36-vs-qwen38-27b-2026-08/ exists on this branch PREREGISTRATION-AMENDMENT-1.md gains Amendment 1c (insertion-only, in the style of 1a/1b): corrects the section 8 present-tense claim that the evidence manifest and delivery validator are CI-enforced via make verify-study - no such target exists yet. 1c enumerates exactly what verify-methodology.yml enforces now and commits the full verify-study reproduction gate to Phase D, wired before any results PR merges. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…DEVIATIONS 1+2) - delivery_validator: split api_error handling - HTTP 400 exceed_context_size with >=1 model turn is now the terminal MODEL classification "context-exhausted" (delivery false, no quarantine, no rerun); all other api_errors (connection refused/reset, 5xx, timeouts) stay infra. Unit tests cover both directions (suite 65 -> 69 tests, CI floor 64). - harness: scrub root-owned per-run workspace residue inside an alpine container (audit A4b cleanup pattern, never sudo) before reuse, so a genuine infra failure can actually rerun. Pre-fix, the unprivileged rm -rf check=True raised CalledProcessError on root-owned sandbox output and killed every rerun attempt in ~2s with zero model turns. - DEVIATIONS.md: deviation 1 (classification clarification; the original misclassification FAVOURED q38 by suppressing a real q38 delivery failure) and deviation 2 (rerun-path bug + fix, attempts 2-4 void-infra-misfire), with the restored attempt-1 outcome record of p3_market_q38-diag-t03-nothink-s101_v1 and the rerun-ledger correction committed under deviations/. Zero confirmatory-arm cells affected (144 official rows, zero api_error). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clean-base corrective methodology branch
Supersedes draft PR #47 per the independent Codex stop-ship audit. PR #47 was based on the
exploratory results branch head (
fb3fa85), so it dragged the entire exploratory-resultshistory; audit finding A1 requires the methodology to enter review from a clean branch off
main. This branch is built directly onorigin/main(eaaa8ca) by path-copy frommethodology/qwen-corrective-2026-08@06b2ae6d.Audit findings remediated here
origin/mainEXCEPT the results-only content(
benchmarks/qwen36-vs-qwen38-27b-2026-08/**,claims.yaml,benchmarks/findings-index.md,KNOWN-LIMITATIONS.md, the results.gitattributesCSV pin) — 116 paths. The campaign-critical files (
tooling/harness.py,tooling/scripts/run_microbench.sh,tooling/graders/v2/**,tooling/tasks/v2/**,tooling/fixtures/**; 62 tracked files)are verified sha256-identical to the live campaign working tree, so the manifest pins
(
manifest/pins-2026-08-16.json) remain valid on this branch.tooling/fixtures/p3_market/**marked-diff -text(binary/generated fixture policy, hash-preserving); bytes are pinned by sha256 in the
manifest, so whitespace-hygiene checks must not apply.
git diff --checkpasses across thefull branch range.
PREREGISTRATION-AMENDMENT-1.md(Amendment 1a)makes the conditional BF16 arm a preregistered exception to the no-new-arms rule with an
exact mechanical decision rule (|paired delivery delta| >= 15 pp AND exact McNemar p < 0.05
within either model between Q4 and Q8), fixed N (12 families x 2 seeds x 2 models = 48
cells), identical analysis, and the claim-language constraint that a triggered BF16 arm
reopens rather than settles the quant question until it completes.
enumeration of all 2^12 = 4096 family sign assignments (2^11 = 2048 if
p3_marketisdemoted per protocol section 3); p-values are exact, no sampling. Insertion-only:
PREREGISTRATION.mdis not edited; superseded sentences are listed by section.commit time that zero quant-pilot cells exist in the campaign ledgers (arms present:
official-nothink,official-think,diag-t03only).Related
as insertion-only commit
a1ae6ac—findings.md/AUDIT.mdbyte-identical to theirmethodology-branch versions. PR benchmarks: Qwen3.6-27B vs Qwen3.8-27B head-to-head (802-cell freeze) — no-think delivery regression, grader-defect overlay, five retractions #46 stays draft/held.
BRANCH-NOTE.mdrecords that the exploratory entry referenced by relative path from thecorrective entry lives on the PR benchmarks: Qwen3.6-27B vs Qwen3.8-27B head-to-head (802-cell freeze) — no-think delivery regression, grader-defect overlay, five retractions #46 branch, not here.
The live campaign continues to run from the
methodology/qwen-corrective-2026-08workingtree at
06b2ae6d; this branch carries byte-identical campaign-critical assets, so resultsingest onto this base without re-running anything.
🤖 Generated with Claude Code