Make the eval harness runnable, then run it

[Defaultuser361]

The harness in scripts/ is carefully built — paired conditions, blind judging, a weighted rubric, a release gate, resumable runs, and deliberate isolation from operator config — and as far as I can tell it has never been run: no results committed, no numbers in the README, no workflow invoking it.

I wanted to know whether the skill actually works, so I tried to run it. This is what that took, and what it found.

Why it couldn't be run as-is

score aggregates rows a human has already written. There is no judge, so a 3-trial pass means hand-scoring 84 responses across five dimensions — 420 numbers, blind. That is the step standing between the harness and an answer.

scripts/judge.py fills it in. It grades a whole (case, trial) group in one call so conditions are compared rather than scored in isolation, and blinding is structural instead of a convention the grader is asked to respect: conditions are relabelled A/B/C with the order permuted per group from a digest of the group key, so position carries no signal and a resumed run reproduces its earlier labels. Stdlib only, matching the existing scripts.

Four defects, three of them only visible once it executes

The runner adopted this repo as its working directory. The subprocess inherited cwd=ROOT, so the agent CLI answered prompts by inspecting the harness. One baseline response opened by identifying scripts/run_evals.py; another degenerated into narrating tool calls for tools it did not have. Over a 14-case pass it hit 8 baseline responses and 2 candidate ones. The asymmetry is the problem — it favours whichever condition discourages exploration, so it lands on the number under test. The runners already pass --setting-sources "" for precisely this reason; the working directory was an unguarded channel for the same class of leak.

The eval graded text that never ships. _condition_prompt injected SKILL.md verbatim, frontmatter included, so the candidate prompt carried disable-model-invocation: true and the hermes metadata. hooks/always-on.sh strips that before injecting. The new _strip_frontmatter mirrors the hook's awk — byte-identical output on the current SKILL.md.

rubric.md leaked the blinding vocabulary. It ends with "Release the candidate only when… higher than baseline". Feeding the whole document to a blind grader hands over the exact words the blinding exists to hide, plus release-gate rules the grader has no business seeing. Now delimited with <!-- judge:begin --> / <!-- judge:end -->. Worth noting this got through every unit test and was caught only by the end-to-end one.

--tools "" ate the judge prompt. Appending the prompt as a trailing argument let the empty-valued flag consume it. Generation only escapes this because --max-budget-usd happens to sit in between — remove the budget flag from a runner and it breaks the same way. The judge passes the prompt on stdin.

Each fix ships with a test that fails against the previous behaviour; the working-directory one uses a stub runner that just reports its own pwd. CI now runs the suite (stdlib unittest, no new dependency) — tests/ was not referenced by any workflow, so the scoring maths and release gate could regress with CI green.

Result

14 cases, 3 trials, claude-opus-4-8, Claude Code 2.1.220. Full numbers and caveats in evals/RESULTS.md.

Dimension	Weight	Baseline	Candidate	Δ
Correctness	35%	4.333	4.524	+0.190
Autonomy	25%	3.762	4.167	+0.405
Actionability	20%	3.905	4.619	+0.714
Safety	10%	4.643	4.667	+0.024
Concision	10%	3.429	4.571	+1.143
Weighted		4.045	4.473	+0.427

Blocking findings 7 → 3. The candidate wins 10 of 14 cases, ties 2, loses 2.

Every dimension moves in the candidate's favour, including the two that would expose a style rule buying brevity with substance. The gains concentrate in cases about reporting state — multi-step-progress +2.53 and error-report +2.40 — while cases with an explicit output contract (code-answer, long-form-request) are unchanged, which is what the escape-hatch clauses are supposed to do.

It still fails your release gate, on rule 1. That rule is absolute where its neighbours are comparative, so more than halving the blocker count is not enough to pass. Two of the three remaining come from a case nothing can pass (see the companion issue). Whether that asymmetry is intended seems worth deciding deliberately rather than discovering at a release.

Caveats are in RESULTS.md and I would not want them skipped: three trials is few (per-case SD reaches 0.95), the judge shares a model family with the responses it grades, and 3 of 84 responses still carry tool-call syntax as plain text because the CLI primes tool use even with --tools "".

Notes

• Happy to drop the RESULTS.md commit if you would rather generate the numbers yourself — the other three stand on their own, and the harness is what matters.
• Also happy to split this into separate PRs. I kept it together because the fixes are what running it found, and they read better as one story, but it is your call.
• Two case-design issues came out of this that change what the eval measures rather than how it runs, so they are filed separately rather than patched here.

🤖 Generated with Claude Code