Context
Two check-authoring mistakes cost real runs on 2026-07-13/14, and neither is covered by the current check-writing rules (skill/README). Both are judgment errors the harness can't fully prevent — but the docs can name them, and #38's --baseline makes the first one testable.
1. Don't demand evidence the spec doesn't supply
A review-swarm check required every report to cite a pinned spec file — including on the one lane deliberately given NO spec (the protocol has none). The worker correctly answered UNVERIFIABLE and was failed by the check. Rule for the docs: before failing a worker for a missing input, re-read the task's inputs — if the spec didn't supply it, the check can't demand it.
2. Executed checks catch laziness, not plausible-wrong — dogfood artifacts need own-validator tests
A fix-swarm lane silently deleted a spec-required field (serverInfo.version) while adding a similarly-named one. Every test and probe stayed green because nothing ran the repo's own validator against the repo's own artifact. Patch review caught it. Rule for the docs: for every dogfood artifact a swarm touches, an 'our own artifact passes our own validator' test is the cheapest permanent reviewer — and orchestrator patch review stays mandatory regardless of check status.
Size: S, docs only.
🤖 Generated with Claude Code
Context
Two check-authoring mistakes cost real runs on 2026-07-13/14, and neither is covered by the current check-writing rules (skill/README). Both are judgment errors the harness can't fully prevent — but the docs can name them, and #38's
--baselinemakes the first one testable.1. Don't demand evidence the spec doesn't supply
A review-swarm check required every report to cite a pinned spec file — including on the one lane deliberately given NO spec (the protocol has none). The worker correctly answered UNVERIFIABLE and was failed by the check. Rule for the docs: before failing a worker for a missing input, re-read the task's inputs — if the spec didn't supply it, the check can't demand it.
2. Executed checks catch laziness, not plausible-wrong — dogfood artifacts need own-validator tests
A fix-swarm lane silently deleted a spec-required field (
serverInfo.version) while adding a similarly-named one. Every test and probe stayed green because nothing ran the repo's own validator against the repo's own artifact. Patch review caught it. Rule for the docs: for every dogfood artifact a swarm touches, an 'our own artifact passes our own validator' test is the cheapest permanent reviewer — and orchestrator patch review stays mandatory regardless of check status.Size: S, docs only.
🤖 Generated with Claude Code