derive_counters.py --check gates absolute totals but not delta claims, and never reads CHANGELOG.md

[alirezarezvani]

Summary

scripts/derive_counters.py --check is a strong gate on absolute totals in five named files, but two whole classes of counter prose sit outside it. A wrong counter can be written into CHANGELOG.md or a CLAUDE.md release note and pass CI green, every time.

This surfaced concretely in #993: the changelog and the CLAUDE.md release note both claimed commands 147 → 150 for a plugin that adds exactly one command. The gate passed. It was caught by a human/bot reviewer reading the prose, not by CI.

Reproduction

Appending a fabricated delta to CHANGELOG.md and running the gate on an otherwise clean, correct tree:

$ printf '\nCounters: skills 999 → 1234; commands 12 → 999.\n' >> CHANGELOG.md
$ python3 scripts/derive_counters.py --check
Counter check passed: README.md, CLAUDE.md, marketplace.json, mkdocs.yml, .codex-plugin match derived values.

Green, with skills 999 → 1234 sitting in the changelog.

The two gaps

1. File coverage. The gate reads README.md, root CLAUDE.md (only its Current Scope / Status: lines), marketplace.json metadata.description, mkdocs.yml site_description, and .codex-plugin/plugin.json descriptions. CHANGELOG.md is not read at all, and CLAUDE.md's release-note sections — which carry a counter line per release — are outside the two checked lines.

2. Claim shape. CLAIM_PATTERNS recognises seven fixed phrasings of an absolute total:

"skills":  r"(\d+)\s+production-ready (?:Claude Code )?skills"
"commands": r"(\d+)\s+slash commands"
...

Nothing matches a delta of the form X → Y, in any file — including the five that are checked. Since release notes and changelog entries state deltas rather than totals by convention, the dominant form of counter prose in this repo is the one form the gate cannot see.

Token-budget figures (2,101 / 4,000 tokens) are outside the vocabulary too, and the comma grouping would defeat (\d+) even if a pattern existed. #993 shipped CLAUDE.md claiming 2,066 on one line and 2,101 twelve lines later, about the same file.

Why deltas are the higher-risk form

An absolute total is wrong only if it is wrong. A delta is a pair — old → new — and stays silently wrong when only one half is updated. Both #993 defects were exactly that:

• 147 → 150: the branch merged dev, the baseline moved 147 → 149, and three of the four figures on the line were updated while commands was carried through. A half-rebased delta.
• 2,066 → 2,101: the number was correct when measured, then a later commit changed the file and one of the two copies was updated.

Neither is a typo. Both are the predictable result of hand-editing a derived number in more than one place.

Sketch of a fix

Roughly in order of value per effort:

1. Add a delta pattern — recognise <counter> A → B (and A -> B), then assert B equals the derived value. A can't be checked against the tree (it's a historical baseline), but asserting the right-hand side alone would have caught 147 → 150.
2. Widen file coverage to CHANGELOG.md and all of CLAUDE.md, not just its two headline lines.
3. Generate rather than validate. The reviewer on feat(engineering): compile OpenAI's Spinning Up in Deep RL into a knowledge-base plugin #993 put it as: assert counters from a single generated source rather than copy-pasting across files. A derive_counters.py --write that rewrites the marked spans, with --check verifying the rendered output, removes the drift class instead of detecting it.

Option 1 alone closes the specific hole that shipped. Option 3 is the real fix and is a larger change.

Prior art in this repo

The gate already grew twice in response to live escapes — the multi-occurrence scan was added after PR #985 shipped a README banner saying 380 while a section heading still said 370, and the badge and per-domain-table checks were added for the same reason. This is the same failure mode one level out: correct derivation, hand-typed prose.

Scope note

Deliberately not fixed in #993. That PR was a skill compilation plus an emitter bug fix; changing the repo-wide counter gate inside it would have widened the diff well past its subject. Filing separately as agreed.