seed: add the 20 metrics the every_eval_ever harness converters publish (+ inspect_ai harness)

[borgr]

What / why

evaleval/every_eval_ever's three harness converters (lm-evaluation-harness, HELM, Inspect) now publish a metric_id on every result so a consumer can join the same metric across sources. Resolving their metric names against this registry at 8b83e9c left 22 of 45 with no entry, so they ship as <harness>.<name> — a stable join key inside one harness and no global identity. This PR closes 20 of those 22.

Everything here is a metric one of those converters publishes today. Each entry's metadata.source is the upstream definition it was read off — HELM's static/schema_*.yaml and metric modules, lm-evaluation-harness's api/metrics.py/api/task.py, sacrebleu's metric classes — and nothing was added because it looked like a plausible spelling.

16 new metrics

id	why it is not a spelling of something already here	source
quasi-exact-match	exact match after lowercasing, punctuation/article removal, whitespace fixing — a different and higher number than exact-match on the same run	schema_classic.yaml
prefix-exact-match	prediction starts with the reference; systematically ≥ exact-match	schema_classic.yaml
quasi-prefix-exact-match	the prefix rule after the quasi normalization	schema_classic.yaml
micro-f1	micro-averaged over classes; separate from f1 for the same reason macro-f1 already is	classification_metrics.py, schema_classic.yaml
math-equivalent	mathematical equivalence to the reference, not string equality	schema_classic.yaml
math-equivalent-chain-of-thought	the same under CoT prompting; HELM reports the two separately	schema_lite.yaml, schema_classic.yaml
ifeval-strict-accuracy	fraction of an instance's instructions followed, not of instances correct	schema_capabilities.yaml, ifeval_metrics.py
length-normalized-accuracy	lm-eval's acc_norm: argmax over loglikelihood / len(choice), a different argmax from acc	api/task.py
brier-score	squared error of a probability forecast, 0–2 (see below)	api/task.py
bits-per-byte	bits, unbounded above; not a perplexity	api/metrics.py
word-perplexity	exp(-weighted_mean(logprob)) normalized by words	api/metrics.py
byte-perplexity	same, normalized by bytes — tokenizer-independent, so not comparable with the word form or with perplexity	api/metrics.py
rouge-lsum	ROUGE-L computed after sentence splitting; differs from rouge-l on the same pair	rouge_score via evaluate
truthfulqa-mc1	mirrors the existing truthfulqa-mc2 entry, which had no MC1 counterpart	TruthfulQA; lm-eval v0.3 / lighteval stat name mc1
chrf	plain chrF, not the existing chrf-plus-plus — sacrebleu is chrF++ only at word_order=2 (see below)	sacrebleu.CHRF via api/metrics.py
ter	Translation Edit Rate, an error rate: lower_is_better: true, no ceiling	sacrebleu.TER via api/metrics.py

brier-score is 0 to 2, not 0 to 1. lm-eval computes mean(sum((softmax(loglikelihoods) - one_hot(gold)) ** 2)), so a model putting all its mass on one wrong class scores 2.0. The two-class definition people usually quote maxes at 1.0. A [0, 1] bound here makes a legitimate score look out of range.

Four carry max_score: null because they have no upper bound: bits-per-byte (from 0.0), word-perplexity / byte-perplexity (from 1.0, the perplexity of a perfect predictor), and ter (from 0.0 — a hypothesis can need more edits than the reference has words).

null is the spelling every unbounded entry in seed/metrics.yaml uses today. #49 proposes replacing it with .inf and reserving null for a genuinely-undefined bound; these four are exactly its "unbounded above" species, so whichever way that goes they should move with the rest. I kept main's spelling here rather than pre-applying an unmerged proposal — happy to flip them in either PR, whichever a maintainer prefers.

chrf and ter: why they are here and not waiting on #57

I said on #57 I would hold these until the sacrebleu scale question was settled. That was the wrong call and I have sent them: #57 is about scale, not identity or direction, and neither entry needs #57 answered to be correct.

• chrf takes [0, 1], the bounds the existing chrf-plus-plus and bleu entries already use.
• ter takes [0.0, null], the bounds the existing wer and cer entries already use.

So each one follows the convention its own siblings follow. If #57 lands on 0–100, it moves the whole sacrebleu family together; holding these two out would only have created a third convention in the meantime. Both metadata.scale fields say sacrebleu reports ×100 and point at #57, so nothing here is silently assuming an answer.

Two things the sourcing turned up, both verified against lm-evaluation-harness at main rather than inferred, and both now recorded in curation/UPSTREAM_DATA_ISSUES.md:

ter — lm-eval registers the direction backwards. lm_eval/api/metrics.py has @register_metric(metric="ter", higher_is_better=True) while the aggregation it registers says "Lower is better" in its own docstring and computes sacrebleu.corpus_ter. This is not a display hint: higher_is_better reaches is_higher_better() (api/registry.py) → Task.higher_is_better(), which lm-eval writes into its results JSON. The same registration table is correct (False) for perplexity, word_perplexity, byte_perplexity, bits_per_byte and brier_score, so ter is the lone inverted entry rather than a convention I am misreading. This registry states lower_is_better: true — the metric definition wins over one harness's registration.

chrf is documented as chrF++ but computes chrF. lm-eval's docstring for the metric calls it "chrF++" and links the chrF++ paper; the function calls sacrebleu.corpus_chrf(preds, refs) with the defaults, and sacrebleu.CHRF.WORD_ORDER = 0 — chrF++ is word_order=2. Different numbers on the same run, which is why this entry is chrf and not an alias onto chrf-plus-plus. The entry's metadata.note records the mislabel so a future contributor doesn't merge them back.

Neither is a data defect this registry corrects for other people, so UPSTREAM_DATA_ISSUES.md carries both with Action: flag to the lm-evaluation-harness maintainers, matching that file's existing shape, and says to delete each entry once upstream lands the fix.

4 aliases onto existing entries

entry	alias added	why
macro-f1	classification_macro_f1	HELM's stat name
matthews-correlation	mcc	lm-eval's stat name
cot-correct	chain_of_thought_correctness	HELM's stat name, whose display_name is this entry's "COT correct"
truthfulqa-mc2	mc2	lm-eval v0.3 / lighteval stat name

One correction to a reviewed entry — please look at this one

cot-correct carried two aliases, "COT correct" and "Equivalent (CoT)". Those are the display names of two different HELM metrics:

• chain_of_thought_correctness → display_name: COT correct (schema_capabilities.yaml) — "Fraction of correct answers after chain of thought"
• math_equiv_chain_of_thought → display_name: Equivalent (CoT) (schema_lite.yaml) — mathematical equivalence under CoT prompting

They were merged by a display-name scrape. "Equivalent (CoT)" moves off cot-correct onto the new math-equivalent-chain-of-thought, and cot-correct gains a metadata.note saying which of the two it is so it does not drift back. This is the only change to existing reviewed data, and it is the only alias this PR removes from anywhere.

I checked the move against the publish path rather than assuming: seeding without --prune-stale over the pre-change fixtures rewrites that alias row's canonical_id in place rather than leaving a second row, and afterwards 0 metric surfaces resolve to more than one canonical id (30,487 alias rows checked).

One code change

packages/eval-entity-resolver/src/eval_entity_resolver/eee.py — _METRIC_KEYWORDS gains macro[\s_-]*f1 and micro[\s_-]*f1 above the generic \bf1\b.

Without them the generic pattern matches inside both, so extract_metric("Micro F1") == "F1": the new micro-f1 entry would be unreachable from an EEE metric name, and macro-f1 already was. This is the same "compound before generic" shape the table already uses for ast[\s_-]*accuracy before \baccuracy\b. extract_metric("Tokenized F1") still returns "F1". Covered by test_class_averaged_f1_distinct_from_f1.

The comment two functions up used "Equivalent (CoT)" → cot-correct as its worked example; it now names the entry that alias actually resolves to.

2 names deliberately held back

ece and calibration_error — HELM never emits either bare name. It emits ece_10_bin, ece_1_bin, platt_ece_10_bin, platt_ece_1_bin (basic_metrics.py::compute_calibration_metrics), and lm-evaluation-harness has no ECE at all. A generic ece entry would be an entry nothing resolves to, and a bare ece alias would have to swallow ece_10_bin and throw away the bin count — a 10-bin and a 1-bin ECE are not the same number on the same run.

That is a slug-family question the registry has to take a position on, the same species as HELM's @k metrics which pass-at-1 / recall-at-5 answer one way, so it is #61 rather than a guess in a seed PR. I will send the four calibration entries with the same sourcing as this PR once #61 has an answer.

Verification

find fixtures -name '*.parquet' -delete
uv run eval-card-registry seed --local --prune-stale     # metrics 400 -> 416, harnesses 11 -> 12
uv run pytest                                            # 818 passed (817 before + the new test)

Cross-checked from the consumer side with every_eval_ever's tools/verify_metric_ids.py against this branch's seed: 20 of the 22 namespaced names now resolve, one per entry/alias above, with 0 ambiguous, and the 2 that remain are exactly the 2 held back.

18 mapped, 45 names checked

MAPPED ID NO LONGER IN THE REGISTRY: 0

NOW RESOLVABLE, STILL NAMESPACED: 20
  acc_norm -> length-normalized-accuracy
  bits_per_byte -> bits-per-byte
  brier_score -> brier-score
  byte_perplexity -> byte-perplexity
  chain_of_thought_correctness -> cot-correct
  chrf -> chrf
  classification_macro_f1 -> macro-f1
  classification_micro_f1 -> micro-f1
  ifeval_strict_accuracy -> ifeval-strict-accuracy
  math_equiv -> math-equivalent
  math_equiv_chain_of_thought -> math-equivalent-chain-of-thought
  mc1 -> truthfulqa-mc1
  mc2 -> truthfulqa-mc2
  mcc -> matthews-correlation
  prefix_exact_match -> prefix-exact-match
  quasi_exact_match -> quasi-exact-match
  quasi_prefix_exact_match -> quasi-prefix-exact-match
  rougeLsum -> rouge-lsum
  ter -> ter
  word_perplexity -> word-perplexity

AMBIGUOUS IN THE REGISTRY: 0
HARNESS SLUGS: lm-evaluation-harness ok, helm ok, inspect_ai ok

seed/harnesses.yaml: inspect_ai

The Inspect converter publishes inspect_ai as eval_library.name and this registry carried no Inspect entry, so that harness slug did not resolve. Added as draft under the name the converter already emits, so the id it publishes and the id here agree without a migration.

Notes for review

• Every entry is review_status: draft. score_type: continuous throughout, matching all 400 existing entries.
• Each entry keeps the raw stat name as an alias even where normalization already covers it (brier_score folds onto brier-score anyway), so a maintainer can find the entry by the string their tool printed.
• Not added: HELM's short_display_names EM and PEM. It reuses EM for both exact_match and quasi_exact_match, and PEM for both prefix variants, so either would resolve one metric's abbreviation to the other's entry. EM already belongs to exact-match here and stays there.
• truthfulqa-mc1 is kind: dataset_specific, like the truthfulqa-mc2 entry it mirrors: a consumer joining on it is joining within TruthfulQA, not across every benchmark that ever called something "MC1".
• After this merges, every_eval_ever bumps its pinned revision and moves these 20 from <harness>.<name> to the canonical ids in a follow-up PR. Records already published under the namespaced form need remapping, which is why that is a separate change with its own review.

Pre-PR checklist

• uv run eval-card-registry seed --local --prune-stale succeeds from a clean fixture state
• uv run pytest green from a clean tree on this branch (818 passed)
• New aliases target an existing canonical; no canonical renamed or duplicated. The one alias removal is a metric-identity correction, explained above, and verified against the non-pruning publish path
• No org-split changes

[borgr]

Rebased onto main (the branch was DIRTY against seed/metrics.yaml). Two things the conflict resolution decided, both worth a reviewer's eye rather than being buried in the diff.

length-normalized-accuracy is gone. main has since gained a reviewed normalized-accuracy whose own note names it as lm-eval's acc_norm, so the two entries were one quantity under two ids and would have collided on the acc_norm alias. The precise source string this PR had written for it now sits on normalized-accuracy instead, so nothing is lost and no second id is minted. That leaves 13 new metrics plus chrf and ter.

cot-correct keeps the change this PR made to it. The alias Equivalent (CoT) moves off cot-correct onto math-equivalent-chain-of-thought, because cot-correct was carrying the display names of two different HELM metrics — chain_of_thought_correctness and math_equiv_chain_of_thought. cot-correct gains chain_of_thought_correctness as its alias and the HELM source it reads off.

All 38 gate invariants and the seed suite pass locally on a clean fixture rebuild. The dry-run red is test_noncatalog_regen_is_confluent_with_committed, which fails on main unchanged.