feat(benchmarks): benchmark-backed model-card skeleton (#194) - #216
Merged
Conversation
…ema export (#194) Add the machine-enforced-comparability core for benchmark-backed model cards: BenchmarkResult / BenchmarkSource / ComparisonCohort frozen models (import-free), an import-free benchmark_registry with group_comparable that refuses to co-rank scores across cohorts (dataset revision, metric, threshold policy, harness...), and provenance validation (published needs a url, measured needs a harness version). Export to schemas/guardrail_benchmarks.json + a BenchmarkResult JSON Schema, both --check in pre-commit. The registry ships EMPTY — a ready-to-fill skeleton; numbers are hand-authored in src/any_guardrail/_benchmark_data.py. Part of #194. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ers, operational) Add the top-level, unshipped benchmarks/ package (outside src/, never in the wheel, never run in CI): a GuardBench moderation-function adapter over Guardrail.validate(), stub runners for what GuardBench lacks (judge pointwise, span/streaming, PII), and an operational-measurement helper (p50/p95 latency + memory estimate). Heavy deps (guardbench, presidio-analyzer) live in the benchmarks dependency-group and are imported lazily. README documents the methodology, the per-dataset license/access table, and the comparability invariant. Part of #194. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…#194) Extend generate_api_docs so each guardrail page appends a ## Benchmarks table (grouped by category; missing value -> em dash never 0; provenance, contamination, and comparison-cohort keys shown, so scores are never silently treated as comparable) and a ## License section (vendor, default_license, and the per-variant license table from #211). Add docs/benchmarks.md (methodology + how to add numbers) linked from SUMMARY, and document the subsystem in CLAUDE.md. Part of #194. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lock in the machine-enforced comparability guard with the misleading comparisons from the issue review: ToxicChat 1123 vs 0124, Optimal-F1 vs F1@0.5, AUC-in-F1, cross-harness baselines all land in distinct cohorts; missing stays None (never 0); a source without provenance is rejected. Plus registry coverage, JSON/schema parity, leaf-import invariants, rendering (provenance/contamination/cohort keys, empty note, license variant table), and a no-guardrail-import subprocess check. Part of #194. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Introduces the in-repo “skeleton” for benchmark-backed model cards: a dependency-light benchmark result registry + JSON exports/schemas, plus docs-generation hooks that render per-guardrail Benchmarks and License sections from registries (with the actual benchmark runs intended to happen out-of-repo).
Changes:
- Add frozen Pydantic models (
BenchmarkResult,BenchmarkSource,ComparisonCohort) and an import-free benchmark registry fed by a (currently empty)_benchmark_data.pysource. - Add generators + pre-commit checks for
schemas/guardrail_benchmarks.jsonandschemas/guardrail_benchmarks.schema.json. - Extend API doc generation and docs navigation to include Benchmarks/License model-card sections and a benchmarks methodology page; add an unshipped
benchmarks/harness scaffold.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_benchmarks.py | Adds unit tests for provenance validation, comparability grouping, rendering, and leaf/no-import invariants. |
| src/any_guardrail/benchmarks.py | Adds frozen benchmark/provenance/cohort models intended to stay leaf (stdlib + pydantic only). |
| src/any_guardrail/benchmark_registry.py | Adds import-free registry and group_comparable cohort partitioning. |
| src/any_guardrail/_benchmark_data.py | Adds empty, hand-authored benchmark results source (ships empty). |
| src/any_guardrail/types.py | Re-exports benchmark models via any_guardrail.types. |
| src/any_guardrail/init.py | Re-exports benchmark models at package top-level. |
| scripts/generate_benchmarks_json.py | Adds deterministic JSON export generator + --check mode. |
| scripts/generate_benchmark_schema.py | Adds JSON Schema generator for BenchmarkResult + --check mode. |
| scripts/generate_api_docs.py | Appends Benchmarks and License sections to guardrail API pages based on registries. |
| schemas/guardrail_benchmarks.json | Adds committed benchmarks export (currently {}). |
| schemas/guardrail_benchmarks.schema.json | Adds committed JSON Schema for benchmark results. |
| .pre-commit-config.yaml | Adds --check hooks to keep benchmark JSON and schema in sync. |
| pyproject.toml | Adds mypy overrides for benchmark deps/scripts and a benchmarks dependency-group. |
| docs/benchmarks.md | Adds methodology/documentation page for benchmark-backed model cards. |
| docs/SUMMARY.md | Links the new Benchmarks & Model Cards page in docs navigation. |
| CLAUDE.md | Documents the new benchmark generators and artifacts in repo contributor guidance. |
| benchmarks/init.py | Adds unshipped harness package overview (top-level, not in wheel). |
| benchmarks/README.md | Documents harness layout, comparability invariant, and dataset license/access table. |
| benchmarks/guardbench_adapter.py | Adds GuardBench adapter scaffold (validate → moderation callback) with lazy import. |
| benchmarks/runners.py | Adds stubs for non-GuardBench evaluation runners. |
| benchmarks/operational.py | Adds operational measurement helpers (latency + memory estimate). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+272
to
+273
| except Exception: | ||
| return "" |
Contributor
Author
There was a problem hiding this comment.
Fixed in bd7c082 — narrowed to except ImportError (only a partial install without the benchmark registry is swallowed); a real renderer/registry bug now propagates instead of silently dropping the section.
Comment on lines
+297
to
+298
| except Exception: | ||
| return "" |
Contributor
Author
There was a problem hiding this comment.
Fixed in bd7c082 — narrowed to except ImportError for the same reason; a metadata-shape regression now fails loudly rather than silently omitting the License section.
| samples_ms.append((time.perf_counter() - start) * 1000.0) | ||
|
|
||
| ordered = sorted(samples_ms) | ||
| p95_index = min(len(ordered) - 1, round(0.95 * (len(ordered) - 1))) |
Contributor
Author
There was a problem hiding this comment.
Fixed in bd7c082 — switched to a ceil-based nearest-rank index: min(n-1, max(0, math.ceil(0.95*n) - 1)), which never underestimates p95 the way bankers-rounding round() can.
| "p50_ms": statistics.median(ordered), | ||
| "p95_ms": ordered[p95_index], | ||
| "mean_ms": statistics.fmean(ordered), | ||
| "n": float(len(ordered)), |
Contributor
Author
There was a problem hiding this comment.
Fixed in bd7c082 — n is now returned as an int count (still compatible with the dict[str, float] return via int→float promotion).
- Narrow the model-card _benchmarks_section / _license_section catches from Exception to ImportError, so only a partial install is swallowed and real renderer/metadata bugs surface instead of silently producing incomplete docs. - operational.measure_latency: use a ceil-based nearest-rank p95 index (round() bankers-rounding could underestimate p95), and return the sample count n as an int. - Add 'fpr' (false-positive rate) to the codespell ignore list — this was the CI run-linter failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…240) * feat(benchmarks): populate the registry from the Group E harness run Fills `_benchmark_data.py` — which shipped empty by design, pending "out-of-repo experiment runs" — with 181 measured BenchmarkResults covering 29 of the 30 open-weight guardrails, and regenerates schemas/guardrail_benchmarks.json. Produced by any-guardrail-benchmarks on one 80GB H100: a ~20h sweep at n=285 per cell, plus prior router_judge / bir results ingested rather than re-run. Each result carries its ComparisonCohort (dataset revision, split, label mapping, metric, threshold policy, harness) so measured and published numbers can never be ranked against each other, and contamination is flagged where a guardrail is in-distribution with a dataset (e.g. deepset x deepset-prompt-injections, wild_guard x wildguardmix). Coverage: content_safety, prompt_injection, general_judge, tool_use, bias, hallucination, pii, off_topic, toxicity. Not included: - qwen3_guard_stream (0 results): its remote modeling code requires transformers<5 and the harness runs 5.5.0, so all six cells errored. - 7 rows on the router_judge `trace` / `trail` suites: agent-trajectory evaluation has no GuardrailCategory, and `category` must be one, so they stay in the harness repo's records. - Per-variant numbers: BenchmarkResult has no model field, so a guardrail measured on several model variants is committed at its **default** variant only. The harness keeps the full size curve (e.g. shieldgemma 2b/9b/27b, DynaGuard 1.7b/4b/8b — both non-monotonic). Three tests asserted the registry ships empty, which landing data necessarily breaks; they now assert coverage and correct grouping instead, and the empty-section renderer test picks a guardrail that has no results rather than relying on the whole registry being empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Address review: force the empty-benchmarks case instead of searching for one The empty-section test picked a guardrail via `next(name for name in GuardrailName if not get_benchmarks(name))`, which raises a bare StopIteration once every guardrail has committed results — an opaque failure exactly when someone completes the coverage. It also silently stopped testing anything if the search happened to pick a guardrail whose results arrived later. Monkeypatches `get_benchmarks` to return empty instead, so the empty-render path is covered deterministically regardless of what is committed. Verified the test no longer depends on registry contents. (For reference, the guardrails with no results today are the nine API-key ones — azure/bedrock/lakera/patronus/... which this harness does not measure — plus qwen3_guard_stream.) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Resolve conflicts from main's parameter runtime-requirements (#220), evaluate() dispatcher (#230), batched judge inference (#227), and the Llama-Guard-4-12B drop (#236): - CLAUDE.md: keep main's rewritten guardrail_parameters.json bullet (env_var/secret/effectively_required/requirement_groups) alongside the benchmark PR's new guardrail_benchmarks.json bullet. Correct the stale, self-contradictory "registry ships empty" claim in two places (the registry now ships 181 hand-authored BenchmarkResults across 29 guardrails from #240). - tests/unit/test_benchmarks.py: repoint the per-variant-license assertion from the dropped meta-llama/Llama-Guard-4-12B to Llama-Guard-3-8B (whose llama-3.1 license still differs from the llama-3.2 default, so it keeps exercising the variant-table path). - docs/api/*.md: regenerate so every guardrail model-card page carries the PR's new ## Benchmarks / ## License sections (main had regenerated these pages before the benchmark-rendering code existed). All 1313 unit+docs tests pass; pre-commit (ruff v0.16.1, mypy strict, and all 10 --check generator hooks) is green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The complete in-repo machinery for benchmark-backed model cards (#194), built so out-of-repo experiments can fill it in. No benchmark runs are performed and no
measurednumbers are committed — the registry ships empty and every piece is exercised by tests.Addresses #194 (the skeleton; the eval runs + harvest are out-of-repo follow-up, below).
What's in the skeleton
Comparability as a machine-enforced invariant (incorporating the issue reviewer's suggestion):
BenchmarkResult/BenchmarkSource/ComparisonCohortfrozen, import-free models (benchmarks.py).ComparisonCohort(dataset revision, label mapping, metric + threshold policy, harness, aggregation).benchmark_registry.group_comparablerefuses to co-rank scores across cohorts, so ToxicChat 1123 vs 0124, Optimal-F1 vs F1@0.5, AUC-in-an-F1-column, and cross-harness baselines can't become one ranked column. A missing score isNoneand renders as—, never0; provenance (published:<url>/measured:<harness-version>) is required.Artifacts & generators (mirroring the existing registry pattern):
schemas/guardrail_benchmarks.json(export) +schemas/guardrail_benchmarks.schema.json(JSON Schema for aBenchmarkResult), each with a--checkpre-commit hook.src/any_guardrail/_benchmark_data.py(ships empty).Model-card rendering —⚠️ , and cohort keys shown) and a
generate_api_docsnow appends a## Benchmarkstable (grouped by category; provenance, contamination## Licensesection (vendor +default_license+ the per-variant table from #211) to every guardrail page.docs/benchmarks.mddocuments the methodology, linked from SUMMARY.Unshipped harness — top-level
benchmarks/package (outsidesrc/, never in the wheel, never in CI): GuardBench adapter (Guardrail.validate()→ moderation callback), runner stubs (judge pointwise / span / streaming / PII), operational measurement (p50/p95 latency + memory), and a README with the per-dataset license/access table. Heavy deps live in thebenchmarksdependency-group (guardbench,presidio-analyzer— both verified on PyPI).Verification
pytest tests/unit/test_benchmarks.py— 57 passed, incl. the adversarial cohort fixtures (the 6 misleading comparisons are the load-bearing acceptance test), rendering, JSON/schema parity, and leaf/no-import invariants.pytest tests/unit tests/docs— 961 passed.pre-commit— ruff, ruff-format, mypy (strict), and the two new--checkhooks pass.python scripts/convert_to_gitbook.pybuilds the site andvalidate_summarypasses; model-card pages render the new sections.Work required outside this repo (this PR is the skeleton only)
BenchmarkResult(...)taggedmeasured:<harness-version>.published:<url>.benchmarks/guardbench_adapter.py::evaluate_with_guardbench(thevalidate()→callback conversion is done; registration is the seam) and implement the runner stubs.HF_TOKEN+ accept terms.workflow_dispatch-only benchmark workflow; publish runs to the GuardBench HF leaderboard; a pairwise judge entry point (unlocks JudgeBench/RewardBench 2).🤖 Generated with Claude Code