Skip to content

CHANGELOG.md

Latest commit

 

History

History
295 lines (200 loc) · 30.2 KB

CHANGELOG.md

File metadata and controls

295 lines (200 loc) · 30.2 KB
Raw

Changelog

0.5.2 (2025-10-16)

Chores

  • require manual install for cyber plugin (#252) (090f801)

0.5.1 (2025-10-16)

Bug Fixes

Chores

Refactor

  • extract cybersecurity benchmarks to plugin (#251) (df829e2)

0.5.0 (2025-10-10)

⚠ BREAKING CHANGES

  • added more groupings under benchmarks catalog (#244)

Features

  • add clockbench evaluation framwork and script for synthesizing public dataset. (#159) (3ba9836)
  • add IFEval (#182) (8d1b939)
  • add local openbench implementation of groq provider in inspect (#131) (52aea35)
  • add mmmlu eval (#193) (a42c2d5)
  • add mmstar benchmark (#174) (5d085ab)
  • add new openbench documentation (#169) (f3e6a37)
  • add overarching bbh command to run all 18 BBH tasks (463a25f)
  • add preset eval group infrastructure (#215) (d9ea03a)
  • added more groupings under benchmarks catalog (#244) (d932cb0)
  • ArabicMMLU: add remaining 32 Arabic exam subsets, total 41 subsets (#219) (006e248)
  • benchmark: add support for arc-agi (#158) (3f32253)
  • benchmark: add support for detailbench (#154) (23fbca5)
  • benchmark: add support for TUMLU (#160) (#161) (885be75)
  • benchmark: multichallenge implementation (#170) (cf2ab4f)
  • change default model to groq/openai/gpt-oss-20b (#138) (8f7f42f)
  • components: export the run_eval entrypoint method (#157) (acbe7f4)
  • configure release-please for pre-v1.0 version bumping (#133) (c432934)
  • cybench: ported over code for cybench (#207) (7949425)
  • cybersecurity, changelog, more docs (39f123c)
  • display results patch to include task duration stats (#167) (e4e480c)
  • docs: add changelog page (#225) (7db9135)
  • docs: add release notes section and update index with new features for v0.5 (#245) (09ab78e)
  • docs: added feature card and docs page for exercism (#243) (2b38147)
  • docs: Added feature eval docs pages and cache command docs (#191) (50501f1)
  • eval: add support for json output (#14) (f335418)
  • exercism: added support for exercism tasks w/ agent support for aider, roo, claude, opencode (#151) (d86f0da)
  • graphwalks token filter (#115) (e38658c)
  • groq reasoning effort + bugfix to override inspect's "groq" (#142) (b919cc7)
  • lighteval: Add 7 core commonsense reasoning benchmarks from LightEval (#197) (7792c45)
  • lighteval: add BigBench eval (122 MCQ tasks) (9f35b1d)
  • lighteval: Add cross-lingual understanding benchmarks (XCOPA, XStoryCloze, XWinograd) (917667a)
  • lighteval: add Global-MMLU eval (42 languages) (3542213)
  • lighteval: register BigBench benchmarks in config and registry (77018e9)
  • lighteval: register Global-MMLU benchmarks in config and registry (156f509)
  • link to subscription form on main page (#240) (988d08c)
  • livemcpbench: Adding support for liveMCPBench (#127) (222f678)
  • make evals dash/undescore insensitive (#185) (5ec5177)
  • mbpp (#117) (93ad88b)
  • mcq_eval: enable abstraction of MCQ eval (#181) (2f53db2)
  • mmmu-pro: added support for mmmu_mcq, mmmu_open, mmmu_pro, mmmu_pro_vision (#134) (a875378)
  • openrouter: add OpenRouter provider support (#145) (47b579e)
  • openrouter: add provider routing args support (#180) (12e1d81)
  • otis-mock-aime: added support for otis mock aime 2024-2025 (#218) (1b9fd5c)
  • plugins: add entry point system for external benchmarks (#216) (71e7257)
  • return eval logs from run_eval function (#173) (ee459d9)
  • rootly_terraform: add initial implementation of Rootly Terraform evals (#195) (cd3acae)

Bug Fixes

  • allow for more python versions (#164) (e6682fe)
  • close headqa metadata entry (947522d)
  • cybench: moved cybench dependency into dependency group (#237) (8d30715)
  • handle missing SciCode dependency lazily in solver (#186) (fed4e88)
  • improve BBH target extraction to handle multi-char answers (147e3e0)
  • load Neue Regrade font in Mintlify docs (#177) (550c7f5)
  • make core package actually install (#235) (edeb4b8)
  • normalize benchmark keys during entry point merge (#217) (d285664)
  • register headqa_en and headqa_es variants (6a19aa1)
  • render inspect error correctly (#241) (97ccd10)
  • resolve registry import conflict (98a1c79)
  • scicode: add support for test split, fix test_data.h5 import error (#149) (23fa8cb)
  • update bbh function for programmatic access only (811ce9e)
  • use generic type ignore for bbh task decorator (fd67171)

Documentation

  • readme: clarify benchmark case-sensitivity and grader requirements (#135) (c34a5a3)

Chores

  • add @nmayorga7 to CODEOWNERS (abad7bf)
  • alphabetize available benchmarks error (#214) (68f46e9)
  • benchmark: removed combined cti-bench eval (#183) (a77852c)
  • bugbot fixes for MCQ (#190) (6ecaefc)
  • docs: add docs for openrouter and MCQEval (#188) (7f8cd83)
  • docs: alphabetize benchmarks metadata (#187) (ce77812)
  • docs: benchmarks each on new line (#184) (b3c40f8)
  • docs: minor cleanup (#179) (80c9e09)
  • fixed fonts in openbench docs (#178) (6e3c2a5)
  • GitHub Terraform: Create/Update .github/workflows/stale.yaml [skip ci] (d0018d1)
  • mcq-eval: accept more dataset types (#194) (cb5e038)
  • move all metrics to discrete files in /metrics (#168) (7caa042)
  • release-please pre-1.0: treat BREAKING as minor (3c88a42)
  • remove pre-commit benchmark checks for easier CI (#213) (67b07a7)
  • rename OpenBench to openbench (#196) (0621b46)
  • rename task to sample in time metrics (#172) (96e1817)
  • sync packaging pyproject (#234) (940a879)
  • update Claude workflows to enhance permissions and streamline triggers (#136) (effb7da)
  • update readme and contributing (#176) (ea606ba)
  • update release-please configuration and add lockfile update workflow (#146) (2d6ad9b)
  • user agent (#163) (e20f3c1)

CI

  • add benchmarks validation pre commit hook (#171) (3725638)
  • remove PR trigger from release-please (#166) (6440d44)

0.4.1 (2025-08-29)

Bug Fixes

  • rootly_gmcq: handle both string and list content types in scorer (#129) (376624d)

0.4.0 (2025-08-28)

Features

  • add boolq (#70) (edbd1cc)
  • add BrowseComp (#118) (498c706)
  • add CITATION.cff for software citation (#102) (16960de)
  • add CTI-Bench cybersecurity benchmark suite (#96) (8465075)
  • add GitHub issue and PR templates (#103) (68f0ef0)
  • add gmcq (#114) (bb3c89d)
  • add MuSR variants and grouped metrics (#107) (10ae935)
  • add robust answer extraction scorers from gpt-oss to MathArena benchmarks and gpqa_diamond (#97) (251ba66)
  • add Vercel AI Gateway inference provider (#98) (38e211a)
  • jsonschemabench (#95) (e3d842d)
  • mmmu: added support for mmmu benchmark and all of its subdomains (#121) (801bceb)

Bug Fixes

  • format mmlu_pro.py dataset file (2a9ee65)
  • handle skipped integration tests in CI (#120) (dae9378)
  • hle: added multimodal support for hle (#128) (8c3f212)
  • jsonschemaeval: match paper methodology and add openai subset (#113) (1b6470b)
  • make claude-code-review job optional to prevent PR blocking (#100) (6aad080)

Documentation

  • emphasize pre-commit hooks installation requirement (#106) (e765464)
  • refresh CONTRIBUTING.md and update README references (#105) (bf66747)
  • update installation instructions and clarify dependency architecture in CLAUDE.md and CONTRIBUTING.md (#126) (cd962fd)
  • update README citation to match CITATION.cff (#104) (6219e8c)

Chores

CI

  • add automated PyPI publishing to release workflow (#99) (eddbf70)

0.3.0 (2025-08-14)

Features

  • add --debug flag to eval-retry command (b26afaa)
  • add -M and -T flags for model and task arguments (#75) (46a6ba6)
  • add 'openbench' as alternative CLI entry point (#48) (68b3c5b)
  • add AI21 Labs inference provider (#86) (db7bde7)
  • add Baseten inference provider (#79) (696e2aa)
  • add Cerebras and SambaNova model providers (1c61f59)
  • add Cohere inference provider (#90) (8e6e838)
  • add Crusoe inference provider (#84) (3d0c794)
  • add DeepInfra inference provider (#85) (6fedf53)
  • add Friendli inference provider (#88) (7e2b258)
  • Add huggingface inference provider (#54) (f479703)
  • add Hyperbolic inference provider (#80) (4ebf723)
  • add initial GraphWalks benchmark implementation (#58) (1aefd07)
  • add Lambda AI inference provider (#81) (b78c346)
  • add MiniMax inference provider (#87) (09fd27b)
  • add Moonshot inference provider (#91) (e5743cb)
  • add Nebius model provider (#47) (ba2ec19)
  • add Nous Research model provider (#49) (32dd815)
  • add Novita AI inference provider (#82) (6f5874a)
  • add Parasail inference provider (#83) (973c7b3)
  • add Reka inference provider (#89) (1ab9c53)
  • add SciCode (#63) (3650bfa)
  • add support for alpha benchmarks in evaluation commands (#92) (e2ccfaa)
  • push eval data to huggingface repo (#65) (acc600f)

Bug Fixes

  • add missing newline at end of novita.py (ef0fa4b)
  • remove default sampling parameters from CLI (#72) (978638a)

Documentation

  • docs for 0.3.0 (#93) (fe358bb)
  • fix directory structure documentation in CONTRIBUTING.md (#78) (41f8ed9)

Chores

  • fix GraphWalks: Split into three separate benchmarks (#76) (d1ed96e)
  • update version (8b7bbe7)

Refactor

  • move task loading from registry to config and update imports (de6eea2)

CI

  • Enhance Claude code review workflow with updated prompts and model specification (#71) (b605ed2)

0.2.0 (2025-08-11)

Features

Documentation

  • update CLAUDE.md with pre-commit and dependency pinning requirements (f33730e)

Chores

  • GitHub Terraform: Create/Update .github/workflows/stale.yaml [skip ci] (1a00342)

0.1.1 (2025-07-31)

Bug Fixes

  • add missing init.py files and fix package discovery for PyPI (#10) (29fcdf6)

Documentation

  • update README to streamline setup instructions for OpenBench, use pypi (16e08a0)

0.1.0 (2025-07-31)

Features

Chores

  • ci: update release-please workflow to allow label management (b70db16)
  • drop versions for release (58ce995)
  • GitHub Terraform: Create/Update .github/workflows/stale.yaml [skip ci] (555658a)
  • update project metadata for version 0.1.0, add license, readme, and repository links (9ea2102)