Skip to content

[FEATURE] Add FrontierScience benchmark #263

Description

@donovanr

Feature Description

Add support for FrontierScience (OpenAI, Jan 2026), a benchmark of expert-level scientific reasoning tasks across physics, chemistry, and biology. It has two tracks: Olympiad (100 gold-set short-answer problems, numeric/algebraic/fuzzy-string gradable) and Research (60 gold-set open-ended problems graded against 10-point rubrics via a model judge).

Dataset: https://huggingface.co/datasets/openai/frontierscience

Use Case

Existing science benchmarks (GPQA, MMLU) are near-saturated by frontier models. FrontierScience is unsaturated (top model GPT-5.2: 77% Olympiad, 25% Research), giving us headroom to differentiate frontier and OLMo-class models on expert scientific reasoning. The Research track's rubric-based grading also gives per-step diagnostic signal rather than just final-answer correctness.

Proposed Solution

  • Olympiad track: standard generative task with a model-judge equivalence checker (given attempted + reference answer → CORRECT/INCORRECT). Judge prompt is in the paper (Appendix B).
  • Research track: rubric-based model judge that returns total points earned (0–10); default success threshold = 7/10. Judge prompt also in Appendix B.
  • Both use a strong judge model (paper uses GPT-5 high effort); make judge model configurable.
  • Report accuracy overall and split by subject (physics/chem/bio).

Alternatives Considered

  • Olympiad-only (skip rubric grading) — simpler, but loses the open-ended Research signal that's the more interesting/unsaturated part.
  • Reusing an existing rubric-judge harness if we already have one (e.g. HealthBench-style) rather than building bespoke grading.

Additional Context

  • Paper: FrontierScience (arXiv:2601.21165)
  • Scoring config from paper: Olympiad averaged over 20 trials, Research over 30 trials.
  • Text-only, no browsing / tools.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions