Feature Description
Add support for FrontierScience (OpenAI, Jan 2026), a benchmark of expert-level scientific reasoning tasks across physics, chemistry, and biology. It has two tracks: Olympiad (100 gold-set short-answer problems, numeric/algebraic/fuzzy-string gradable) and Research (60 gold-set open-ended problems graded against 10-point rubrics via a model judge).
Dataset: https://huggingface.co/datasets/openai/frontierscience
Use Case
Existing science benchmarks (GPQA, MMLU) are near-saturated by frontier models. FrontierScience is unsaturated (top model GPT-5.2: 77% Olympiad, 25% Research), giving us headroom to differentiate frontier and OLMo-class models on expert scientific reasoning. The Research track's rubric-based grading also gives per-step diagnostic signal rather than just final-answer correctness.
Proposed Solution
- Olympiad track: standard generative task with a model-judge equivalence checker (given attempted + reference answer → CORRECT/INCORRECT). Judge prompt is in the paper (Appendix B).
- Research track: rubric-based model judge that returns total points earned (0–10); default success threshold = 7/10. Judge prompt also in Appendix B.
- Both use a strong judge model (paper uses GPT-5 high effort); make judge model configurable.
- Report accuracy overall and split by subject (physics/chem/bio).
Alternatives Considered
- Olympiad-only (skip rubric grading) — simpler, but loses the open-ended Research signal that's the more interesting/unsaturated part.
- Reusing an existing rubric-judge harness if we already have one (e.g. HealthBench-style) rather than building bespoke grading.
Additional Context
- Paper: FrontierScience (arXiv:2601.21165)
- Scoring config from paper: Olympiad averaged over 20 trials, Research over 30 trials.
- Text-only, no browsing / tools.
Feature Description
Add support for FrontierScience (OpenAI, Jan 2026), a benchmark of expert-level scientific reasoning tasks across physics, chemistry, and biology. It has two tracks: Olympiad (100 gold-set short-answer problems, numeric/algebraic/fuzzy-string gradable) and Research (60 gold-set open-ended problems graded against 10-point rubrics via a model judge).
Dataset: https://huggingface.co/datasets/openai/frontierscience
Use Case
Existing science benchmarks (GPQA, MMLU) are near-saturated by frontier models. FrontierScience is unsaturated (top model GPT-5.2: 77% Olympiad, 25% Research), giving us headroom to differentiate frontier and OLMo-class models on expert scientific reasoning. The Research track's rubric-based grading also gives per-step diagnostic signal rather than just final-answer correctness.
Proposed Solution
Alternatives Considered
Additional Context