Skip to content

Add DALPHA MLE-Bench Results #129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
BenMyoung wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add DALPHA MLE-Bench Results #129

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5ae0e81
Add CobraAgent results
BenMyoung Mar 16, 2026
541c16d
Merge remote-tracking branch 'upstream/main'
BenMyoung Mar 23, 2026
9a4ab65
Resolve runs/run_group_experiments.csv
BenMyoung Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ Code for the paper ["MLE-Bench: Evaluating Machine Learning Agents on Machine Le
## Leaderboard
| Agent | LLM(s) used | Low == Lite (%) | Medium (%) | High (%) | All (%) | Running Time (hours) | Date | Source Code Available | Grading Reports Available |
|-------|-------------|-----------------|------------|----------|---------|----------------------|------|----------------------|---------------------------|
| [CobraAgent](https://dalphakr.github.io/CobraAgent/)<br>(Dalpha) | Ensemble (Gemini-3.1-Pro-Preview, Gemini-3-Flash-Preview) | 86.36 ± 0.00 | 81.58 ± 1.52 | 62.22 ± 2.22 | 79.11 ± 0.44 | 24 | 2026-03-17 | X | ✓ |
| [Famou-Agent 2.0](https://github.com/baidubce/FM-Agent) | Gemini-3-Pro-Preview | 80.3 ± 1.52 | 64.04 ± 2.32 | 42.22 ± 2.22 | 64.44 ± 1.18 | 24 | 2026-02-23 | X | ✓ |
| [AIBuildAI](https://github.com/aibuildai/AI-Build-AI) | Claude-Opus-4.6 | 77.27 ± 0.00 | 61.40 ± 0.88 | 46.67 ± 0.00 | 63.11 ± 0.44 | 24 | 2026-03-06 | X | ✓ |
| [CAIR](https://research.google/teams/cloud-ai-research/) [MARS+](https://arxiv.org/pdf/2602.02660) | Gemini-3-Pro-Preview | 78.79 ± 1.52 | 60.53 ± 1.52 | 44.44 ± 2.22 | 62.67 ± 0.77 | 24 | 2026-02-17 | X | ✓ |
Expand Down
3 changes: 2 additions & 1 deletion runs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,4 +60,5 @@ table below.
| MLEvolve | Gemini-3-Pro-preview, 12 hours, 21 vCPUs, 234GB of RAM, and 1 H200 GPU |
| MARS | CAIR MARS, 24 hours, 12 vCPUs, 220GB of RAM, and 1 A100-40GB GPU |
| MARS+ | CAIR MARS+, 24 hours, 48 vCPUs, 220GB of RAM, and 2 × H100 GPUs |
| AIBuildAI | Claude-Opus-4.6, 24 hours, 24 vCPUs, 256GB of RAM, and 1 A100 GPU |
| AIBuildAI | Claude-Opus-4.6, 24 hours, 24 vCPUs, 256GB of RAM, and 1 A100 GPU |
| CobraAgent | Ensemble (Gemini-3.1-Pro-Preview, Gemini-3-Flash-Preview), 24 hours, (16 vCPUs, 32 GB RAM, and 1 RTX 4060 GPU / 11 vCPUS, 128 GB RAM, and 1 A100 GPU) |
3 changes: 3 additions & 0 deletions runs/cobra_agent_group1/grading_report_group_1.json
View file
Open in desktop
Git LFS file not shown
3 changes: 3 additions & 0 deletions runs/cobra_agent_group2/grading_report_group_2.json
View file
Open in desktop
Git LFS file not shown
3 changes: 3 additions & 0 deletions runs/cobra_agent_group3/grading_report_group_3.json
View file
Open in desktop
Git LFS file not shown
4 changes: 2 additions & 2 deletions runs/run_group_experiments.csv
View file
Open in desktop
Git LFS file not shown