Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data available for which problems were solved by which models? #3

Open
rawwerks opened this issue Jan 2, 2025 · 0 comments
Open

data available for which problems were solved by which models? #3

rawwerks opened this issue Jan 2, 2025 · 0 comments

Comments

@rawwerks
Copy link

rawwerks commented Jan 2, 2025

@paul-gauthier - i'm really inspired by this benchmark!

in your blog post, you mentioned "The new benchmark uses the 225 problems that were solved by 3 or fewer models. "

do you have the data on which problems were solved by which models? i looked here but it only seems to be the summaries.

it would be helpful to see this data to help me partition the benchmark into easy/medium/hard problems. i'm also interested in running optimizations to get a specific model to overcome problems it previously got wrong (without having to run the whole benchmark every time, which for some models is expensive).

thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant