Skip to content

Add aiXamine evaluator org, family, and 9 service composites - #63

Open
fatihdeniz wants to merge 1 commit into
evaleval:mainfrom
fatihdeniz:seed/aixamine-evaluator
Open

Add aiXamine evaluator org, family, and 9 service composites#63
fatihdeniz wants to merge 1 commit into
evaleval:mainfrom
fatihdeniz:seed/aixamine-evaluator

Conversation

@fatihdeniz

Copy link
Copy Markdown

Registers aiXamine (aixamine.qcri.org, paper) as an evaluator so its EEE records resolve to real entities.

  • seed/orgs.yamlaixamine org (tags: ["academic","evaluator"])
  • seed/families.yamlaixamine family grouping the 9 service leaderboards
  • seed/composites.yaml — 9 aixamine-<service> composites mapping each source_config

Seed-only (+91 lines). seed --local ingests cleanly; gate suite passes (the one failure is an unrelated models.dev drift check).

@borgr

borgr commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Verified locally against current main and I recommend merging.

Ran the same steps the dry-run job runs, on main + this branch:

rm -f fixtures/*.parquet && LOCAL_MODE=true uv run eval-card-registry seed --local
LOCAL_MODE=true uv run pytest

Result: 1 failed, 868 passed. The single failure is test_refresh_modelsdev_dedup.py::test_noncatalog_regen_is_confluent_with_committed, which is a pre-existing break on main — tests/fixtures/modelsdev_api.snapshot.json has not moved since 923986e while seed/models/sources/models_dev.generated.yaml has been refreshed six times since, most recently d608d6b. Unmodified main gives the identical 1 failed, 868 passed, so this branch adds no failure. Fixing it means re-pinning the snapshot, which is separate work.

The content reads clean to me. One evaluator org, one family, nine service composites each with a single config and a category/tags pair that matches what the service measures. Additive only, no existing entry touched.

Worth landing this together with the adapter in evaleval/every_eval_ever#283, since that adapter's records resolve against these entities.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants