Skip to content

Introduce model set list#505

Open
KartikP wants to merge 8 commits intomasterfrom
kp/introduce-model-sets
Open

Introduce model set list#505
KartikP wants to merge 8 commits intomasterfrom
kp/introduce-model-sets

Conversation

@KartikP
Copy link
Contributor

@KartikP KartikP commented Feb 9, 2026

image filtered_leaderboard

To do

  • Use runnable status instead of registered status by loading the model.

Summary

Here we categorize the benchmarks_model table into distinct groups:

  • reference
  • top10_2024
  • base
  • NULL

Using these groups, future benchmark submissions will only be evaluated on reference + top10_2024 + base. Supports expanding to other groups.

Reference:

  • hmax
  • alexnet
  • CORnet-S
  • resnet-50-robust
  • voneresnet-50-non_stochastic
  • grcnn
  • custom_model_cv_18_dagger_408
  • ViT_L_32_imagenet1k
  • mobilenet_v2_1_4_224
  • pixels

Top10_2024:

  • cvt_cvt-w24-384-in22k_finetuned-in1k_4
  • resnext101_32x8d_wsl
  • effnetb1_cutmixpatch_SAM_
  • effnetb1_cutmixpatch_augmix_robust32_avge4e7_manylayers_324x288
  • resnext101_32x32d_wsl
  • effnetb1_272x240
  • resnext101_32x48d_wsl
  • pnasnet_large_halfPrecision
  • resnet-152_v2
  • focalnet_tiny_lrf_in1k

Base:

Represents ~95 models that provide diverse coverage neural region performance, behavioral performance, and non-overlapping top 42 global scores

Selection Criteria:

  1. Registered and public (only considers models that are registered and public - goes from >600 registered models to 266 registered and public).
  2. Exclude models already part of reference and top10_2024 sets
  3. Selection strategy (in order):
    • Top 5 models per neural area (V1, V2, V4, IT) with overlap handling (~20 models)
    • Top 20 behavioral performers (behavior_vision) not already selected
    • Fill remaining with next top-ranked models until reaching 100 models

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant