diff --git a/benchmarks/templates/benchmarks/tutorials/models/deepdive_2.html b/benchmarks/templates/benchmarks/tutorials/models/deepdive_2.html index bb7446c7..56d480fc 100644 --- a/benchmarks/templates/benchmarks/tutorials/models/deepdive_2.html +++ b/benchmarks/templates/benchmarks/tutorials/models/deepdive_2.html @@ -196,7 +196,62 @@

Part 6: Exploring the model.py Fil
-

Part 7: Exploring an (optional) requirements.txt File

+

Part 7: Exploring the (optional) Region-Layer Mapping

+

+ Brain-Score allows users to specify a region-layer mapping for models. This mapping assigns a specific layer from the model.py file + to a particular brain region, ensuring it remains fixed when evaluated across different benchmarks. +

+

+ We provide two methods for specifying a region-layer mapping: +

+
    +
  1. (The preferred method) Passing a region_layer_mapping argument in your __init__.py model_registry + entry (Example).
  2. + +
    
    +1       model_registry['resnet50_tutorial'] = lambda: ModelCommitment(identifier='resnet50_tutorial', 
    +2           activations_model=get_model('resnet50_tutorial'), 
    +3           layers=get_layers('resnet50_tutorial'), 
    +4           region_layer_mapping={"V1": "layer1", "V2": "layer3", "V4": "layer4", "IT": "fc"})
    +
    +    

    + +
  3. Providing a {model_identifier}.json file in a region_layer_map/ + folder within your submission package (Example).
  4. + +
    
    +1       {
    +2           "V1": "layer1", 
    +3           "V2": "layer3", 
    +4           "V4": "layer4",
    +5           "IT": "fc" 
    +6       }
    +    
    +
+ + Importantly, when using the JSON file approach, each layer-mapping for each model identifier must be contained in its own JSON file and be named {model_identifier}.json. + +

+

+ While optional, we recommend specifying this mapping for models with layers designed to replicate specific brain regions. + If pre-committed, the layer names must match those returned by the get_layers() function in the model.py. +

+

+ Note: If a submission does not include a region_layer_mapping JSON file, the layer mapping will be automatically computed before scoring using + STANDARD_REGION_BENCHMARKS. +

+

+ This standard mappping assigns layers to each region independently, meaning: +

  • A region does not need to be mapped to a unique layer.
  • +
  • Regions do not have to follow a specific order.
  • +

    +

    + This intentional flexibility is particularly relevant for models with layers that were intended to replicate specific brain regions, as there is no guarantee that the layer names will match the region names unless pre-committed. + As such, for these types of models, we strongly recommend specifying a region-layer mapping. +

    +
    +
    +

    Part 8: Exploring an (optional) requirements.txt File

    The (optional) requirements.txt file is where you can add any requirements that your model needs (such as a specific version of a package or an external git @@ -213,7 +268,7 @@

    Part 7: Exploring an (optional) re

    -

    Part 8: Putting it All Together

    +

    Part 9: Putting it All Together

    You are almost done! If you were actually submitting a model, the final step prior to submission would be to run your model locally to ensure that everything is in working order. You can do this by first following the @@ -265,7 +320,7 @@

    Part 8: Putting it All Together
    -

    Part 9: Model Summary Tools

    +

    Part 10: Model Summary Tools

    For models built using PyTorch, the torchsummary package can be utilized to get a summary of model information. Install it via pip if necessary