Skip to content
61 changes: 58 additions & 3 deletions benchmarks/templates/benchmarks/tutorials/models/deepdive_2.html
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,62 @@ <h3 class="benefits_heading is-size-3-mobile">Part 6: Exploring the model.py Fil
</code></pre>
</div>
<div class="box leaderboard-table-component">
<h3 class="benefits_heading is-size-3-mobile">Part 7: Exploring an (optional) requirements.txt File</h3>
<h3 class="benefits_heading is-size-3-mobile">Part 7: Exploring the (optional) Region-Layer Mapping</h3>
<p class="benefits_info is-size-5-mobile">
Brain-Score allows users to specify a region-layer mapping for models. This mapping assigns a specific layer from <span class="special_format">model.py</span> file
to a particular brain region, ensuring it remains fixed when evaluated across different benchmarks.
</p>
<p class="benefits_info is-size-5-mobile">
We provide two methods for specifying a region-layer mapping:
</p>
<ol class="benefits_info is-size-5-mobile ordered_list">
<li>(The preferred method) Passing a <span class="special_format">region_layer_mapping</span> argument in your <span class="special_format">__init__.py</span> model_registry
entry (<a href="https://github.com/brain-score/vision/blob/39b214b9201cf067b4aa9b67777bf9c063871fe0/brainscore_vision/models/voneresnet_50_3/__init__.py#L8">Example</a>).</li>

<pre class="modified_1"><code>
1 model_registry['resnet50_tutorial'] = lambda: ModelCommitment(identifier='resnet50_tutorial',
2 activations_model=get_model('resnet50_tutorial'),
3 layers=get_layers('resnet50_tutorial'),
4 region_layer_mapping={"V1": "layer1", "V2": "layer3", "V4": "layer4", "IT": "fc"})

</code></pre><br>

<li>Providing a <span class="special_format">{model_identifier}.json</span> file in a <span class="special_format">region_layer_map/</span>
folder within your submission package (<a href="https://github.com/brain-score/vision/tree/master/brainscore_vision/models/resnet50_tutorial/region_layer_map">Example</a>).</li>

<pre class="modified_1"><code>
1 {
2 "V1": "layer1",
3 "V2": "layer3",
4 "V4": "layer4",
5 "IT": "fc"
6 }
</code></pre>
</ol>

Importantly, when using the JSON file approach, each layer-mapping for each model identifier must be contained in its own JSON file and be named <span class="special_format">{model_identifier}.json</span>.

</p>
<p class="benefits_info is-size-5-mobile has-text-weight-bold is-italic">
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just an opinion but I find the bold italic text here a little difficult to read.

While optional, we <b>recommend</b> specifying this mapping for models with layers designed to replicate specific brain regions.
If pre-committed, the layer names must match those returned by the <span class="special_format">get_layers()</span> function in the <span class="special_format">model.py</span>.
</p>
<p class="benefits_info is-size-5-mobile">
<b>Note:</b> If a submission does not include a <span class="special_format">region_layer_mapping</span> JSON file, the layer mapping will be automatically computed before scoring using
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about when the layer mapping is specified as in method 1?

<a href="https://github.com/brain-score/vision/blob/39b214b9201cf067b4aa9b67777bf9c063871fe0/brainscore_vision/model_helpers/brain_transformation/__init__.py#L13">STANDARD_REGION_BENCHMARKS</a>.
</p>
<p class="benefits_info is-size-5-mobile ordered_list">
This standard mappping assigns layers to each region <b>independently</b>, meaning:
<li>A region does not need to be mapped to a unique layer.</li>
<li>Regions do not have to follow a specific order.</li>
</p>
<p class="benefits_info is-size-5-mobile">
This intentional flexibility is particularly relevant for models with layers that were intended to replicate specific brain regions, as there is no guarantee that the layer names will match the region names unless pre-committed.
As such, for this types of models, we <b>strongly recommend</b> specifying a region-layer mapping.
</p>
</div>
<div class="box leaderboard-table-component">
<h3 class="benefits_heading is-size-3-mobile">Part 8: Exploring an (optional) requirements.txt File</h3>
<p class="benefits_info is-size-5-mobile shorter">
The (optional) <a href="https://github.com/brain-score/vision/blob/master/brainscore_vision/models/resnet50_tutorial/requirements.txt">requirements.txt file</a>
is where you can add any requirements that your model needs (such as a specific version of a package or an external git
Expand All @@ -214,7 +269,7 @@ <h3 class="benefits_heading is-size-3-mobile">Part 7: Exploring an (optional) re
</p>
</div>
<div class="box leaderboard-table-component">
<h3 class="benefits_heading is-size-3-mobile">Part 8: Putting it All Together </h3>
<h3 class="benefits_heading is-size-3-mobile">Part 9: Putting it All Together </h3>
<p class="benefits_info is-size-5-mobile shorter">
You are almost done! If you were actually submitting a model, the final step prior to submission would be to run
your model locally to ensure that everything is in working order. You can do this by first following the
Expand Down Expand Up @@ -266,7 +321,7 @@ <h3 class="benefits_heading is-size-3-mobile">Part 8: Putting it All Together </
</div>

<div class="box leaderboard-table-component">
<h3 class="benefits_heading is-size-3-mobile">Part 9: Model Summary Tools</h3>
<h3 class="benefits_heading is-size-3-mobile">Part 10: Model Summary Tools</h3>
<p class="benefits_info is-size-5-mobile shorter">
For models built using PyTorch, the <span class="special_format">torchsummary</span>
package can be utilized to get a summary of model information. Install it via pip if necessary
Expand Down