-
Notifications
You must be signed in to change notification settings - Fork 7
Add region layer mapping information to deep dive 2 #380
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
KartikP
wants to merge
11
commits into
master
Choose a base branch
from
kp/layer-mapping-JSON-tutorial
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 6 commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
5905f10
Add region layer mapping information to deep dive
KartikP c778790
Additional details
KartikP dd18416
Improved grammar and formatting
KartikP 81ee5c0
Changed bolding around
KartikP 10f93b8
Detail about file naming convention
KartikP 6d300b2
Fixed some
KartikP d0c844d
Update benchmarks/templates/benchmarks/tutorials/models/deepdive_2.html
KartikP 38bb673
Update benchmarks/templates/benchmarks/tutorials/models/deepdive_2.html
KartikP 5709964
Merge branch 'master' into kp/layer-mapping-JSON-tutorial
KartikP d5e18d6
Merge branch 'master' into kp/layer-mapping-JSON-tutorial
KartikP 9308ab8
Merge branch 'master' into kp/layer-mapping-JSON-tutorial
KartikP File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -197,7 +197,62 @@ <h3 class="benefits_heading is-size-3-mobile">Part 6: Exploring the model.py Fil | |
| </code></pre> | ||
| </div> | ||
| <div class="box leaderboard-table-component"> | ||
| <h3 class="benefits_heading is-size-3-mobile">Part 7: Exploring an (optional) requirements.txt File</h3> | ||
| <h3 class="benefits_heading is-size-3-mobile">Part 7: Exploring the (optional) Region-Layer Mapping</h3> | ||
| <p class="benefits_info is-size-5-mobile"> | ||
| Brain-Score allows users to specify a region-layer mapping for models. This mapping assigns a specific layer from <span class="special_format">model.py</span> file | ||
| to a particular brain region, ensuring it remains fixed when evaluated across different benchmarks. | ||
| </p> | ||
| <p class="benefits_info is-size-5-mobile"> | ||
| We provide two methods for specifying a region-layer mapping: | ||
| </p> | ||
| <ol class="benefits_info is-size-5-mobile ordered_list"> | ||
| <li>(The preferred method) Passing a <span class="special_format">region_layer_mapping</span> argument in your <span class="special_format">__init__.py</span> model_registry | ||
| entry (<a href="https://github.com/brain-score/vision/blob/39b214b9201cf067b4aa9b67777bf9c063871fe0/brainscore_vision/models/voneresnet_50_3/__init__.py#L8">Example</a>).</li> | ||
|
|
||
| <pre class="modified_1"><code> | ||
| 1 model_registry['resnet50_tutorial'] = lambda: ModelCommitment(identifier='resnet50_tutorial', | ||
| 2 activations_model=get_model('resnet50_tutorial'), | ||
| 3 layers=get_layers('resnet50_tutorial'), | ||
| 4 region_layer_mapping={"V1": "layer1", "V2": "layer3", "V4": "layer4", "IT": "fc"}) | ||
|
|
||
| </code></pre><br> | ||
|
|
||
| <li>Providing a <span class="special_format">{model_identifier}.json</span> file in a <span class="special_format">region_layer_map/</span> | ||
| folder within your submission package (<a href="https://github.com/brain-score/vision/tree/master/brainscore_vision/models/resnet50_tutorial/region_layer_map">Example</a>).</li> | ||
|
|
||
| <pre class="modified_1"><code> | ||
| 1 { | ||
| 2 "V1": "layer1", | ||
| 3 "V2": "layer3", | ||
| 4 "V4": "layer4", | ||
| 5 "IT": "fc" | ||
| 6 } | ||
| </code></pre> | ||
| </ol> | ||
|
|
||
| Importantly, when using the JSON file approach, each layer-mapping for each model identifier must be contained in its own JSON file and be named <span class="special_format">{model_identifier}.json</span>. | ||
|
|
||
| </p> | ||
| <p class="benefits_info is-size-5-mobile has-text-weight-bold is-italic"> | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just an opinion but I find the bold italic text here a little difficult to read. |
||
| While optional, we <b>recommend</b> specifying this mapping for models with layers designed to replicate specific brain regions. | ||
| If pre-committed, the layer names must match those returned by the <span class="special_format">get_layers()</span> function in the <span class="special_format">model.py</span>. | ||
| </p> | ||
| <p class="benefits_info is-size-5-mobile"> | ||
| <b>Note:</b> If a submission does not include a <span class="special_format">region_layer_mapping</span> JSON file, the layer mapping will be automatically computed before scoring using | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What about when the layer mapping is specified as in method 1? |
||
| <a href="https://github.com/brain-score/vision/blob/39b214b9201cf067b4aa9b67777bf9c063871fe0/brainscore_vision/model_helpers/brain_transformation/__init__.py#L13">STANDARD_REGION_BENCHMARKS</a>. | ||
| </p> | ||
| <p class="benefits_info is-size-5-mobile ordered_list"> | ||
| This standard mappping assigns layers to each region <b>independently</b>, meaning: | ||
| <li>A region does not need to be mapped to a unique layer.</li> | ||
| <li>Regions do not have to follow a specific order.</li> | ||
| </p> | ||
| <p class="benefits_info is-size-5-mobile"> | ||
| This intentional flexibility is particularly relevant for models with layers that were intended to replicate specific brain regions, as there is no guarantee that the layer names will match the region names unless pre-committed. | ||
| As such, for this types of models, we <b>strongly recommend</b> specifying a region-layer mapping. | ||
KartikP marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| </p> | ||
| </div> | ||
| <div class="box leaderboard-table-component"> | ||
| <h3 class="benefits_heading is-size-3-mobile">Part 8: Exploring an (optional) requirements.txt File</h3> | ||
| <p class="benefits_info is-size-5-mobile shorter"> | ||
| The (optional) <a href="https://github.com/brain-score/vision/blob/master/brainscore_vision/models/resnet50_tutorial/requirements.txt">requirements.txt file</a> | ||
| is where you can add any requirements that your model needs (such as a specific version of a package or an external git | ||
|
|
@@ -214,7 +269,7 @@ <h3 class="benefits_heading is-size-3-mobile">Part 7: Exploring an (optional) re | |
| </p> | ||
| </div> | ||
| <div class="box leaderboard-table-component"> | ||
| <h3 class="benefits_heading is-size-3-mobile">Part 8: Putting it All Together </h3> | ||
| <h3 class="benefits_heading is-size-3-mobile">Part 9: Putting it All Together </h3> | ||
| <p class="benefits_info is-size-5-mobile shorter"> | ||
| You are almost done! If you were actually submitting a model, the final step prior to submission would be to run | ||
| your model locally to ensure that everything is in working order. You can do this by first following the | ||
|
|
@@ -266,7 +321,7 @@ <h3 class="benefits_heading is-size-3-mobile">Part 8: Putting it All Together </ | |
| </div> | ||
|
|
||
| <div class="box leaderboard-table-component"> | ||
| <h3 class="benefits_heading is-size-3-mobile">Part 9: Model Summary Tools</h3> | ||
| <h3 class="benefits_heading is-size-3-mobile">Part 10: Model Summary Tools</h3> | ||
| <p class="benefits_info is-size-5-mobile shorter"> | ||
| For models built using PyTorch, the <span class="special_format">torchsummary</span> | ||
| package can be utilized to get a summary of model information. Install it via pip if necessary | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.