Store endpoint score object in S3

[KartikP]

No clear purpose at the moment to preserve this information but good to have. Cheap to store.

Stores under appropriate domain subdirectory in brainscore-storage bucket under a new folder called score-obj

Estimated cost for storage for score metadata

Benchmark Data:

• Alexnet MajajHong - 46kb (small)
• Alexnet Papale2025 IT - 122kb
• Alexnet NSD IT - 28.5mb (large)

S3 cost: $0.023/GB/month

• 100 small benchmarks x 100 models = 0.46GB → $0.01/month
• 100 large benchmarks x 100 models = 285GB → $6.56/month

Flow

from brainscore_vision import score

model_id = 'alexnet'
benchmark_id = 'MajajHong2015.IT-pls'

score_result = score(model_identifier=model_id, benchmark_identifier=benchmark_id)
print(f'Score: {score_result}')

[/opt/anaconda3/envs/vision-2026/lib/python3.11/site-packages/torchvision/models/_utils.py:208](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/vision-2026/lib/python3.11/site-packages/torchvision/models/_utils.py:208): UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
[/opt/anaconda3/envs/vision-2026/lib/python3.11/site-packages/torchvision/models/_utils.py:223](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/vision-2026/lib/python3.11/site-packages/torchvision/models/_utils.py:223): UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
cross-validation:   0%|          | 0/10 [00:00<?, ?it/s]/opt/anaconda3/envs/vision-2026/lib/python3.11/site-packages/sklearn/cross_decomposition/_pls.py:113: ConvergenceWarning: Maximum number of iterations reached
  warnings.warn("Maximum number of iterations reached", ConvergenceWarning)
cross-validation: 100%|██████████| 10/10 [01:25<00:00,  8.59s/it]
Score: <xarray.Score ()>
array(0.41748003)
Attributes:
    error:                 <xarray.Score ()>\narray(0.00315545)
    raw:                   <xarray.Score ()>\narray(0.58574912)\nAttributes:\...
    ceiling:               <xarray.DataArray ()>\narray(0.82184059)\nAttribut...
    model_identifier:      alexnet
    benchmark_identifier:  MajajHong2015.IT-pls
    comment:               layers: {'IT': 'features.12', 'V4': 'features.7', ...

import pickle
import tempfile
import boto3

s3_bucket = 'brainscore-storage'
s3_key = f'brainscore-vision/score-obj/{model_id}__{benchmark_id}.pkl'

with tempfile.NamedTemporaryFile(suffix='.pkl', delete=True) as tmp:
    pickle.dump(score_result, tmp, protocol=pickle.HIGHEST_PROTOCOL)
    tmp.flush()
    s3_client = boto3.client('s3')
    s3_client.upload_file(tmp.name, s3_bucket, s3_key)

print(f'Uploaded to s3://{s3_bucket}/{s3_key}')

Uploaded to s3://brainscore-storage/brainscore-vision/score-obj/alexnet__MajajHong2015.IT-pls.pkl

# Verify round-trip: download and check the score object
obj = s3_client.get_object(Bucket=s3_bucket, Key=s3_key)
score_downloaded = pickle.loads(obj['Body'].read())

print(f'Downloaded Score: {score_downloaded}')

xarray.Score
array(0.41748003)
Coordinates: (0)
Attributes:
error :
<xarray.Score ()>
array(0.00315545)
raw :
<xarray.Score ()>
array(0.58574912)
Attributes:
    raw:      <xarray.Score (split: 10, neuroid: 168)>\narray([[0.38244496, 0...
    error:    <xarray.Score ()>\narray(0.00315545)
ceiling :
<xarray.DataArray ()>
array(0.82184059)
Attributes:
    raw:      <xarray.DataArray (split: 10, neuroid: 168)>\narray([[0.3385611...
    error:    <xarray.DataArray ()>\narray(0.00058713)
model_identifier :
alexnet
benchmark_identifier :
MajajHong2015.IT-pls
comment :
layers: {'IT': 'features.12', 'V4': 'features.7', 'V2': 'features.7', 'V1': 'features.2'}

[deirdre-k]

Looks great!