Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
a8184ad
added code to help with inserting sparse mlps into TinyModel
noanabeshima Jul 5, 2024
cc8c00b
sparse mlp registration
noanabeshima Jul 5, 2024
1f91855
added some more default mlps
noanabeshima Jul 7, 2024
2eb138f
updated default sparse mlp names
noanabeshima Jul 7, 2024
b3277d0
more fix
noanabeshima Jul 7, 2024
2dc9261
fixes around error grads and added option to disable flashattn to the…
noanabeshima Jul 8, 2024
8a30176
disable flashattn code
noanabeshima Jul 8, 2024
f0ef473
renamed mlp_map to transcoder and changed tags. mlp_out is now M and …
noanabeshima Jul 9, 2024
9a76e26
added mlp out sae to tinymodel
noanabeshima Jul 9, 2024
b9ad149
more support
noanabeshima Jul 9, 2024
e557b3c
changes
noanabeshima Jul 12, 2024
8646f92
changed default saes
noanabeshima Jul 16, 2024
fa2c059
updated version
noanabeshima Jul 16, 2024
f946a67
added words.json
noanabeshima Jul 16, 2024
70ba176
modified .gitignore
noanabeshima Jul 16, 2024
6f5f0f4
added words
noanabeshima Jul 16, 2024
1f9718f
Add res_pre_attn_sae and res_pre_mlp_sae support and defaults
noanabeshima Jul 26, 2024
2ecb6d3
Add res_pre_mlp/res_pre_attn support
noanabeshima Jul 26, 2024
fef75a4
Update version
noanabeshima Jul 26, 2024
dac8fce
add weights_only=True to torch.load calls
noanabeshima Sep 4, 2024
d2e8fae
Add get_sliced_mlp, get_masked_mlp
noanabeshima Sep 4, 2024
db3a47a
tweaks to lm.py
noanabeshima Sep 4, 2024
163115f
Remove assertion in get_state_dict specifying which models are available
noanabeshima Sep 4, 2024
d2143a6
Remove get_state_dict docstring
noanabeshima Sep 4, 2024
353d8f2
Merge branch 'main' into feature-circuits
noanabeshima Sep 4, 2024
2fdd1cb
updated version
noanabeshima Sep 4, 2024
5c7233d
renamed tiny_model to tinymodel
noanabeshima Sep 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ testing.ipynb
dist/
*.gitignore
mlp_map_test/
attn_test/
simulation.py
test.pt
attn_test/
test.pt
mlp/

2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This library is in an alpha state, it probably has some bugs. Please let me know


```
from tiny_model import TinyModel, tokenizer
from tinymodel import TinyModel, tokenizer

lm = TinyModel()

Expand Down
6 changes: 3 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
[tool.poetry]
name = "tinymodel"
version = "0.1.2-1"
version = "0.1.2-2"
description = "A small TinyStories LM with SAEs and transcoders"
authors = ["Noa Nabeshima <[email protected]>"]
readme = "README.md"
packages = [{include = "tiny_model"}]
packages = [{include = "tinymodel"}]

[tool.poetry.dependencies]
python = "^3.11"
Expand All @@ -19,7 +19,7 @@ datasets = "^2.20.0"


[[tool.poetry.source]]
name = "tiny_model"
name = "tinymodel"
url = "https://github.com/noanabeshima/tiny_model"
priority = "explicit"

Expand Down
261 changes: 0 additions & 261 deletions tiny_model/lm.py

This file was deleted.

38 changes: 0 additions & 38 deletions tiny_model/sparse_mlp.py

This file was deleted.

4 changes: 2 additions & 2 deletions tiny_model/__init__.py → tinymodel/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
from .lm import TinyModel
from .lm import TinyModel, parse_mlp_tag
from .sparse_mlp import SparseMLP
from .tokenization.tokenization import dec, enc, tok_see, tokenizer, raw_toks, toks
from .tokenization.tokenization import dec, enc, tok_see, tokenizer, raw_toks, pretty_toks, multi_tok_words, words
Loading