Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Review changes for sweep experiment #4

Open
wants to merge 89 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
2cb88f7
concept editing experiments
AlexTMallen Sep 23, 2023
6ffb97e
top1 and loss matrices
AlexTMallen Sep 23, 2023
99ca187
fix segfault
AlexTMallen Sep 23, 2023
c2d4749
both loss matrices
AlexTMallen Sep 23, 2023
e9250b9
plotting loss matrices
AlexTMallen Sep 23, 2023
bfadef8
Merge branch 'main' into editing-experiments
AlexTMallen Sep 23, 2023
5407aae
use double precision
AlexTMallen Sep 24, 2023
bbdff6c
Merge branch 'main' into editing-experiments
AlexTMallen Sep 24, 2023
c93ca68
naive quadratic erasure and editing
AlexTMallen Sep 24, 2023
870a365
create sentiment dataset
AlexTMallen Sep 26, 2023
ed96b3f
Merge branch 'main' into editing-experiments
AlexTMallen Sep 26, 2023
06a9054
Prepare for gutting Sweep.run
norabelrose Sep 26, 2023
58d3dfe
Refactor MVP
norabelrose Sep 26, 2023
ffbfb93
Don't early stop too early
norabelrose Sep 26, 2023
ba1d62e
minor changes
AlexTMallen Sep 26, 2023
7582992
Merge branch 'refactor' into editing-experiments
AlexTMallen Sep 26, 2023
5ca1ec0
Better hparams
norabelrose Sep 26, 2023
18c050e
Merge branch 'refactor' into editing-experiments
AlexTMallen Sep 26, 2023
2d6b815
Remove duplicate code in Sweep.run
norabelrose Sep 26, 2023
2f8cb88
Backtracking in Probe.fit
norabelrose Sep 26, 2023
45b1dea
batched eval; use CIFAR test; multiple random seeds
AlexTMallen Sep 26, 2023
3e3518e
Sweep now calls preprocessor on test set
norabelrose Sep 26, 2023
9bfb71a
more efficient eval
AlexTMallen Sep 26, 2023
ed6acd3
Merge remote-tracking branch 'origin/refactor' into editing-experiments
AlexTMallen Sep 26, 2023
d08f398
MlpProbe now has a residual architecture
norabelrose Sep 27, 2023
4597bd3
sentiment erasure, fix binary loss
AlexTMallen Sep 27, 2023
99488cc
label smoothing
norabelrose Sep 27, 2023
0f894b2
merge with refactor
AlexTMallen Sep 27, 2023
87fdbea
Merge branch 'editing-experiments' of github.com:EleutherAI/mdl into …
AlexTMallen Sep 27, 2023
917de33
Merge branch 'refactor' into editing-experiments
AlexTMallen Sep 27, 2023
71be6e7
remove gradscaler
norabelrose Sep 27, 2023
2fc340d
improved visionprobe for editing experiment
AlexTMallen Sep 27, 2023
a1c85ff
Merge branch 'refactor' into editing-experiments
AlexTMallen Sep 27, 2023
de35b2c
use vanilla LEACE in sweep
AlexTMallen Sep 30, 2023
351e7f8
save state
luciaquirke Oct 22, 2024
77b7c2c
Save state
luciaquirke Oct 22, 2024
7dbcf99
add gist with two more nets - vit and convnext
luciaquirke Oct 24, 2024
acb478d
Add probes; add logging; add new cli for sweep
luciaquirke Oct 24, 2024
e4c07c4
log distance from init in model params
luciaquirke Oct 24, 2024
bc9220a
update gitignore
luciaquirke Oct 24, 2024
5de0031
Add scalable resnet probe; fix trivial bugs
luciaquirke Oct 24, 2024
25a6fae
automate wandb run naming
luciaquirke Oct 24, 2024
21a2a8c
resolve stray todos
luciaquirke Oct 25, 2024
77ec2f4
remove old cli
luciaquirke Oct 28, 2024
7f452aa
plot mdl
luciaquirke Oct 30, 2024
c78161d
restrict logging to final MDL chunk; plot cleanup; check for duplicat…
luciaquirke Oct 31, 2024
d8d68b6
save progress
luciaquirke Nov 14, 2024
8af8fbd
Add muP; add checkpointing; add MlpProbe activation variations (swigl…
luciaquirke Nov 22, 2024
b72edcb
Add mdl and loss plots with activation functions plotted together
luciaquirke Nov 23, 2024
79ea580
add polyapprox
luciaquirke Dec 8, 2024
b38bdc1
Update plot
luciaquirke Dec 8, 2024
999ba7c
merge
luciaquirke Dec 8, 2024
ead5f68
Save progress
luciaquirke Dec 9, 2024
b00db14
Save progress
luciaquirke Dec 9, 2024
1d753a4
add cifarnet
luciaquirke Dec 12, 2024
843d729
Enable Muon in MLPProbe; enable datasets larger than can fit in VRAM …
luciaquirke Dec 13, 2024
6d80bd3
Merge branch 'quad-mlp' of github.com:EleutherAI/mdl into quad-mlp
luciaquirke Dec 13, 2024
7d4729a
merge
luciaquirke Dec 13, 2024
c70939e
fix bug
luciaquirke Dec 13, 2024
13afcb1
Enable training on cifarnet
luciaquirke Dec 16, 2024
c2977da
Add alf-qleace
luciaquirke Dec 16, 2024
0e77e57
merge
luciaquirke Dec 16, 2024
656c985
Update plots
luciaquirke Dec 17, 2024
bacd9c3
merge
luciaquirke Dec 17, 2024
e1db7c5
add alf-qleace to sweep
luciaquirke Dec 17, 2024
84f521d
changes
luciaquirke Dec 18, 2024
2c1910d
fixup
luciaquirke Dec 18, 2024
16aaa66
Add shrinkage and orth; add lr sweep
luciaquirke Dec 20, 2024
4407231
quadratically guard
luciaquirke Dec 20, 2024
dbe8552
tweak iterative quadratic erasure hyperparameters
luciaquirke Jan 5, 2025
2a41748
Add iterative erasure; add parameter-matched LeNet; measure gain from…
luciaquirke Jan 9, 2025
c447491
Set seed
luciaquirke Jan 9, 2025
2bad6fd
Remove redundant code from svd
luciaquirke Jan 10, 2025
7eaf791
Plot sample erased images
luciaquirke Jan 20, 2025
736b0eb
Update plots
luciaquirke Jan 21, 2025
0cc7065
Remove /mnt location
luciaquirke Jan 24, 2025
4fc5d7b
Merge
luciaquirke Jan 24, 2025
114b0bd
update gitignore
luciaquirke Jan 24, 2025
52b1420
fix bug
luciaquirke Jan 24, 2025
5a68dc5
Update plots
luciaquirke Jan 24, 2025
b40a495
Merge branch 'quad-mlp' of github.com:EleutherAI/mdl into quad-mlp
luciaquirke Jan 24, 2025
90a0bc5
support leaced and erased ds
luciaquirke Jan 24, 2025
5e66bb9
Simplify
luciaquirke Jan 27, 2025
2e7ad4d
Update plots
luciaquirke Jan 27, 2025
e8272b5
Improve seed file handling
luciaquirke Jan 27, 2025
4e77ab0
merge
luciaquirke Jan 27, 2025
cbe50ca
Get old vision dimensions working
luciaquirke Jan 28, 2025
d0875c5
torch compile
luciaquirke Jan 28, 2025
d31c751
merge
luciaquirke Jan 28, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
data

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
Expand Down Expand Up @@ -158,3 +160,16 @@ cython_debug/
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/


erasers_cache
lightning_logs
wandb
images/

probe-ckpts/
*.bsh
wandb*

24-11-21
24-11-21-seeds
4 changes: 0 additions & 4 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,3 @@ repos:
hooks:
- id: ruff
args: [--fix, --exit-non-zero-on-fix]
- repo: https://github.com/codespell-project/codespell
rev: v2.2.4
hooks:
- id: codespell
Loading