Code

Adds mixing loader for FSL datasets#70

Code

Merged

undfined merged 61 commits intomainallenai/OLMo-core:mainfrom undfined/mixing-loaderallenai/OLMo-core:undfined/mixing-loaderCopy head branch name to clipboardNov 1, 2024

+1,231-93

Commits

Commits on Oct 18, 2024

• WIP: Generate a mixture dataset

  undfined

  committed
  a8b5ec6

  Copy full SHA for a8b5ec6

• WIP: Adds dry run

  undfined

  committed
  637fee9

  Copy full SHA for 637fee9

• Test cleanup

  undfined

  committed
  346135c

  Copy full SHA for 346135c

Commits on Oct 19, 2024

• WIP: Make it fast

  undfined

  committed
  53def38

  Copy full SHA for 53def38

• WIP: Simple benchmark

  undfined

  committed
  8649cc8

  Copy full SHA for 8649cc8

Commits on Oct 23, 2024

• WIP: Refactor

  undfined

  committed
  e3d7011

  Copy full SHA for e3d7011

• Launch script

  undfined

  committed
  efe766b

  Copy full SHA for efe766b

• temp changes to test

  undfined

  committed
  5dff40c

  Copy full SHA for 5dff40c

• deps for now

  undfined

  committed
  2703538

  Copy full SHA for 2703538

• Try with session

  undfined

  committed
  3ee3278

  Copy full SHA for 3ee3278

• Try internal client

  undfined

  committed
  cd1c6d2

  Copy full SHA for cd1c6d2

• Try boto3

  undfined

  committed
  9895b23

  Copy full SHA for 9895b23

Commits on Oct 24, 2024

• Fixes

  undfined

  committed
  3c15f52

  Copy full SHA for 3c15f52

• ?

  undfined

  committed
  0c9355b

  Copy full SHA for 0c9355b

• Cleanup + session stuff

  undfined

  committed
  abb362a

  Copy full SHA for abb362a

• Use environ

  undfined

  committed
  82a1af9

  Copy full SHA for 82a1af9

• JUST use env vars please boto

  undfined

  committed
  d0a80ba

  Copy full SHA for d0a80ba

• No unions of containers

  undfined

  committed
  e621f8e

  Copy full SHA for e621f8e

• prepare first

  undfined

  committed
  0689c42

  Copy full SHA for 0689c42

• Loader handles prepare

  undfined

  committed
  8ab2e99

  Copy full SHA for 8ab2e99

• Try recording torch exceptions

  undfined

  committed
  dcfda67

  Copy full SHA for dcfda67

• Don't need overrides

  undfined

  committed
  23a0806

  Copy full SHA for 23a0806

• Figure out why config/creds are missing

  undfined

  committed
  8cfa282

  Copy full SHA for 8cfa282

• fmt

  undfined

  committed
  fd1a508

  Copy full SHA for fd1a508

• Env not ready yet

  undfined

  committed
  01a40ea

  Copy full SHA for 01a40ea

• print beaker user

  undfined

  committed
  8bde2b3

  Copy full SHA for 8bde2b3

• uncomment eval file

  undfined

  committed
  ae208f6

  Copy full SHA for ae208f6

• replicate CommonComponents setup

  undfined

  committed
  ce9d06f

  Copy full SHA for ce9d06f

• Some class init stuff

  undfined

  committed
  d1eb4df

  Copy full SHA for d1eb4df

• Some more config logging

  undfined

  committed
  dbce279

  Copy full SHA for dbce279

• Conflict in CHANGELOG

  undfined

  committed
  5daa274

  Copy full SHA for 5daa274

• checks cleanup

  undfined

  committed
  980e05a

  Copy full SHA for 980e05a

Commits on Oct 25, 2024

• Fixes for duplicate paths in mixture

  undfined

  committed
  f27bd73

  Copy full SHA for f27bd73

• In case there a ton of files

  undfined

  committed
  c69b228

  Copy full SHA for c69b228

• Maybe fix trainer launch

  undfined

  committed
  18efafd

  Copy full SHA for 18efafd

• Match other example

  undfined

  committed
  5ceed46

  Copy full SHA for 5ceed46

• More tests

  undfined

  committed
  4c7513e

  Copy full SHA for 4c7513e

• Try diff gpus

  undfined

  committed
  8401580

  Copy full SHA for 8401580

• keep fsdp

  undfined

  committed
  0d77422

  Copy full SHA for 0d77422

• checks

  undfined

  committed
  d22ed10

  Copy full SHA for d22ed10

• Less tokens

  undfined

  committed
  68a4d28

  Copy full SHA for 68a4d28

• Exclude ai2/allennlp-elanding-a100-40g temp

  undfined

  committed
  c35514c

  Copy full SHA for c35514c

Commits on Oct 28, 2024

• Merge branch 'main' of github.com:allenai/OLMo-core into undfined/mixing-loader

  undfined

  committed
  02cb49b

  Copy full SHA for 02cb49b

• Feedback

  undfined

  committed
  c453e65

  Copy full SHA for c453e65

• Drop examples

  undfined

  committed
  a288d9e

  Copy full SHA for a288d9e

• A bit more cleanup

  undfined

  committed
  9c49f25

  Copy full SHA for 9c49f25

• Outdated changelog

  undfined

  committed
  89504bc

  Copy full SHA for 89504bc

• Unused deps

  undfined

  committed
  3aa5c35

  Copy full SHA for 3aa5c35

• One more dep

  undfined

  committed
  8f729dd

  Copy full SHA for 8f729dd

• uncomment test assertions

  undfined

  committed
  a848195

  Copy full SHA for a848195

• Drop todo

  undfined

  committed
  5322bf1

  Copy full SHA for 5322bf1

• 0 is an invalid token

  undfined

  committed
  5c22665

  Copy full SHA for 5c22665

Commits on Oct 29, 2024

• More feedback

  undfined

  committed
  87e9168

  Copy full SHA for 87e9168

• Randomly sample instances when segmenting

  undfined

  committed
  fe50a32

  Copy full SHA for fe50a32

Commits on Oct 30, 2024

• Memray + limit marker

  undfined

  committed
  cffcba3

  Copy full SHA for cffcba3

• Add dep

  undfined

  committed
  293be02

  Copy full SHA for 293be02

• Lint

  undfined

  committed
  e45d2c3

  Copy full SHA for e45d2c3

• Bigger array is more informative

  undfined

  committed
  ffe7660

  Copy full SHA for ffe7660

• Merge branch 'main' into undfined/mixing-loader

  undfined

  authored
  19db2a9

  Copy full SHA for 19db2a9

• Feedback

  undfined

  committed
  1fdb995

  Copy full SHA for 1fdb995

Commits on Nov 1, 2024

• Merge branch 'main' into undfined/mixing-loader

  undfined

  authored
  74d4c5a

  Copy full SHA for 74d4c5a