Code

Adding Pile as a SeqIO task#8

Code

Draft

thomasw21 wants to merge 37 commits intomainbigscience-workshop/architecture-objective:mainfrom thomas/script_for_pile_datasetbigscience-workshop/architecture-objective:thomas/script_for_pile_datasetCopy head branch name to clipboard

+295

Commits

Commits on Nov 5, 2021

• First draft of how I think seqio works

  thomasw21

  committed
  54810ce

  Copy full SHA for 54810ce

• Improve pipeline

  thomasw21

  committed
  9a3146c

  Copy full SHA for 9a3146c

Commits on Nov 6, 2021

• Added script to download pile to gcp bucket

  thomasw21

  committed
  bccaa5f

  Copy full SHA for bccaa5f

• I need to send to remove file progressively as I don't have access to enough disk memory

  thomasw21

  committed
  ff7a387

  Copy full SHA for ff7a387

• Woops

  thomasw21

  committed
  adbc5a9

  Copy full SHA for adbc5a9

• Woops2

  thomasw21

  committed
  4cb7ae8

  Copy full SHA for 4cb7ae8

• convert o map instead

  thomasw21

  committed
  1096de7

  Copy full SHA for 1096de7

• Woops 3

  thomasw21

  committed
  f53f21b

  Copy full SHA for f53f21b

• Make it sequential

  thomasw21

  committed
  a1f6e0a

  Copy full SHA for a1f6e0a

• Remove recursive option

  thomasw21

  committed
  a9d69f7

  Copy full SHA for a9d69f7

• Update script to remove uncompressed file as well

  thomasw21

  committed
  68255a1

  Copy full SHA for 68255a1

• Test out the rest of the script

  thomasw21

  committed
  744cc35

  Copy full SHA for 744cc35

• Woops

  thomasw21

  committed
  e4502a6

  Copy full SHA for e4502a6

• Add back download step

  thomasw21

  committed
  340ed01

  Copy full SHA for 340ed01

Commits on Nov 8, 2021

• Cache seqio task

  thomasw21

  committed
  4a58865

  Copy full SHA for 4a58865

• Add script in order to run caching

  thomasw21

  committed
  8995fbf

  Copy full SHA for 8995fbf

• Test something out

  thomasw21

  committed
  844c426

  Copy full SHA for 844c426

• test something else:

  thomasw21

  committed
  3e294d9

  Copy full SHA for 3e294d9

• Woops

  thomasw21

  committed
  774e3e7

  Copy full SHA for 774e3e7

• Setup file needs to be names setup.py

  thomasw21

  committed
  964340d

  Copy full SHA for 964340d

• Make a another package

  thomasw21

  committed
  e17d454

  Copy full SHA for e17d454

• Somehow task is not part of pile package

  thomasw21

  committed
  cc74fb4

  Copy full SHA for cc74fb4

• Revert "Make a another package"

  Show description for 7da3b2a
  thomasw21

  committed
  7da3b2a

  Copy full SHA for 7da3b2a

• Remove __init__

  thomasw21

  committed
  a9edd8a

  Copy full SHA for a9edd8a

• Rename to pile

  thomasw21

  committed
  385f262

  Copy full SHA for 385f262

• Fix

  thomasw21

  committed
  0282f7b

  Copy full SHA for 0282f7b

• We need to install t5

  thomasw21

  committed
  c4cf146

  Copy full SHA for c4cf146

• No need to commit egg-info

  thomasw21

  committed
  f3ea974

  Copy full SHA for f3ea974

• Some more fixing

  thomasw21

  committed
  a91207a

  Copy full SHA for a91207a

• Use 32 workers, ie the number of files

  thomasw21

  committed
  3ced63b

  Copy full SHA for 3ced63b

• update script

  thomasw21

  committed
  4cd94ea

  Copy full SHA for 4cd94ea

• woops

  thomasw21

  committed
  630143e

  Copy full SHA for 630143e

• Woops read wrong doc

  thomasw21

  committed
  7591d12

  Copy full SHA for 7591d12

• Choose machine type

  thomasw21

  committed
  8131647

  Copy full SHA for 8131647

• Preprocess only the two first files

  thomasw21

  committed
  c85fbb3

  Copy full SHA for c85fbb3

Commits on Nov 9, 2021

• Revert "Preprocess only the two first files"

  Show description for 01f9c6d
  thomasw21

  committed
  01f9c6d

  Copy full SHA for 01f9c6d

• Update path of cache

  thomasw21

  committed
  ec3989b

  Copy full SHA for ec3989b