Code

DCLM Style Deduplications#214

Code

Open

revbucket wants to merge 11 commits intomainallenai/dolma:mainfrom mattj/bff-0924allenai/dolma:mattj/bff-0924Copy head branch name to clipboard

+1,243-723

Commits

Commits on Sep 19, 2024

• Untested commit for many DCLM-dedup, plus many more changes

  

  Matt Jordan

  committed
  cb57a09

  Copy full SHA for cb57a09

Commits on Sep 20, 2024

• Fixed some CLI stuff -- paragraph seems to work, but still need to test dclm

  

  Matt Jordan

  committed
  69839ed

  Copy full SHA for 69839ed

Commits on Sep 25, 2024

• Pushing local changes to s3 to verify against bff --hopefully working

  

  Matt Jordan

  committed
  e0992c4

  Copy full SHA for e0992c4

Commits on Sep 26, 2024

• bugfixy for some edge cases

  

  Matt Jordan

  committed
  bc28fc9

  Copy full SHA for bc28fc9

Commits on Sep 27, 2024

• Maybe some slightly better logging?

  

  Matt Jordan

  committed
  316283e

  Copy full SHA for 316283e

• Updated docs/examples wth new dedupe method

  

  Matt Jordan

  committed
  da10949

  Copy full SHA for da10949

Commits on Sep 30, 2024

• Merge branch 'main' into mattj/bff-0924

  

  Matt Jordan

  committed
  0af2db6

  Copy full SHA for 0af2db6

Commits on Oct 16, 2024

• Tests failing, need to chack back with main

  

  Matt Jordan

  committed
  0903484

  Copy full SHA for 0903484

• oops

  

  Matt Jordan

  committed
  e858388

  Copy full SHA for e858388

• Oops x2

  

  Matt Jordan

  committed
  67a4a9a

  Copy full SHA for 67a4a9a

Commits on Jan 15, 2025

• DCLM style tests written and all tests passing

  

  Matt Jordan

  committed
  70dc9d4

  Copy full SHA for 70dc9d4