Code

Add tensor parallelism for RWKV#1237

Code

Open

jahatef wants to merge 30 commits intomainEleutherAI/gpt-neox:mainfrom rwkv-tpEleutherAI/gpt-neox:rwkv-tpCopy head branch name to clipboard

+470-63

Commits

Commits on Jun 4, 2024

• inital tp commits

  jahatef

  committed
  4c7cb11

  Copy full SHA for 4c7cb11

Commits on Jun 19, 2024

• setup

  jahatef

  committed
  46904d5

  Copy full SHA for 46904d5

Commits on Sep 25, 2024

• configs

  jahatef

  committed
  e2933ef

  Copy full SHA for e2933ef

Commits on Oct 3, 2024

• merge

  jahatef

  committed
  d1112ab

  Copy full SHA for d1112ab

• time mixing tp

  jahatef

  committed
  43d641d

  Copy full SHA for 43d641d

Commits on Oct 11, 2024

• time-mixing

  jahatef

  committed
  de02f37

  Copy full SHA for de02f37

Commits on Oct 12, 2024

• time mixing debugging

  jahatef

  committed
  dd441b6

  Copy full SHA for dd441b6

Commits on Oct 13, 2024

• reset time_faaaa

  jahatef

  committed
  a418670

  Copy full SHA for a418670

Commits on Oct 31, 2024

• Add additional asserts and update post training readme (#1300)

  Show description for 540d856
  

  2 peopleauthored and

  jahatef

  committed
  540d856

  Copy full SHA for 540d856

• Fix failling tests (#1301)

  Show description for 12aac35
  

  2 peopleauthored and

  jahatef

  committed
  12aac35

  Copy full SHA for 12aac35

• inital tp commits

  jahatef

  committed
  97c7915

  Copy full SHA for 97c7915

Commits on Nov 5, 2024

• merge

  jahatef

  committed
  5f89ed8

  Copy full SHA for 5f89ed8

• Add ERROR logging prefix and sort the prefixes alphabetically (#1308)

  Show description for 91cb759
  

  TheBatmanofButler

  authored and

  jahatef

  committed
  91cb759

  Copy full SHA for 91cb759

• inital tp commits

  jahatef

  committed
  49b263a

  Copy full SHA for 49b263a

Commits on Nov 6, 2024

• cleanup

  jahatef

  committed
  48de682

  Copy full SHA for 48de682

• cleanup

  jahatef

  committed
  c6fac96

  Copy full SHA for c6fac96

• Update local_setup.yml

  jahatef

  authored
  5a259c0

  Copy full SHA for 5a259c0

Commits on Nov 10, 2024

• add Triton FLA

  jahatef

  committed
  c2d6c85

  Copy full SHA for c2d6c85

Commits on Nov 12, 2024

• change version of rwkv-fla

  jahatef

  committed
  bdb3658

  Copy full SHA for bdb3658

Commits on Nov 29, 2024

• fix a GQA issue (#1314) (#1315)

  Show description for ff7f328
  

  tiandeyu-cs

  authored and

  jahatef

  committed
  ff7f328

  Copy full SHA for ff7f328

• fix 'intermediate_size' in Llama configuration files after the 'mlp_type' option was removed (#1309)

  Show description for 1350b2c
  

  3 peoplecommitted
  1350b2c

  Copy full SHA for 1350b2c

• Python 3.10 support (#1313)

  Show description for c4d7a54
  

  2 peopleauthored and

  jahatef

  committed
  c4d7a54

  Copy full SHA for c4d7a54

• Fix documentation for converting SFT/DPO weights back to HF Llama (#1318)

  

  jacobthebanana

  authored and

  jahatef

  committed
  ee2f142

  Copy full SHA for ee2f142

• fix bug (#1311)

  

  AI-WAIFU

  authored and

  jahatef

  committed
  6e81f0b

  Copy full SHA for 6e81f0b

• Add support for dropout in sparse attention (#1312)

  

  michaelc-yu

  authored and

  jahatef

  committed
  df95419

  Copy full SHA for df95419

• adds pyproject files and tests (#1302)

  Show description for d682529
  

  2 peopleauthored and

  jahatef

  committed
  d682529

  Copy full SHA for d682529

• undo merge error (#1325)

  

  Quentin-Anthony

  authored and

  jahatef

  committed
  0bc11d6

  Copy full SHA for 0bc11d6

• inital tp commits

  jahatef

  committed
  c6db95c

  Copy full SHA for c6db95c

• setup

  jahatef

  committed
  daac503

  Copy full SHA for daac503

Commits on Dec 19, 2024

• Merge branch 'main' into rwkv-tp

  Quentin-Anthony

  committed
  bf478ce

  Copy full SHA for bf478ce