Skip to content

Llama4 FP8 Training Debug - fairscale #1183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open

Llama4 FP8 Training Debug - fairscale #1183

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
ac5e712
fix for high gpu reserved memory
Apr 11, 2022
1915531
Get rid of warning
stephenroller May 11, 2022
ba38cf3
bf16 changes and attribute for cpu activations
ngoyal2707 Jun 28, 2022
8e5c416
changes for pp
Sep 22, 2023
e910320
changes for fp8
vedanuj Sep 28, 2023
0db6e62
Fix fsdp+pp+te WPS decreasing issue (#1139)
jianyuh Oct 2, 2023
71aeffe
Removed extra `cat` before reduce-scatter (#1141)
awgu Oct 3, 2023
17ecf4a
Cleared backward hooks to avoid accumulating over iterations (#1143)
awgu Oct 10, 2023
a8189f0
fix no shard case (#1150)
artkorenev Nov 28, 2023
3b7cc24
[not to be merged yet] added temp changes for fp32 main grad, might n…
ngoyal2707 Dec 7, 2023
a4f02ef
Added reshard hook for frozen params in backward
Jan 12, 2024
f2bb56f
Avoid calling _free_fp16_param_shard() too early with PR 1159
jiecaoyu Feb 21, 2024
1307b1d
Added requires_grad check for params_with_grad method (#1171)
whbldhwj Mar 25, 2024
5faca97
Changed to only run reshard hook if all gradients computed (#1166)
awgu Apr 1, 2024
7bcbc80
Add cast input argument (#1175)
whbldhwj Apr 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading