Updated Megatron version #85
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Selvaraj Anandaraj <[email protected]>
Signed-off-by: jiemingz <[email protected]>
Signed-off-by: jiemingz <[email protected]>
Signed-off-by: jiemingz <[email protected]>
Signed-off-by: Selvaraj Anandaraj <[email protected]>
add is_first_microbatch for TE See merge request ADLR/megatron-lm!1033
Need a switch at NeMo level to enable Atomic GEMM See merge request ADLR/megatron-lm!1017
Add distributed checkpoint support to non-TE based models See merge request ADLR/megatron-lm!1005
Signed-off-by: Selvaraj Anandaraj <[email protected]>
Signed-off-by: Selvaraj Anandaraj <[email protected]>
Support for activation offloading to CPU in M-LM See merge request ADLR/megatron-lm!1016
add rope and swiglu fusion See merge request ADLR/megatron-lm!946
Add jit_fuser to switch between torch.jit.script and torch.compile See merge request ADLR/megatron-lm!1036
Run black on megatron/optimizer See merge request ADLR/megatron-lm!1050
…ommunication Compute norm once per batch (instead of once per microbatch) and once per bucket (instead of once per param)
Fix NaN checking in grads: should be performed before data-parallel all-reduce See merge request ADLR/megatron-lm!989
…ry ffn_hidden_size
Move to Draco OCI See merge request ADLR/megatron-lm!1137
Print number of transformer and embedding parameters separately See merge request ADLR/megatron-lm!1159
Mcore LLaVA model See merge request ADLR/megatron-lm!1151
[OMNIML-614] AMMO ptq + TensorRT-LLM export examples for megatron-lm See merge request ADLR/megatron-lm!1013
Make throughput and memory footprint formulae compatible with arbitrary ffn_hidden_size See merge request ADLR/megatron-lm!1169
Experimental Yaml configs See merge request ADLR/megatron-lm!1134
This reverts commit fe1f23c.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
30 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.