New example: [REDACTED] - #7121
Conversation
Fixed-config script plus its SLURM launcher: SFTTrainer with a load-time DistributedConfig (tp_size=8, fsdp_size=8, expert parallelism with token dispatch) on 64 GPUs. Defaults are the measured operating point; the constraints behind them are in the docstring.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e4a616384b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…d rendezvous ignores
1 participant
LoRA SFT on [REDACTED] on 8 nodes with
SFTTrainer, sharded at load time by transformers'DistributedConfig.Same shape as #6869's examples.
Needs huggingface/transformers#48204.
Note
Low Risk
Documentation and a self-contained example only; no changes to TRL trainers or library APIs.
Overview
Adds
sft_moe_expert_parallel, a distributed-training example that runs LoRA SFT on a large MoE withSFTTraineracross 8 nodes (64 GPUs). The training script wirestransformers.distributed.DistributedConfigintomodel_init_kwargs(tp_size=8,fsdp_size=8, expert parallel + token dispatch) so the checkpoint is sharded at load time, uses packed sequences,save_only_model=True(no resume), and documents pins on transformersep-fsdp-2d-meshand peft main.A companion
sft_moe_expert_parallel.slurmlaunchestorchrunvia Slurm with IPv4 rendezvous.docs/source/example_overview.mdgains an index row for the new folder.Reviewed by Cursor Bugbot for commit 878a8c3. Bugbot is set up for automated code reviews on this repo. Configure here.