Skip to content

docs: describe training launches in terms of primus-cli - #976

Open
WangLingxun wants to merge 2 commits into
mainfrom
refactor/primus-cli-launchers
Open

docs: describe training launches in terms of primus-cli#976
WangLingxun wants to merge 2 commits into
mainfrom
refactor/primus-cli-launchers

Conversation

@WangLingxun

@WangLingxun WangLingxun commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Documentation-only. Rewrites every guide, example README and config header comment that still told users to export EXP and run examples/run_*.sh, so the whole repository describes launches in terms of runner/primus-cli.

No behaviour changes and no scripts deleted here. examples/run_pretrain.sh and friends still exist on this branch and still work; this PR only stops the documentation from presenting them as the way to launch training. Retiring the scripts, migrating their downstream callers, and the fixes that came out of that migration are in the stacked follow-up #999, which is based on this branch.

  • Launch snippets rewritten to the three runner modes — primus-cli direct, primus-cli container, primus-cli slurm srun — across docs/, examples/, and the header comments of the diffusion and SFT configs. Container environment options stay at the container layer rather than leaking past the -- into the Primus CLI.
  • examples/README.md gains a mode table explaining when to use direct / container / slurm, the -- argument shape, and where the environment defaults and setup hooks live (runner/helpers/envs/, runner/helpers/hooks/, runner/.primus.yaml).
  • References repointed at the code that owns the behaviour instead of at the launcher script: the HipBLASLt tuning master switch and its stage variable at runner/helpers/hooks/train/pretrain/prepare_experiment.sh, deterministic mode at runner/helpers/hooks/05_deterministic.sh and runner/helpers/envs/base_env.sh, and ANP / NCCL_NET_PLUGIN selection at runner/helpers/hooks/03_enable_ainic.sh. The HipBLASLt section previously omitted that the stage is ignored unless PRIMUS_HIPBLASLT_TUNING=1; that is now stated where users hit it.
  • New guard test tests/runner/test_docs_cli_examples.py fails when a Markdown command invokes a .sh that is not in the tree, so the next launcher move cannot leave stale instructions behind. It matches only bash / sh / source invocations, so prose that merely names a script stays out of scope. A second assertion pins the hybrid-models guide's --volume / --env DATA_PATH placement, which is the shape that is easy to get wrong when translating a snippet.

Scope boundary

Documentation whose statements only become true once the launchers are actually gone is deliberately not in this PR — it lives in #999 alongside the code that makes it true:

The rule applied throughout: this branch must be self-consistent on its own. Nothing here references a file that does not exist yet, and nothing contradicts the tree as it stands.

Test plan

  • pre-commit run --from-ref origin/main --to-ref HEAD — clean.
  • tests/runner/run_all_tests.sh — 10/10 suites.
  • pytest tests/runner/ — 2 passed, including test_docs_cli_examples.py, which confirms every documented script invocation resolves against this branch's tree (the old examples/run_*.sh scripts are still present here and still referenced where they have not been rewritten).
  • Stacking verified: this branch plus refactor(runner): retire the examples launchers in favour of primus-cli #999 reproduces the pre-split tree byte for byte (identical tree hash), so splitting lost nothing.

@WangLingxun
WangLingxun force-pushed the refactor/primus-cli-launchers branch 10 times, most recently from 5d583ef to 3016b30 Compare August 14, 2026 09:14
@WangLingxun
WangLingxun marked this pull request as ready for review August 14, 2026 10:24
@WangLingxun
WangLingxun force-pushed the refactor/primus-cli-launchers branch from 3016b30 to 25ade5c Compare August 17, 2026 03:07
Every guide, config header and example still told users to export EXP and call
examples/run_*.sh, even though `primus-cli` has been the supported entry point
for a while. The instructions now use the runner modes and reference the hooks
that own the behavior.

- Rewrite launch snippets to direct/container/slurm forms and keep container
  environment options at the container layer, with a smoke test preventing the
  `-- --env DATA_PATH` parsing regression.
- Document the HipBLASLt tuning master switch: the stage variable is only
  honored when `PRIMUS_HIPBLASLT_TUNING=1` and deterministic mode is off, which
  the prepare_experiment.sh hook enforces.
- Point environment, determinism and ANP/NCCL_NET_PLUGIN references at the
  hooks and env files that actually set them instead of at the launcher script.
- Add a docs test that fails when a Markdown command invokes a script the tree
  does not contain. The test matches only bash/sh/source invocations, so prose
  that merely names a script stays out of scope.

The launcher scripts themselves are not removed here; retiring them, and the
documentation that only becomes true once they are gone, follows in a stacked
PR on top of this one.
@WangLingxun
WangLingxun force-pushed the refactor/primus-cli-launchers branch from 25ade5c to 3196102 Compare August 20, 2026 07:25
@WangLingxun WangLingxun changed the title Refactor training launchers around primus-cli docs: describe training launches in terms of primus-cli Aug 20, 2026
The repository root ships a `primus-cli` wrapper that execs
`runner/primus-cli`, and the top-level README already spells launches as
`./primus-cli`. Point the migrated examples at that entry point too, so the
documentation does not carry two spellings of the same command.

The directory tree in the diffusion structure guide keeps `runner/primus-cli`:
it records where the implementation lives rather than how to invoke it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant