[rollout] feat: trainer-side FP8 weight quantization for colocated and disaggregated modes

[yxs]

What does this PR do?

Move FP8 blockwise weight quantization from rollout GPU to trainer GPU in the weight sync path, controlled by a new config flag trainer_quantize_fp8. This halves transfer bandwidth in disaggregated mode (1 byte vs 2 bytes per param) and reduces peak memory during bucketed transfer in colocated mode.

Supports both colocated (async quantization via quant_weights_by_name) and disaggregated (sync quantization via quant_weights_by_name_sync for NCCL/NIXL/HCCL checkpoint engines) modes.

Related: #5836 (Q2 roadmap → Weight refit optimization → "fp8 rollout: quantize weights on trainer side")

Checklist Before Starting

• Search for similar PRs. https://github.com/verl-project/verl/pulls?q=fp8+trainer+quantize
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, veomni, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data, cfg, reward, fully_async, one_step_off
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

Environment: 8×H100, Qwen2.5-7B, real GSM8K (7473 samples), SGLang rollout, FSDP trainer

1. Disaggregated mode (NCCL checkpoint engine, 4 trainer + 4 rollout GPUs)

Trainer-side FP8 quantization + NCCL weight transfer verified:

• Log: FP8 trainer-side quantization enabled (disaggregated)
• Log: Skipping FP8 quantization, weights pre-quantized on trainer side
• Log: Rank 0 send weights done, time cost: 0.42s
• Exit code: 0

2. Convergence comparison (127 global steps, disaggregated mode)

Step	Trainer-side FP8 score	Rollout-side FP8 score
3	0.15	0.20
15	0.26	0.29
31	0.35	0.40
47	0.46	0.31
63	0.49	0.45
79	0.59	0.54
95	0.65	0.63
111	0.73	0.72
127	0.73	0.75

Both converge to ~0.73-0.75. No convergence regression.

3. MoE model verification (Qwen3-30B-A3B, 128 experts)

• FP8 weight sync succeeded (send weights done, time cost: 6.42s)
• MoE router (mlp.gate.weight) correctly skipped by quantization (unit test: 25/25 PASS)
• Full MoE training requires more GPUs (30B model OOMs on 4 trainer GPUs), but weight sync path fully verified

4. Selective quantization verification

Parameter type	Action
q/k/v/o_proj	FP8 quantize
gate/up/down_proj (dense + MoE expert)	FP8 quantize
MoE router (mlp.gate.weight)	Skip
embed_tokens	Skip
layernorm / norm	Skip
lm_head	Skip
bias	Skip

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always

[gemini-code-assist]

Code Review

This pull request implements trainer-side FP8 quantization for model weights in engine_workers.py and fsdp_workers.py before they are transmitted to the rollout worker. The sglang_rollout logic is updated to recognize pre-quantized weights and skip redundant quantization steps. Feedback highlights issues with hardcoded quantization parameters and data types, as well as opportunities to optimize the code by reusing quantizer instances and avoiding redundant imports.

[wuxibin89]

Support trainer side fp8 quantization in disaggregate mode is higher priority than colocate mode. cc @sophiayyya #5972

[wuxibin89]

A few comments:

• How do we support vllm? Since it quantize weight in GPU worker: https://github.com/verl-project/verl/blob/main/verl/workers/rollout/vllm_rollout/utils.py#L257-L261
• Need a strict unit test to verify correctness, e.g https://github.com/verl-project/verl/blob/main/tests/checkpoint_engine/test_correctness_on_gpu.py

[wuxibin89]

We can use ensure_async_iterator in Checkpoint engines to async for iterate weights, and eliminate this sync version.