[fully_async] feat: standalone log prob server (Model Engine Server) support

[sl-1314]

What does this PR do?

This PR introduces a standalone Model Engine Server for the fully_async training pipeline to compute log_probs. In the existing design, old_log_probs are recomputed by the actor training engine, which requires saving/restoring actor weights. This PR decouples that computation into a dedicated inference server(allocate additional GPUs for log_probs computation) that runs concurrently with rollout generation.

Currently, only support megatron backend.

The implementation follows the existing RolloutReplica / BaseRollout / CheckpointEngineWorker architecture:

Class	Role
ModelEngineReplica	RolloutReplica subclass — resource allocation, lifecycle, weight sync
ModelEngineWorker	Ray Actor per GPU — CheckpointEngineWorker subclass, receives weights
ModelEngineServerAdapter	BaseRollout adapter — wraps TrainingWorker for forward-only inference
ModelEngineServer	Ray Actor — async batch scheduler with pause_serving/resume_serving protocol

To use Model Engine Server, you need to use mbridge and apply this PR: ISEEKYAN/mbridge#117

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: ...
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, veomni, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data, cfg, reward, fully_async, one_step_off
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

Using Model Engine Server, will slightly increase timing_s/gen, but it can effectively eliminate timing_s/old_log_prob and reduce end-to-end time.

Compared to the original old_log_prob calculating method(16 gpus training and 16 gpus rollout), Model Engine Serve(8 gpus addition), resulting in an end-to-end time speedup of approximately 1.64x. Considering the increased resource consumption, the speedup ratio is about 1.09x.

API and Usage Example

Enable with:

model_engine_server.enable_standalone: True
model_engine_server.nnodes: nnodes
model_engine_server.n_gpus_per_node: n_gpus_per_node
# independent model_parallel config
model_engine_server.megatron.pipeline_model_parallel_size= pp_size
model_engine_server.megatron.tensor_model_parallel_size=tp_size
......

example script:

verl/experimental/fully_async_policy/shell/dapo_7b_math_megatron_16_16_8.sh

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)
• If your PR is related to the recipe submodule, please also update the reference to the submodule commit via git submodule update --remote or cd recipe && git pull origin main.

[gemini-code-assist]

Code Review

This pull request introduces a standalone ModelEngineServer to compute 'old' log probabilities for fully asynchronous training. Key changes include the implementation of ModelEngineReplica and ModelEngineWorker, updates to the agent loop to handle response_oldlogprobs, and enhancements to Megatron utilities for weight synchronization via async generators. A critical bug was identified in the tool_agent_loop.py where old log probabilities were being appended to the wrong data list, which would lead to incorrect training data.

[ArronHZG]

I feel it also needs to support multiple replicas and have a server manager for data distribution.

Furthermore, modifications to the main branch need to be reduced, converging to fully async.

[ArronHZG]

In the experiment, even without rejection sampling correction, the rollout mismatch metric needs to be enabled to observe the results.

Additionally, the current implementation can display the Entropy metric.