Skip to content

Commit 0f58f47

Browse files
mamazi0131mamazi0131
authored
Merge branch 'main' into main
2 parents ea5f40e + e0f4dc2 commit 0f58f47

265 files changed

Lines changed: 29102 additions & 6932 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 86 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# verl-recipe
22

3-
`verl-recipe` hosts recipes based on [verl](https://github.com/volcengine/verl) contributed by the community.
3+
`verl-recipe` hosts recipes based on [verl](https://github.com/verl-project/verl) contributed by the community.
44

55
## Usage
66

@@ -12,25 +12,107 @@ cd verl
1212
git submodule update --init --recursive recipe
1313
```
1414

15-
## Available Recipes
15+
## Required `verl` version per recipe
16+
17+
Every recipe directory ships a small **`REQUIRED_VERL.txt`** next to its `README.md` (same filename everywhere). That file is the canonical place for:
18+
19+
- upstream git URL (today [`verl-project/verl`](https://github.com/verl-project/verl), historically `volcengine/verl`),
20+
- whether the recipe **tracks `main`** (`pip install -e .` from the same tree) or **pins** a git commit / release tag,
21+
- a copy-pastable `pip install …` line when a pin exists.
22+
23+
**Rolling recipes** (`MODE=rolling` and similar): `REQUIRED_VERL.txt` now lists **exact commit IDs** taken from this workspace when the file was last refreshed:
24+
25+
- **`VERL_COMMIT`**: `git rev-parse HEAD` in the parent **verl** repository (core `pip install verl@git+…@VERL_COMMIT`).
26+
- **`RECIPE_SUBMODULE_COMMIT`**: `git rev-parse HEAD` inside the **`recipe/`** submodule checkout embedded at that verl revision.
27+
- **`RECIPE_FOLDER_LAST_COMMIT`**: `git log -1 --format=%H -- <folder>` inside **`recipe/`** for that recipe’s subdirectory only.
28+
29+
Together these pin both the library and the bundled recipe tree. **`REFRESH=`** at the bottom of each file is the command line used to recompute the fields after you bump verl or the submodule.
30+
31+
Each recipe `README.md` links to its `REQUIRED_VERL.txt` in a short **Required `verl` version** section. The repository root [`README.md`](../README.md) also points here for discoverability.
32+
33+
### One-shot installer: [`install_verl.sh`](install_verl.sh)
34+
35+
`install_verl.sh` reads a recipe's `REQUIRED_VERL.txt` and installs the pinned core `verl` for you. It understands every `MODE=` value listed above and prints the command it will run before executing it (use `--show` for a dry-run).
36+
37+
```bash
38+
# From the root of this repo (the directory containing install_verl.sh):
39+
40+
# List every recipe + its pinned `pip install` line
41+
./install_verl.sh --list
42+
43+
# Dry-run: see exactly which pip command a recipe would execute
44+
./install_verl.sh --recipe dapo --show
45+
46+
# Install the pinned verl core for a recipe via pip (default)
47+
./install_verl.sh --recipe retool
48+
49+
# Clone upstream verl at the pinned commit, init the recipe submodule,
50+
# and `pip install -e .` from the checkout (useful when you want an
51+
# editable install or plan to modify verl itself).
52+
./install_verl.sh --recipe langgraph_agent --method git --dest ./verl
53+
54+
# Recipes that expose multiple variants: pick one with --option
55+
./install_verl.sh --recipe dapo --option reproduction # DAPO paper SHA
56+
./install_verl.sh --recipe flowrl --option A # FlowRL v0.4.0 tag
57+
./install_verl.sh --recipe spin --option baseline # SPIN v0.3.0.post1
58+
```
59+
60+
Flags: `--recipe NAME` (e.g. `gkd/megatron`, `specRL/histoSpec`) or `--file PATH/TO/REQUIRED_VERL.txt`, `--method pip|git` (default `pip`), `--dest DIR` (default `./verl`, only used with `--method git`), `--show` (dry-run), `--yes` (skip confirmation), `--list`, `--help`.
61+
62+
The script requires only `bash`, `git`, `awk`, and `pip`/`pip3` on `PATH`. It does **not** `source` or `eval` the `REQUIRED_VERL.txt` file — values are extracted with `awk` and the exact command to be executed is echoed before it runs.
63+
64+
| Recipe | `REQUIRED_VERL.txt` |
65+
| --- | --- |
66+
| char_count | [`recipe/char_count/REQUIRED_VERL.txt`](char_count/REQUIRED_VERL.txt) |
67+
| collabllm | [`recipe/collabllm/REQUIRED_VERL.txt`](collabllm/REQUIRED_VERL.txt) |
68+
| dapo | [`recipe/dapo/REQUIRED_VERL.txt`](dapo/REQUIRED_VERL.txt) |
69+
| deepeyes | [`recipe/deepeyes/REQUIRED_VERL.txt`](deepeyes/REQUIRED_VERL.txt) |
70+
| entropy | [`recipe/entropy/REQUIRED_VERL.txt`](entropy/REQUIRED_VERL.txt) |
71+
| fapo | [`recipe/fapo/REQUIRED_VERL.txt`](fapo/REQUIRED_VERL.txt) |
72+
| fault_recover | [`recipe/fault_recover/REQUIRED_VERL.txt`](fault_recover/REQUIRED_VERL.txt) |
73+
| flash_rl_ascend | [`recipe/flash_rl_ascend/REQUIRED_VERL.txt`](flash_rl_ascend/REQUIRED_VERL.txt) |
74+
| flowrl | [`recipe/flowrl/REQUIRED_VERL.txt`](flowrl/REQUIRED_VERL.txt) |
75+
| genrm_remote | [`recipe/genrm_remote/REQUIRED_VERL.txt`](genrm_remote/REQUIRED_VERL.txt) |
76+
| gkd/megatron | [`recipe/gkd/megatron/REQUIRED_VERL.txt`](gkd/megatron/REQUIRED_VERL.txt) |
77+
| gvpo | [`recipe/gvpo/REQUIRED_VERL.txt`](gvpo/REQUIRED_VERL.txt) |
78+
| infigui-g1 | [`recipe/infigui-g1/REQUIRED_VERL.txt`](infigui-g1/REQUIRED_VERL.txt) |
79+
| langgraph_agent | [`recipe/langgraph_agent/REQUIRED_VERL.txt`](langgraph_agent/REQUIRED_VERL.txt) |
80+
| minicpmo | [`recipe/minicpmo/REQUIRED_VERL.txt`](minicpmo/REQUIRED_VERL.txt) |
81+
| nemo_gym | [`recipe/nemo_gym/REQUIRED_VERL.txt`](nemo_gym/REQUIRED_VERL.txt) |
82+
| open_math_reasoning | [`recipe/open_math_reasoning/REQUIRED_VERL.txt`](open_math_reasoning/REQUIRED_VERL.txt) |
83+
| partial_rollout | [`recipe/partial_rollout/REQUIRED_VERL.txt`](partial_rollout/REQUIRED_VERL.txt) |
84+
| prime | [`recipe/prime/REQUIRED_VERL.txt`](prime/REQUIRED_VERL.txt) |
85+
| qat | [`recipe/qat/REQUIRED_VERL.txt`](qat/REQUIRED_VERL.txt) |
86+
| r1 | [`recipe/r1/REQUIRED_VERL.txt`](r1/REQUIRED_VERL.txt) |
87+
| r1_ascend | [`recipe/r1_ascend/REQUIRED_VERL.txt`](r1_ascend/REQUIRED_VERL.txt) |
88+
| rep_exp | [`recipe/rep_exp/REQUIRED_VERL.txt`](rep_exp/REQUIRED_VERL.txt) |
89+
| retool | [`recipe/retool/REQUIRED_VERL.txt`](retool/REQUIRED_VERL.txt) |
90+
| specRL/histoSpec | [`recipe/specRL/histoSpec/REQUIRED_VERL.txt`](specRL/histoSpec/REQUIRED_VERL.txt) |
91+
| spin | [`recipe/spin/REQUIRED_VERL.txt`](spin/REQUIRED_VERL.txt) |
92+
| spo | [`recipe/spo/REQUIRED_VERL.txt`](spo/REQUIRED_VERL.txt) |
93+
| sppo | [`recipe/sppo/REQUIRED_VERL.txt`](sppo/REQUIRED_VERL.txt) |
94+
| swe_agent | [`recipe/swe_agent/REQUIRED_VERL.txt`](swe_agent/REQUIRED_VERL.txt) |
95+
96+
## Available Recipes (high level)
1697

1798
- [retool](https://github.com/verl-project/verl-recipe/tree/main/retool): Reinforcement Learning for Strategic Tool Use in LLMs
1899
- [langgraph_agent](https://github.com/verl-project/verl-recipe/tree/main/langgraph_agent): A tiny example to demonstrate multi-turn rollout with [LangGraph ReactAgent](https://langchain-ai.github.io/langgraph/agents/overview/) to solve math expression.
19100
- [spo](https://github.com/verl-project/verl-recipe/tree/main/spo): [Single-stream Policy Optimization](https://arxiv.org/abs/2509.13232).
101+
- [partial_rollout](./partial_rollout/): synchronous RL with cross-step rollout interruption + resume to reclaim long-tail GPU bubbles ([APRIL](https://arxiv.org/pdf/2509.18521)-style).
20102
- TBA...
21103

22104
## Contribution
23105

24106
### Version Specification
25107

26-
Recipes are recommended to specify the verl version required, e.g.,
108+
Add or update **`REQUIRED_VERL.txt`** whenever a recipe gains a new tested pin or intentionally moves forward on `main`. Examples of valid `pip` forms:
27109

28110
```
29111
# release version
30112
verl==0.6.0
31113
32114
# dev version
33-
verl@git+https://github.com/volcengine/verl.git@313dfdb2199124a37189e32e6d4a6c654379f2d4
115+
verl@git+https://github.com/verl-project/verl.git@313dfdb2199124a37189e32e6d4a6c654379f2d4
34116
```
35117

36118
### Code Linting and Formatting

‎async_flow/README.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# AsyncFlow
2+
3+
AsyncFlow 是基于 verl 实现的 **训推异步流水线** 方案。通过 `TransferQueue (TQ)` 与 `CheckpointEngine (CE)` 将推理（Rollout）与训练（Trainer）解耦，配合 Staleness 机制与权重异步同步，可显著提升 RLHF/GRPO 训练吞吐。
4+
5+
- 详细架构与组件说明：[`docs/architecture.md`](./docs/architecture.md)
6+
- 完整性能配置与精度数据：[`docs/performance.md`](./docs/performance.md)
7+
8+
## 1. 整体架构
9+
10+
![整体架构](./docs/images/4cf702c3-aa06-4246-bacb-d99fb9d866ad.png)
11+
12+
- **AgentLoop Layer**：异步生成层（`AsyncFlowAgentLoopManager` + `AgentLoopWorker`），支持 `model_version` 追踪。
13+
- **TransferQueue**：异步数据流中间件，解耦 Rollout 与下游 Workers。
14+
- **四个独立 Workers**：`ActorForwardWorker`（old_log_probs / entropys）、`ReferenceForwardWorker`（ref_logprobs）、`RewardAdvWorker`（reward / advantage）、`ActorTrainWorker`（PPO loss + 参数更新）。
15+
- **CheckpointEngine**：基于 HCCL 的异步权重广播。
16+
- **FlowControlQueue**：Staleness 控制队列，限制 inflight 请求数。
17+
18+
## 2. 性能收益
19+
20+
> 详细配置见 [`docs/performance.md`](./docs/performance.md)。
21+
22+
| 性能指标 | Verl On-Policy | Async Flow | Fully Async |
23+
| --- | --- | --- | --- |
24+
| `prompt:2k → response:16k` 角色 | All | Rollout-Ref-Fwd-Trainer | Rollout-Train |
25+
| Cluster NPU 切分 | 64 | 50-2-2-8 | 48-16 |
26+
| `perf/throughput` | 59.3 | **226.8** | 149.66 |
27+
| 相对提升 | / | **3.81×** | 2.52× |
28+
29+
精度方面，异步 RL 在 `staleness ≤ 2` 时可与同步 RL 对齐甚至微幅提升（详见性能文档）。
30+
31+
## 3. Quick Start
32+
33+
### 3.1 环境
34+
35+
- 参见 [`REQUIRED_VERL.txt`](./REQUIRED_VERL.txt) 与 [`requirements.txt`](./requirements.txt)
36+
- 推荐 Ascend NPU 环境（HCCL CheckpointEngine）
37+
38+
### 3.2 单机示例（Qwen2.5-0.5B + GSM8K）
39+
40+
```bash
41+
bash recipe/async_flow/run_async_grpo_qwen0.5b_gsm8k_local.sh
42+
```
43+
44+
主要环境变量：
45+
46+
```bash
47+
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
48+
export VLLM_USE_V1=1
49+
export VERL_CLUSTER_TRACE=1
50+
export PROMETHEUS_METRICS_ENABLE=true
51+
export PROMETHEUS_METRICS_PORT=9400
52+
export PROMETHEUS_MULTIPROC_DIR=/tmp/prom_metrics
53+
```
54+
55+
入口模块：`recipe.async_flow.grpo_main`，复用 verl Hydra 配置体系，通过 CLI override 覆盖训练参数（`data.*`、`actor_rollout_ref.*`、`async_resources.*` 等）。
56+
57+
### 3.3 7B 示例
58+
59+
```bash
60+
bash recipe/async_flow/run_async_grpo_qwen7b_gsm8k_local.sh
61+
```
62+
63+
## 4. 目录结构
64+
65+
```
66+
recipe/async_flow/
67+
├── agent_loop/ # AsyncFlowAgentLoopManager / AgentLoopWorker
68+
├── config/ # Hydra 配置
69+
├── docs/ # 详细架构、性能文档
70+
├── tests/ # 单元 / 集成测试
71+
├── utils/ # TransferQueue、cluster_trace、metric 等工具
72+
├── vllm_rollout/ # vLLM Rollout 接入
73+
├── workers/ # 4 个独立 Worker 实现
74+
├── async_flow_trainer.py # 异步 Trainer 主控
75+
├── config.py # 配置 dataclass
76+
├── grpo_main.py # 训练入口
77+
└── run_async_grpo_*.sh # 启动脚本
78+
```

‎async_flow/REQUIRED_VERL.txt‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# async_flow — pinned to verl release tag v0.7.1
2+
UPSTREAM=https://github.com/verl-project/verl.git
3+
MODE=pinned_tag
4+
TAG=v0.7.1
5+
COMMIT=bec9ef74768dd201881cd4e54cd0385e87caae27
6+
PIP_INSTALL=pip install verl==0.7.1
7+
NOTES=async_flow recipe is developed and tested against verl v0.7.1.

‎async_flow/agent_loop/__init__.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#
2+
# Copyright (c) 2026 Huawei Technologies Co., Ltd. All Rights Reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.#
15+
from recipe.async_flow.agent_loop.agent_loop import AsyncFlowAgentLoopManager
16+
17+
__all__ = ["AsyncFlowAgentLoopManager"]

0 commit comments

Comments
 (0)