Agentic Policy Optimization via Instruction-Policy Co-Evolution

If you find this project interesting, please give us a star ⭐ on GitHub to support us. 🙏🙏

💡 Overview

INSPO is a framework that integrates instruction optimization as a dynamic component of the reinforcement learning (RL) loop for training large language models. Unlike conventional approaches that rely on static, manually-crafted instructions, INSPO enables instructions to co-evolve with the agent's policy during training.

• Instruction-Policy Co-Evolution: Instructions and the agent policy improve each other throughout training, rather than keeping instructions fixed.
• Dynamic Instruction Population: A pool of instruction candidates is maintained and automatically evaluated using RL reward signals. Underperforming candidates are periodically pruned.
• On-Policy Reflection: An LLM-based optimizer analyzes the agent's historical experience from the replay buffer to generate and validate improved instructions.

✨ Method

We propose a two-phase instruction-policy co-evolution framework:

• Phase 1: Population Initialization — An LLM-based optimizer generates an initial population of diverse instruction candidates from a seed instruction.
• Phase 2: Co-Evolution — During RL training, instructions are dynamically sampled via importance-weighted selection, periodically pruned via successive halving, and evolved through experience-driven reflection conditioned on the agent's failure trajectories.

🛠️ Setup

1. Install Search-R1

git clone https://github.com/PeterGriffinJin/Search-R1.git
cd Search-R1
conda create -n searchr1 python=3.9
conda activate searchr1
pip install -e .
pip install vllm==0.6.3
pip install flash-attn --no-build-isolation
pip install wandb && wandb login

2. Install INSPO

cd ..
git clone https://github.com/cambridgeltl/inspo.git
cd inspo
pip install -e .

3. Retriever environment

conda create -n retriever python=3.10.16
conda activate retriever
conda install -y pytorch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 pytorch-cuda=12.1 -c pytorch -c nvidia
conda install -y -c pytorch -c nvidia faiss-gpu=1.8.0
pip install transformers==4.51.1 datasets==3.5.0 pyserini==0.44.0 uvicorn==0.34.0 fastapi==0.115.12

4. Download data and index

conda activate searchr1
cd Search-R1
bash download.sh
bash scripts/nq_hotpotqa/data_process.sh

🚀 Quick Start

All training commands are run from Search-R1/. INSPO imports Search-R1 at runtime — no files are copied or patched.

Step 1: Launch the retrieval server (in a separate terminal):

conda activate retriever
cd Search-R1
bash ../inspo/scripts/launch_retriever.sh

Step 2: Run INSPO training:

conda activate searchr1
cd Search-R1
bash ../inspo/scripts/train_inspo.sh

📊 Experiments

Script	Description
scripts/train_inspo.sh	Main INSPO training (reflection optimizer)
scripts/train_baseline.sh	Baseline GRPO with a static instruction
scripts/launch_retriever.sh	Launch the local dense retrieval server

📑 Citation

If you find INSPO useful for your research and applications, please cite using this BibTeX:

@article{zhou2025agentic,
  title={Agentic Policy Optimization via Instruction-Policy Co-Evolution},
  author={Zhou, Han and Wan, Xingchen and Vuli{\'c}, Ivan and Korhonen, Anna},
  journal={arXiv preprint arXiv:2512.01945},
  year={2025}
}

🙏 Acknowledgements

INSPO is built on top of Search-R1 and veRL, and is evaluated on multi-turn retrieval-augmented QA tasks (NQ, HotpotQA, TriviaQA, PopQA, etc.).

• Search-R1 — RL framework for training reasoning-and-searching interleaved LLMs
• veRL — Volcano Engine Reinforcement Learning for LLMs
• vLLM — Fast LLM serving