PPO-DAP: Combining Diffusion Models with PPO to Improve Sample Efficiency and Exploration in Reinforcement Learning

Overview

PPO-DAP is a reinforcement learning framework that integrates diffusion models with Proximal Policy Optimization (PPO) to enhance sample efficiency and exploration capabilities. This project, implemented using the robomimic framework, utilizes the D4RL dataset for experiments, demonstrating improved performance in environments with limited data.

Training Artifacts

All training datasets, pretrained models, training logs, and videos can be accessed through the following Google Drive link:

Google Drive - PPO-DAP Training Artifacts

Project Structure

├── datasets/                   # Directory for storing datasets
├── models/                     # Pretrained models
├── scripts/                    # Scripts for training, evaluation, and visualization
│   ├── train.py                # Script for training the model
│   ├── evaluate.py             # Script for evaluating the model
│   └── visualize_results.py    # Script for visualizing results
├── notebooks/                  # Jupyter Notebooks for analysis and visualization
├── configs/                    # Configuration files
│   └── PPO.json                # Configuration for the PPO algorithm
├── README.md                   # Project documentation
└── requirements.txt            # Python dependencies

Getting Started

Prerequisites

To get started with PPO-DAP, ensure that you have the following software installed:

• Python 3.8
• Conda (optional, but recommended for managing environments)

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/PPO-DAP.git
   cd PPO-DAP

2. Create and activate a Python virtual environment:

   conda create -n ppo-dap_env python=3.8
   conda activate diffppo_env

3. Install the required dependencies:

   pip install -r requirements.txt

Dataset

The project utilizes the D4RL dataset. You can download the dataset using the provided script:

bash scripts/download_dataset.sh

Alternatively, you can refer to the D4RL documentation for more details.

Usage

Training

To train the model, use the following command:

python scripts/train.py --config configs/PPO.json

Evaluation

After training, evaluate the model's performance using:

python scripts/evaluate.py --model-path models/my_trained_model.pth

Visualization

Visualize the training results with:

python scripts/visualize_results.py --log-dir logs/

Results

The experiments conducted in this project demonstrate that integrating diffusion models to generate synthetic trajectories significantly enhances the sample efficiency and exploration capabilities of the PPO algorithm. Below is an example of the cumulative rewards achieved across different tasks:

Contribution

We welcome contributions to PPO-DAP. If you would like to contribute, please follow these steps:

1. Fork the repository.
2. Create a new branch (git checkout -b new-feature).
3. Commit your changes (git commit -am 'Add new feature').
4. Push to the branch (git push origin new-feature).
5. Create a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.