PPO From Scratch

A clean, dependency-free implementation of Proximal Policy Optimization (PPO) in PyTorch. No Stable-Baselines. No RLlib. Just the algorithm — readable, tested, and reproducible.

Trains a policy from scratch on CartPole-v1 → LunarLander-v2 → LunarLanderContinuous-v2, with full support for both discrete and continuous action spaces.

---

Results

Environment	Mean Reward (last 100 ep)	Timesteps	Solved?
CartPole-v1	500 ± 0	~80k	✅
LunarLander-v2	240 ± 30	~500k	✅
LunarLanderContinuous-v2	260 ± 25	~1M	✅

Averaged over 5 seeds. Solved threshold: CartPole ≥ 475, LunarLander ≥ 200.

---

Method

PPO (Schulman et al., 2017) is an on-policy actor-critic algorithm that stabilizes training by constraining how much the policy can change per update.

Key components implemented:

• Clipped surrogate objective — prevents destructively large policy updates
• Generalized Advantage Estimation (GAE-λ) — lower-variance advantage estimates
• Entropy bonus — encourages exploration by penalizing overly deterministic policies
• Orthogonal weight initialization — empirically improves training stability
• Learning rate annealing — linearly decays lr over training
• Gradient norm clipping — prevents exploding gradients

Architecture

Observation → [Linear(obs, 64) → Tanh → Linear(64, 64) → Tanh]
                    │                                │
             Actor head                       Critic head
          (logits / mean+std)                 (scalar V(s))

Shared trunk with separate actor and critic heads. Discrete environments use a Categorical distribution; continuous environments use a diagonal Normal distribution.

---

Quickstart

git clone https://github.com/YOUR_USERNAME/ppo-from-scratch
cd ppo-from-scratch
uv sync

Requires uv. Install it with curl -LsSf https://astral.sh/uv/install.sh | sh.

Train on CartPole (fast — ~2 min):

uv run python train.py --env CartPole-v1 --timesteps 100000

Train on LunarLander (~15 min on CPU):

uv run python train.py --env LunarLander-v2 --timesteps 500000

Train continuous control:

uv run python train.py --env LunarLanderContinuous-v2 --continuous --timesteps 1000000

Evaluate and record a GIF:

uv run python evaluate.py \
    --checkpoint checkpoints/LunarLander-v2_seed42.pt \
    --record \
    --gif-path assets/agent.gif

Run tests:

uv run pytest test_ppo.py -v

---

Project Structure

ppo-from-scratch/
├── ppo/
│   ├── __init__.py
│   ├── agent.py        # ActorCritic network (discrete + continuous)
│   ├── buffer.py       # RolloutBuffer with GAE computation
│   └── trainer.py      # PPOTrainer — the update loop
├── train.py            # Training entrypoint (CLI)
├── evaluate.py         # Evaluation + GIF recording
├── test_ppo.py         # Unit tests for agent + buffer
├── ci.yml              # GitHub Actions CI
├── pyproject.toml
├── uv.lock
└── README.md

---

Hyperparameters

CLI flag	Default	Description
--env	LunarLander-v2	Gymnasium environment ID
--timesteps	500000	Total environment steps
--seed	42	Random seed
--lr	3e-4	Adam learning rate (annealed)
--hidden	64	Hidden layer width
--n-steps	2048	Steps collected per rollout
--batch-size	64	Mini-batch size for updates
--clip-eps	0.2	PPO clipping epsilon
--ent-coef	0.01	Entropy bonus coefficient
--continuous	False	Use continuous action space
--save-dir	checkpoints	Directory to save .pt checkpoints
--run-name	env_seedN	Custom name for checkpoint file

See all options: uv run python train.py --help

---

References

• Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347
• Schulman, J. et al. (2016). High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016
• Huang, S. et al. (2022). The 37 Implementation Details of Proximal Policy Optimization

---

License

MIT