# Proximal Policy Optimization (PPO)

> A stable, sample-efficient policy gradient algorithm for reinforcement learning

- Type: Concept
- By: Schulman et al.
- Year: 2017
- Topics: Deep learning, Reinforcement learning, Policy gradients, Optimization
- Page: https://aipedia.org/ppo

## Resources

- [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347): Paper, Schulman et al., 2017
- [openai/baselines](https://github.com/openai/baselines): Code, Reference implementations
- [Proximal Policy Optimization](https://spinningup.openai.com/en/latest/algorithms/ppo.html): Docs, OpenAI Spinning Up
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://arxiv.org/abs/1506.02438): Paper, Schulman et al., 2015
- [Trust Region Policy Optimization](https://arxiv.org/abs/1502.05477): Paper, Schulman et al., 2015

**Proximal Policy Optimization (PPO)** is the most widely used deep reinforcement learning algorithm. Its core idea is simple: improve the policy a little at a time, and clip any update that would change it too much. That one trick makes it stronger, simpler, and more stable than prior methods like TRPO.

> This page assumes you know what a policy gradient is. If not, read [Policy Gradient](https://aipedia.org/policy-gradient) first, and [Reinforcement Learning](https://aipedia.org/reinforcement-learning) before that.

## Why Students Should Care

- PPO is the default algorithm in most RL libraries and papers — if a project "uses RL", it probably uses PPO.
- It is the classic optimizer behind [RLHF](https://aipedia.org/rlhf), the technique used to align chat assistants.
- The clipping idea is a clean example of a broader lesson: in deep learning, *how big your update steps are* matters as much as their direction.

## The Problem, in Plain English

Policy gradient methods learn by nudging the policy toward actions that worked. The trouble is choosing the nudge size:

- Too large an update → the policy can collapse, and because the policy also *collects the data*, bad updates poison future learning
- Too small an update → learning crawls

Earlier fixes (like TRPO) enforced a "trust region" with heavy math. PPO gets a similar effect with one clipped objective you can implement in a few lines.

## The PPO-Clip Objective

$$
L^{CLIP}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]
$$

where:

- $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ is the **probability ratio**: how much more (or less) likely the new policy is to take the action than the old policy was
- $\hat{A}_t$ is the **advantage** estimate: was this action better or worse than average?
- $\epsilon$ is the clip range (typically 0.1-0.2)

You do not need to memorize the formula. The important idea is: **increase the probability of good actions and decrease the probability of bad ones, but never let a single update shift any action's probability by more than a factor of about $1 \pm \epsilon$.**

## How Clipping Works

The clip function pins the ratio to a narrow band around 1:

$$
\text{clip}(r, 1-\epsilon, 1+\epsilon) = \begin{cases}
1-\epsilon & \text{if } r < 1-\epsilon \\
r & \text{if } 1-\epsilon \leq r \leq 1+\epsilon \\
1+\epsilon & \text{if } r > 1+\epsilon
\end{cases}
$$

Taking the minimum of the clipped and unclipped terms makes the constraint one-sided in exactly the right way:

- If $\hat{A} > 0$ (the action was good): don't increase $r$ beyond $1+\epsilon$ — no runaway enthusiasm
- If $\hat{A} < 0$ (the action was bad): don't decrease $r$ below $1-\epsilon$ — no panic abandonment

Once the ratio leaves the band, the gradient through that sample becomes zero, so the optimizer simply stops pushing in that direction.

## Interactive Visualization

See how the clipping mechanism constrains policy updates:

*Interactive demo: https://aipedia.org/ppo*

## Advantage Estimation

PPO needs to know whether each action was better than expected. It typically uses **Generalized Advantage Estimation (GAE)**, which blends short-horizon and long-horizon estimates:

$$
\hat{A}_t = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}
$$

where $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ is the TD error — the one-step "surprise" relative to the value function's prediction.

The $\lambda$ parameter trades off bias vs variance: small $\lambda$ trusts the value function more (lower variance, more bias), large $\lambda$ trusts observed rewards more.

## Full Algorithm

```python
for iteration in range(num_iterations):
    # Collect trajectories with current policy
    trajectories = collect_rollouts(policy, env)
    
    # Compute advantages
    advantages = compute_gae(trajectories, value_fn)
    
    # Multiple epochs of updates
    for epoch in range(num_epochs):
        for batch in trajectories.batches():
            # PPO update
            loss = ppo_clip_loss(batch, advantages)
            optimizer.step(loss)
```

Note the inner loop: PPO reuses each batch of experience for **several epochs** of gradient updates. That is exactly what clipping makes safe — without it, repeated updates on the same data would push the policy too far from the one that collected it.

## Hyperparameters

| Parameter | Typical Value | Effect |
|-----------|---------------|--------|
| $\epsilon$ (clip) | 0.1-0.2 | Update constraint |
| $\gamma$ (discount) | 0.99 | Future reward weighting |
| $\lambda$ (GAE) | 0.95 | Advantage bias-variance |
| Epochs per update | 3-10 | Sample efficiency |
| Batch size | 32-4096 | Gradient stability |

## Why PPO is Popular

1. **Simple**: Easier to implement than TRPO
2. **Stable**: Clipping prevents catastrophic updates
3. **Sample efficient**: Multiple epochs per rollout
4. **General**: Works on continuous and discrete actions
5. **Scalable**: Parallelizes well across workers

## Applications

PPO powers:

- **RLHF**: Aligning language models (ChatGPT, Claude) — see [RLHF](https://aipedia.org/rlhf)
- **Game AI**: OpenAI Five, DOTA 2
- **Robotics**: Manipulation, locomotion
- **Autonomous driving**: Decision making

## Common Confusion

- **PPO vs. policy gradient**: PPO *is* a policy gradient method. The clipped objective is a safer stand-in for the vanilla policy gradient loss — see [Policy Gradient](https://aipedia.org/policy-gradient) for the base method.
- **PPO vs. TRPO**: both limit how far the policy moves per update. TRPO enforces a hard KL-divergence constraint with second-order optimization; PPO approximates the same effect with first-order clipping.
- **Clipping rewards vs. clipping the ratio**: PPO clips the *probability ratio* between new and old policies, not the rewards themselves.
- **PPO vs. [DQN](https://aipedia.org/dqn)**: PPO learns a policy directly (policy-based); DQN learns action values and derives the policy from them (value-based).

## Where To Go Next

- Read [Policy Gradient](https://aipedia.org/policy-gradient) for the foundation PPO is built on, including REINFORCE and actor-critic.
- Read [RLHF](https://aipedia.org/rlhf) to see PPO's most famous application: fine-tuning language models from human preferences.
- Read [DQN](https://aipedia.org/dqn) for the contrasting value-based approach.
- Read [Reinforcement Learning](https://aipedia.org/reinforcement-learning) if you want the full MDP background.