# Reinforcement Learning

> Learning by trial and error through rewards

- Type: Concept
- Topics: Deep learning, Reinforcement learning, Agents, Decision making
- Page: https://aipedia.org/reinforcement-learning

## Resources

- [Reinforcement Learning: An Introduction (2nd ed.)](http://incompleteideas.net/book/the-book-2nd.html): Book, Sutton & Barto, 2018
- [Spinning Up in Deep RL](https://spinningup.openai.com/en/latest/): Docs, OpenAI
- [Lectures on Reinforcement Learning](https://www.davidsilver.uk/teaching/): Course, David Silver, UCL

**Reinforcement Learning (RL)** is about learning by trial and error. An agent takes actions, sees what happens, and gets rewards or penalties. Over time, it learns which actions lead to better long-term outcomes.

> Good follow-up pages: [Policy Gradient](https://aipedia.org/policy-gradient), [DQN](https://aipedia.org/dqn), [PPO](https://aipedia.org/ppo), and [RLHF](https://aipedia.org/rlhf).

## A Simple Example

Imagine training a robot to leave a maze:

- every move costs `-1`
- reaching the exit gives `+100`
- falling into a trap gives `-50`

The robot is not told the correct path in advance. It has to explore, make mistakes, and gradually discover a strategy that earns the highest total reward.

## The RL Loop

At each step:

1. The agent observes the current **state** $s_t$
2. It chooses an **action** $a_t$
3. The environment returns a **reward** $r_t$ and a new state $s_{t+1}$
4. The process repeats

The goal is to learn a **policy** $\pi(a|s)$: a rule for choosing actions that maximizes long-term reward.

## Markov Decision Process (MDP)

RL problems are usually written as a **Markov Decision Process**:

- **States** $\mathcal{S}$: all situations the agent might be in
- **Actions** $\mathcal{A}$: what the agent can do
- **Transition** $P(s'|s,a)$: how the world changes after an action
- **Reward** $R(s,a,s')$: how good or bad that outcome is
- **Discount** $\gamma$: how much future rewards matter

If that notation is new, the main idea is still simple: **state -> action -> consequence -> reward**.

## The Objective

RL tries to maximize the expected discounted return:

$$
J(\pi) = \mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^{\infty} \gamma^t r_t\right]
$$

That expression just means: add up rewards over time, with optional discounting so near-term rewards count more than distant ones.

## Value Functions

Two functions show up everywhere in RL:

- **State value** $V^\pi(s)$: how good it is to be in state $s$
- **Action value** $Q^\pi(s, a)$: how good it is to take action $a$ in state $s$

They help the agent estimate long-term consequences before it has seen every possible future.

$$
V^\pi(s) = \mathbb{E}_\pi\left[\sum_{t=0}^{\infty} \gamma^t r_t \mid s_0 = s\right]
$$

$$
Q^\pi(s, a) = \mathbb{E}_\pi\left[\sum_{t=0}^{\infty} \gamma^t r_t \mid s_0 = s, a_0 = a\right]
$$

## Bellman Equations

The Bellman equations say that the value of a state equals:

- the immediate reward you expect now
- plus the value of where you are likely to end up next

That recursive structure is what makes dynamic programming and many RL algorithms possible.

$$
V^\pi(s) = \sum_a \pi(a|s) \sum_{s'} P(s'|s,a)[R(s,a,s') + \gamma V^\pi(s')]
$$

## Interactive Visualization

Watch an agent learn to navigate through trial and error:

*Interactive demo: https://aipedia.org/reinforcement-learning*

## Two Big Families of Methods

**Value-based methods**
- learn a score for actions, such as $Q(s,a)$
- then pick the action with the highest score
- examples: Q-learning, [DQN](https://aipedia.org/dqn)

**Policy-based methods**
- learn the policy directly
- work well for continuous or stochastic actions
- examples: [Policy Gradient](https://aipedia.org/policy-gradient), [PPO](https://aipedia.org/ppo)

## Key Algorithms

| Algorithm | Type | Main idea |
|-----------|------|-----------|
| Q-Learning | Value | Update action values from experience |
| DQN | Value | Use a neural network to approximate Q-values |
| REINFORCE | Policy | Push up actions that led to high reward |
| PPO | Policy | Improve the policy while limiting unstable updates |
| SAC | Actor-Critic | Add entropy to encourage exploration |

## Exploration vs Exploitation

The hardest practical tradeoff in RL is:

- **Explore**: try uncertain actions that might be better
- **Exploit**: use the best strategy found so far

If you only exploit, you may miss better strategies. If you only explore, you never settle on one.

## Where RL Shows Up

- Game playing
- Robotics
- Recommender systems
- Resource allocation
- Language model alignment methods such as [RLHF](https://aipedia.org/rlhf)

## What To Remember

- RL is learning from reward, not labeled answers
- The objective is long-term reward, not immediate reward
- Most methods either learn values, policies, or both