Adam Optimizer

Adam (Adaptive Moment Estimation), introduced by Kingma and Ba in 2014, is the most widely used optimizer in deep learning. The core idea is simple: instead of using one fixed step size for every parameter, Adam watches how each parameter’s gradient behaves over time and gives each parameter its own momentum-smoothed, adaptively-scaled update.

If gradients and training loops are new to you, read Backpropagation first. Backprop computes the gradients; an optimizer like Adam decides how to use them to update the weights.

Why Students Should Care

• Adam (or its variant AdamW) is the default optimizer in nearly every modern deep learning project, including Transformers and large language models.
• It combines the benefits of momentum and adaptive learning rates, requiring minimal tuning while working well across diverse problems.
• Understanding Adam explains a lot of practical training advice you will encounter: warmup schedules, weight decay choices, and optimizer memory costs.

The Problem with Vanilla SGD

Plain Stochastic Gradient Descent takes the same size step for every parameter, scaled only by the raw gradient. That causes trouble:

• Same learning rate for all parameters (bad for sparse features, where some parameters get gradients rarely)
• Struggles with saddle points and ravines, where the loss surface is steep in one direction and flat in another
• Requires careful learning rate scheduling by hand

Adam’s Solution

Think of Adam as answering two questions for each parameter, using running averages of its gradient history:

• Which direction have gradients been pointing lately? (momentum)
• How large have gradients been lately? (adaptive scaling)

Concretely, Adam maintains two moving averages for each parameter:

1. First moment (momentum): $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t$
2. Second moment (adaptive LR): $v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2$

Then it updates the weights using:

$$\theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$$

You do not need to memorize the equation. The important idea is: step in the smoothed gradient direction ($\hat{m}_t$), but shrink the step for parameters whose gradients have been large ($\sqrt{\hat{v}_t}$ in the denominator).

Bias Correction

There is one subtlety. The moving averages start at zero, so early in training $m_t$ and $v_t$ are biased toward zero — they underestimate the true averages. Adam corrects this:

$$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}$$

This correction is crucial for proper early training. Without it, the first steps would be far too small (or, after dividing, badly mis-scaled).

Interactive Demo

Compare Adam with other optimizers on a loss landscape:

Optimizer Comparison

Step: 0/100RunReset

SGD

Momentum

Adam

SGD

2.5250

Momentum

2.5250

Adam

2.5250

Adam (Adaptive Moment Estimation)

m = β₁m + (1-β₁)∇L
v = β₂v + (1-β₂)∇L²
θ = θ - α · m̂ / (√v̂ + ε)

The Complete Algorithm

The whole optimizer fits in a few lines:

def adam(params, grads, m, v, t, lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8):
    t += 1
    for p, g, m_i, v_i in zip(params, grads, m, v):
        # Update biased moments
        m_i = beta1 * m_i + (1 - beta1) * g
        v_i = beta2 * v_i + (1 - beta2) * g**2

        # Bias correction
        m_hat = m_i / (1 - beta1**t)
        v_hat = v_i / (1 - beta2**t)

        # Update parameters
        p -= lr * m_hat / (sqrt(v_hat) + eps)

    return t

Default Hyperparameters

Parameter	Default	Description
$\alpha$ (lr)	0.001	Step size
$\beta_1$	0.9	First moment decay
$\beta_2$	0.999	Second moment decay
$\epsilon$	1e-8	Numerical stability

These defaults work remarkably well across many problems — which is a big part of why Adam became so popular.

Why Adam Works

Momentum ($\beta_1$)

• Accumulates gradient direction over time
• Smooths out noisy gradients (mini-batches give noisy estimates)
• Helps escape shallow local minima

Adaptive Learning Rate ($\beta_2$)

• Parameters with large gradients get smaller steps
• Parameters with small gradients get larger steps
• No manual learning rate scheduling needed

Adam Variants

Researchers have proposed many tweaks. The names come up often in papers and codebases:

Variant	Improvement
AdamW	Decoupled weight decay (better generalization)
AMSGrad	Non-increasing step sizes (convergence fix)
RAdam	Rectified Adam (variance correction)
AdaFactor	Memory-efficient for large models
LAMB	Layer-wise adaptive for large batch training
Lion	Simplified, sign-based updates

AdamW: The Modern Default

The one variant you should actually know is AdamW. In standard Adam, L2 regularization gets mixed into the gradient and is therefore rescaled by the adaptive step sizes, which weakens it in unpredictable ways. AdamW decouples weight decay from the gradient update:

$$\theta_{t+1} = \theta_t - \alpha \left( \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} + \lambda \theta_t \right)$$

The takeaway: weight decay is applied directly to the weights, not passed through the adaptive scaling. AdamW is now the default for Transformers and large language models.

Comparison with Other Optimizers

Optimizer	Momentum	Adaptive LR	Memory	Use Case
SGD	Optional	No	1x	Well-tuned vision models
RMSprop	No	Yes	2x	RNNs
Adam	Yes	Yes	3x	General purpose
AdamW	Yes	Yes	3x	Transformers, LLMs

The memory column matters: Adam stores two extra numbers ($m$ and $v$) for every parameter, so optimizer state can be twice the size of the model itself.

When NOT to Use Adam

1. ImageNet training: SGD with momentum often generalizes better
2. Memory-constrained: Adam needs 3x memory of SGD
3. Small datasets: Can overfit more than SGD

Learning Rate Scheduling

Adaptive steps do not remove the need for a schedule. Even with Adam, learning rate schedules help:

• Warmup: Start low, increase gradually (critical for transformers)
• Cosine decay: Smooth decrease to zero
• Step decay: Discrete reductions at milestones

Historical Impact

Adam’s impact:

• Became the default optimizer for most deep learning
• Enabled training without extensive hyperparameter search
• Made research more accessible (less tuning expertise needed)
• Foundation for modern optimizer development

Common Confusion

• Adam vs. AdamW: Adam is the original algorithm; AdamW changes only how weight decay is applied. When a modern paper says “Adam,” it often means AdamW in the code.
• Adaptive learning rate does not mean “no learning rate.” You still choose $\alpha$; Adam only rescales it per parameter.
• Momentum here is not classical momentum. Adam’s $m_t$ is an exponential moving average of gradients, closely related to but not identical to SGD-with-momentum.
• Optimizer vs. loss function: Adam does not change what you optimize (the loss), only how you take steps toward minimizing it.

Where To Go Next

• Read Backpropagation for where the gradients Adam consumes actually come from.
• Read Batch Normalization and Dropout for the other standard training tricks used alongside Adam.
• Read Transformer to see the architecture that made AdamW plus warmup the standard recipe.
• Read Scaling Laws for how training choices play out at very large scale.

Key Papers

• Adam: A Method for Stochastic Optimization – Kingma & Ba, 2014
  https://arxiv.org/abs/1412.6980
• Decoupled Weight Decay Regularization (AdamW) – Loshchilov & Hutter, 2017
  https://arxiv.org/abs/1711.05101
• On the Variance of the Adaptive Learning Rate and Beyond (RAdam) – Liu et al., 2019
  https://arxiv.org/abs/1908.03265
• Symbolic Discovery of Optimization Algorithms (Lion) – Chen et al., 2023
  https://arxiv.org/abs/2302.06675