Diffusion Models

Diffusion Models generate data by learning to reverse a gradual noising process. Starting from pure noise, they iteratively denoise to produce remarkably high-quality images, videos, and audio.

Helpful background: VAE and GAN show the earlier approaches to generation. This page covers diffusion in pixel space; Latent Diffusion covers the efficiency trick behind Stable Diffusion.

Why Students Should Care

Diffusion is the engine behind modern image and video generators — Stable Diffusion, DALL-E 2, and Imagen are all diffusion models.
The training objective is astonishingly simple (a mean-squared error on noise), which is why diffusion trains stably where GANs struggle.
The same “destroy data, learn to rebuild it” recipe generalizes beyond images to audio, video, and molecules.

The Idea in One Paragraph

Take a photo and add a tiny amount of static. Add a little more. Repeat a thousand times and you end up with pure noise — the image is destroyed. Each single step of that destruction is easy to describe. The diffusion insight: train a network to undo one small step of noising. If it can do that at every noise level, you can start from pure random noise and denoise step by step until an image appears — generating data from nothing.

Core Idea

The process has two phases:

Forward process (fixed, no learning): Gradually add Gaussian noise to data over $T$ steps:

q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t} x_{t-1}, \beta_t I)

Reverse process (learned): Denoise step by step:

p_\theta(x_{t-1} | x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t))

The takeaway: the forward direction is a fixed recipe for corrupting data; only the reverse direction — the denoiser — has parameters to train.

The Closed-Form Forward Process

A convenient property: you never have to noise an image step by step during training. We can jump directly to any timestep $t$ :

q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I)

where $\bar{\alpha}_t = \prod_{s=1}^{t} (1-\beta_s)$ .

This means: $x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon$ , where $\epsilon \sim \mathcal{N}(0, I)$ .

In plain English: an image at noise level $t$ is just a weighted blend of the clean image and one fresh draw of Gaussian noise. That makes training samples cheap to produce.

Training Objective

Instead of predicting the denoised mean $\mu_\theta$ directly, it works better to predict the noise $\epsilon$ that was added:

\mathcal{L}_{\text{simple}} = \mathbb{E}_{t, x_0, \epsilon}\left[ \| \epsilon - \epsilon_\theta(x_t, t) \|^2 \right]

The whole training loop is: sample a random timestep $t$ , noise the image, ask the network “what noise did I add?”, and penalize the squared error. Remarkably simple yet effective — no adversarial game, no delicate balance, just regression.

Sampling (Inference)

To generate, start with pure noise $x_T \sim \mathcal{N}(0, I)$ , then repeatedly take one denoising step:

x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \epsilon_\theta(x_t, t)\right) + \sigma_t z

where $z \sim \mathcal{N}(0, I)$ and $\sigma_t$ controls stochasticity.

You do not need to memorize this formula. The important idea is: each step subtracts the network’s noise estimate (scaled appropriately) and optionally re-injects a little fresh randomness, gradually turning noise into data.

Classifier-Free Guidance

How does a text prompt steer generation? Train the model both with and without conditioning, then at sampling time interpolate between the two predictions:

\tilde{\epsilon}_\theta(x_t, t, c) = \epsilon_\theta(x_t, t, \emptyset) + w \cdot (\epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \emptyset))

where $w > 1$ strengthens conditioning (e.g., on text prompts). Intuitively: figure out which direction the prompt pulls the denoising, then exaggerate that direction. This is the “guidance scale” knob in image generators.

Interactive Visualization

Watch the diffusion process in action — forward noising and reverse denoising:

Diffusion Process

Reverse process: Learn to denoise step-by-step to generate images.

Step 0/20Noise → Image

Training: Predict the noise ε at each step.
Inference: Start from pure noise, iteratively denoise.

Architecture: U-Net

The denoising network is typically a U-Net with:

Time embedding: Sinusoidal encoding of $t$ , so one network can handle every noise level
Attention layers: Self-attention and cross-attention (for conditioning on text or labels)
Skip connections: Preserve spatial information between the downsampling and upsampling paths

Key Models

Model	Innovation
DDPM	Foundational formulation
DDIM	Deterministic sampling, fewer steps
Stable Diffusion	Latent space diffusion
DALL-E 2	CLIP guidance
Imagen	Large language model conditioning

Why Diffusion Works

Unlike GANs (adversarial, unstable) or VAEs (blurry), diffusion models:

Have stable training (simple MSE loss)
Produce high-fidelity samples
Enable controllable generation
Scale effectively with compute

Common Confusion

The forward process is not learned. Only the reverse (denoising) network has trainable parameters; the noising schedule is fixed in advance.
The network predicts the noise, not the finished image — the clean image emerges only after many denoising steps.
Diffusion vs. latent diffusion: this page describes diffusion directly on pixels; latent diffusion first compresses images with a VAE and diffuses in that smaller space.
Training steps vs. sampling steps: training picks one random timestep per example, but generation walks through many steps — which is why samplers like DDIM that cut the step count matter so much in practice.

Where To Go Next

Read Latent Diffusion to see how Stable Diffusion makes this process affordable.
Read VAE and GAN for the generative models diffusion is usually compared against.
Read CLIP to understand the text encoders used for conditioning and guidance.
Read Attention Is All You Need for the attention layers inside the U-Net.