# Latent Diffusion Models

> High-resolution image generation by diffusing in learned latent spaces

- Type: Paper
- By: Rombach et al.
- Year: 2021
- Topics: Deep learning, Generative, Vision, Diffusion
- Page: https://aipedia.org/latent-diffusion

## Resources

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752): Paper, Rombach et al., 2021
- [CompVis/latent-diffusion](https://github.com/CompVis/latent-diffusion): Code, Official code
- [CompVis/stable-diffusion](https://github.com/CompVis/stable-diffusion): Code, Stable Diffusion v1
- [The Illustrated Stable Diffusion](https://jalammar.github.io/illustrated-stable-diffusion/): Article, Jay Alammar

**Latent Diffusion Models (LDMs)** make high-resolution image generation practical by running the diffusion process in a compressed latent space rather than pixel space. This is the architecture behind Stable Diffusion.

> Read [Diffusion Models](https://aipedia.org/diffusion-models) first — this page assumes you know the forward/reverse noising process. The [VAE](https://aipedia.org/vae) page explains the autoencoder used for compression here.

## Why Students Should Care

- This is the architecture of **Stable Diffusion**, one of the most widely used generative models ever released.
- It demonstrates a general and reusable pattern: **compress first, then model** — do the expensive learning in a small learned space instead of raw data space.
- The efficiency gains are what moved image generation from datacenter-only to consumer GPUs.

## The Problem with Pixel-Space Diffusion

Diffusing a 512×512×3 image requires processing 786,432 dimensions per step. For thousands of denoising steps, this is extremely expensive. Worse, most of that effort is wasted: the majority of pixel-level detail is imperceptible texture, not the content of the image.

## The Solution: Compress First

The idea: let a small autoencoder handle the perceptual detail, and let diffusion handle only the compressed "essence" of the image. LDMs use a two-stage approach:

1. **Encode**: Compress images to a smaller latent space
2. **Diffuse**: Run diffusion in latent space
3. **Decode**: Reconstruct to pixel space

$$
z = \mathcal{E}(x), \quad x' = \mathcal{D}(\hat{z})
$$

where $\mathcal{E}$ is the encoder, $\mathcal{D}$ is the decoder, and $\hat{z}$ is the denoised latent.

## The Autoencoder

The compressor is typically a **VQ-VAE** or **KL-VAE** trained separately, before any diffusion happens:

$$
\mathcal{L}_{AE} = \|x - \mathcal{D}(\mathcal{E}(x))\|^2 + \lambda \cdot \text{Reg}(\mathcal{E}(x))
$$

In plain English: reconstruct the image well (first term), while keeping the latent space well-behaved (second term). The latent space is typically 8× or 4× smaller per dimension (64× or 16× in total pixels).

## Diffusion in Latent Space

Once the autoencoder is trained and frozen, diffusion proceeds exactly as usual — same noising process, same noise-prediction loss — just on latents instead of pixels:

$$
\mathcal{L}_{LDM} = \mathbb{E}_{z, \epsilon, t}\left[\|\epsilon - \epsilon_\theta(z_t, t, c)\|^2\right]
$$

where $c$ is the conditioning (text, class, etc.). Nothing about diffusion itself changes; only the space it operates in.

## Interactive Visualization

See how the latent space compression enables efficient generation:

*Interactive demo: https://aipedia.org/latent-diffusion*

## Conditioning via Cross-Attention

How does the text prompt get into the model? Text conditioning is injected through cross-attention inside the denoising U-Net:

$$
\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V
$$

where:
- $Q$ comes from the latent (image features)
- $K, V$ come from the text encoder (CLIP or T5)

Intuitively, each spatial location of the image latent "looks up" the most relevant words in the prompt at every denoising step. This is the same attention mechanism as in the [Transformer](https://aipedia.org/transformer), used across two modalities.

## Architecture Overview

```
Text → CLIP → Cross-Attention
                    ↓
Noise → U-Net (in latent space) → Denoised Latent
                                        ↓
                                    Decoder → Image
```

## Efficiency Gains

The numbers make the case:

| Metric | Pixel Diffusion | Latent Diffusion |
|--------|-----------------|------------------|
| Dimensions | 512×512×3 | 64×64×4 |
| Total dims | 786,432 | 16,384 |
| Compression | 1× | 48× |
| GPU memory | ~24GB | ~8GB |

## Stable Diffusion Specifics

- **VAE**: KL-regularized autoencoder
- **U-Net**: ~860M parameters
- **Text encoder**: CLIP ViT-L/14 (frozen)
- **Latent size**: 64×64×4 for 512×512 images

## Training Pipeline

1. **Stage 1**: Train VAE on images (reconstruction + regularization)
2. **Stage 2**: Train U-Net on latents with frozen VAE
3. **Optional**: Fine-tune on specific domains

## Key Insight

By separating perceptual compression (autoencoder) from semantic generation (diffusion), LDMs achieve:
- High-quality generation
- Computational efficiency
- Flexible conditioning

## Common Confusion

- **"Latent diffusion" vs. "Stable Diffusion"**: latent diffusion is the general architecture; Stable Diffusion is one specific, famous instance of it.
- **The VAE here is not the generative model.** It is a fixed compressor/decompressor trained in advance; the diffusion U-Net does the actual generation. Compare with a standalone [VAE](https://aipedia.org/vae), where the VAE itself generates.
- **Two different "latents"**: the VAE's latent (a compressed image) and the diffusion process's noisy states are both called latents — in an LDM, diffusion's noisy states *live in* the VAE's latent space.
- **Cross-attention vs. self-attention**: self-attention lets image features attend to each other; cross-attention is what connects image features to the *text prompt*.

## Where To Go Next

- Read [Diffusion Models](https://aipedia.org/diffusion-models) for the denoising process this page builds on.
- Read [VAE](https://aipedia.org/vae) for how the compression stage is trained.
- Read [CLIP](https://aipedia.org/clip) for the frozen text encoder that powers prompting.
- Read [Attention Is All You Need](https://aipedia.org/attention-is-all-you-need) for the attention mechanism behind cross-attention conditioning.