Latent Diffusion Models

Latent Diffusion Models (LDMs) make high-resolution image generation practical by running the diffusion process in a compressed latent space rather than pixel space. This is the architecture behind Stable Diffusion.

Read Diffusion Models first — this page assumes you know the forward/reverse noising process. The VAE page explains the autoencoder used for compression here.

Why Students Should Care

• This is the architecture of Stable Diffusion, one of the most widely used generative models ever released.
• It demonstrates a general and reusable pattern: compress first, then model — do the expensive learning in a small learned space instead of raw data space.
• The efficiency gains are what moved image generation from datacenter-only to consumer GPUs.

The Problem with Pixel-Space Diffusion

Diffusing a 512×512×3 image requires processing 786,432 dimensions per step. For thousands of denoising steps, this is extremely expensive. Worse, most of that effort is wasted: the majority of pixel-level detail is imperceptible texture, not the content of the image.

The Solution: Compress First

The idea: let a small autoencoder handle the perceptual detail, and let diffusion handle only the compressed “essence” of the image. LDMs use a two-stage approach:

1. Encode: Compress images to a smaller latent space
2. Diffuse: Run diffusion in latent space
3. Decode: Reconstruct to pixel space

$$z = \mathcal{E}(x), \quad x' = \mathcal{D}(\hat{z})$$

where $\mathcal{E}$ is the encoder, $\mathcal{D}$ is the decoder, and $\hat{z}$ is the denoised latent.

The Autoencoder

The compressor is typically a VQ-VAE or KL-VAE trained separately, before any diffusion happens:

$$\mathcal{L}_{AE} = \|x - \mathcal{D}(\mathcal{E}(x))\|^2 + \lambda \cdot \text{Reg}(\mathcal{E}(x))$$

In plain English: reconstruct the image well (first term), while keeping the latent space well-behaved (second term). The latent space is typically 8× or 4× smaller per dimension (64× or 16× in total pixels).

Diffusion in Latent Space

Once the autoencoder is trained and frozen, diffusion proceeds exactly as usual — same noising process, same noise-prediction loss — just on latents instead of pixels:

$$\mathcal{L}_{LDM} = \mathbb{E}_{z, \epsilon, t}\left[\|\epsilon - \epsilon_\theta(z_t, t, c)\|^2\right]$$

where $c$ is the conditioning (text, class, etc.). Nothing about diffusion itself changes; only the space it operates in.

Interactive Visualization

See how the latent space compression enables efficient generation:

Latent Diffusion Pipeline

Pixel Space

512×512×3

Encode

→

Latent Space

64×64×4

Decode

→ 512×512

🖼️

512×512

786k dims

→

VAE

Encoder

→

64×64

16k dims

→

VAE

Decoder

→

✨

Generated

Pixel Diffusion

786,432 dims

~24GB GPU memory

Latent Diffusion

16,384 dims

~8GB GPU memory (48× smaller)

Conditioning via Cross-Attention

How does the text prompt get into the model? Text conditioning is injected through cross-attention inside the denoising U-Net:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V$$

where:

• $Q$ comes from the latent (image features)
• $K, V$ come from the text encoder (CLIP or T5)

Intuitively, each spatial location of the image latent “looks up” the most relevant words in the prompt at every denoising step. This is the same attention mechanism as in the Transformer, used across two modalities.

Architecture Overview

Text → CLIP → Cross-Attention
                    ↓
Noise → U-Net (in latent space) → Denoised Latent
                                        ↓
                                    Decoder → Image

Efficiency Gains

The numbers make the case:

Metric	Pixel Diffusion	Latent Diffusion
Dimensions	512×512×3	64×64×4
Total dims	786,432	16,384
Compression	1×	48×
GPU memory	~24GB	~8GB

Stable Diffusion Specifics

• VAE: KL-regularized autoencoder
• U-Net: ~860M parameters
• Text encoder: CLIP ViT-L/14 (frozen)
• Latent size: 64×64×4 for 512×512 images

Training Pipeline

1. Stage 1: Train VAE on images (reconstruction + regularization)
2. Stage 2: Train U-Net on latents with frozen VAE
3. Optional: Fine-tune on specific domains

Key Insight

By separating perceptual compression (autoencoder) from semantic generation (diffusion), LDMs achieve:

• High-quality generation
• Computational efficiency
• Flexible conditioning

Common Confusion

• “Latent diffusion” vs. “Stable Diffusion”: latent diffusion is the general architecture; Stable Diffusion is one specific, famous instance of it.
• The VAE here is not the generative model. It is a fixed compressor/decompressor trained in advance; the diffusion U-Net does the actual generation. Compare with a standalone VAE, where the VAE itself generates.
• Two different “latents”: the VAE’s latent (a compressed image) and the diffusion process’s noisy states are both called latents — in an LDM, diffusion’s noisy states live in the VAE’s latent space.
• Cross-attention vs. self-attention: self-attention lets image features attend to each other; cross-attention is what connects image features to the text prompt.

Where To Go Next

• Read Diffusion Models for the denoising process this page builds on.
• Read VAE for how the compression stage is trained.
• Read CLIP for the frozen text encoder that powers prompting.
• Read Attention Is All You Need for the attention mechanism behind cross-attention conditioning.