CLIP: Contrastive Language-Image Pre-training

CLIP (Contrastive Language-Image Pre-training) learns to connect images and text by training on 400 million image-text pairs from the internet. Instead of teaching a model a fixed list of categories, CLIP teaches it to answer one question — does this caption describe this image? — and that simple objective produces remarkably flexible visual representations.

This page assumes you know what an encoder is. If not, read Transformer and Vision Transformer first.

Why Students Should Care

• CLIP is the bridge between vision and language — the recipe behind image search, open-vocabulary detection, and the visual side of many multimodal LLMs.
• It popularized zero-shot classification: classifying images from datasets the model was never trained on, just by describing the classes in words.
• Its text embeddings guide image generators like DALL-E and Stable Diffusion (see Latent Diffusion).

A Quick Example

Show CLIP a photo and two captions: “a photo of a dog” and “a photo of a cat”. CLIP embeds the image and both captions into the same vector space and checks which caption’s vector points in the most similar direction to the image’s vector. Whichever caption matches best wins. That is the entire mechanism — everything below is about how those embeddings are learned.

Core Idea: Learn from Captions

Instead of predicting fixed categories, CLIP learns to match images with their natural language descriptions. The match score is cosine similarity between the two embeddings:

$$\text{similarity}(I, T) = \frac{f_I(I) \cdot f_T(T)}{\|f_I(I)\| \cdot \|f_T(T)\|}$$

where $f_I$ is an image encoder (ViT or ResNet) and $f_T$ is a text encoder (Transformer).

In plain English: two encoders map images and captions into a shared space, where matching pairs sit close together.

Contrastive Learning

How do you train those encoders? Take a batch of $N$ image-text pairs. For each image, its own caption is the “right answer” and the other $N-1$ captions are wrong answers — and vice versa for each caption. CLIP maximizes similarity for correct pairs while minimizing it for incorrect ones:

$$\mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \left[ \log \frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^{N} \exp(s_{ij}/\tau)} + \log \frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^{N} \exp(s_{ji}/\tau)} \right]$$

where $s_{ij} = f_I(I_i) \cdot f_T(T_j)$ and $\tau$ is a learned temperature.

You do not need to memorize the loss. The important idea is: every batch is a matching game — pull true pairs together, push mismatched pairs apart.

Zero-Shot Classification

Because CLIP understands captions, you can build a classifier for a dataset it has never seen — with no extra training:

1. Create text prompts: “a photo of a {class}”
2. Encode all prompts with the text encoder
3. Encode the image with the image encoder
4. Predict the class with the highest image-text similarity

$$p(y|x) = \frac{\exp(f_I(x) \cdot f_T(\text{prompt}_y) / \tau)}{\sum_{c} \exp(f_I(x) \cdot f_T(\text{prompt}_c) / \tau)}$$

In other words: the class names themselves become the classifier.

Interactive Visualization

Explore how CLIP matches images to text descriptions:

CLIP: Image-Text Matching

Click an image or text to see similarity scores. CLIP learns to maximize diagonal (matching pairs).

Images

🐱🐕🚗🌲

Similarity Matrix

0.92

0.35

0.12

0.18

0.38

0.89

0.15

0.21

0.11

0.14

0.94

0.08

0.22

0.19

0.07

0.91

Text Prompts

a photo of a cata photo of a doga photo of a cara photo of a tree

Zero-shot: To classify a new image, compute similarity with text prompts like "a photo of a {class}" and pick the highest.

Prompt Engineering

Because the classifier is built from text, how you phrase the text matters. Classification accuracy depends on the prompt:

Prompt Template	ImageNet Acc
”{class}“	63.5%
“a photo of a {class}“	68.3%
“a good photo of a {class}“	69.1%
Ensemble of 80 templates	76.2%

A bare class name underperforms because CLIP’s training captions were full sentences, not single words.

Why CLIP Works

1. Scale: 400M image-text pairs provide diverse supervision
2. Natural language: captures nuanced visual concepts that fixed label sets cannot
3. Contrastive learning: makes efficient use of every batch — each pair supplies many negative examples
4. Zero-shot transfer: no task-specific training needed

Applications

CLIP powers:

• Image search: find images matching text queries
• DALL-E / Stable Diffusion: guide image generation
• Open-vocabulary detection: detect objects by name
• Multimodal models: visual understanding in LLMs

Limitations

• Struggles with fine-grained recognition
• Inherits biases from internet data
• The text encoder limits complex reasoning
• Counting and spatial relationships remain challenging

Common Confusion

• CLIP does not generate images. It scores image-text similarity. Generators like DALL-E and Stable Diffusion use CLIP-style embeddings for guidance, but the generation happens elsewhere — see Diffusion Models.
• “Zero-shot” here means no training on the target dataset — CLIP still saw 400M image-text pairs during pre-training. It is zero-shot with respect to the task, not to data in general.
• CLIP is a training method plus a pair of encoders, not one specific network. The image encoder can be a Vision Transformer or a ResNet; the recipe is the same.

Where To Go Next

• Read Vision Transformer for CLIP’s strongest image encoder
• Read Transformer for the architecture behind the text encoder
• Read Pre-training for the broader paradigm CLIP belongs to
• Read Latent Diffusion to see text embeddings steering image generation