In-Context Learning

In-Context Learning (ICL) is the ability of large language models to pick up a new task from a few examples written directly in the prompt — with no gradient updates and no fine-tuning. You show the model what you want, and it does it.

If you are unsure how language models are trained in the first place, read Pre-training and GPT first. This page is about what a frozen model can do at inference time.

Why Students Should Care

• ICL is the reason prompting works at all. Every “few-shot prompt” you write relies on it.
• It broke a core assumption of machine learning: that learning a new task requires updating weights.
• It is the foundation for Chain-of-Thought prompting and most practical LLM usage.

The Phenomenon

Give the model a few input-output examples, and it generalizes to a new input:

Input: "The movie was terrible" → Sentiment: negative
Input: "I loved every minute" → Sentiment: positive
Input: "It was a waste of time" → Sentiment:

The model completes with “negative” — having learned the task from context alone. Nobody trained it on this exact format; it inferred the pattern from two examples.

Why It’s Surprising

Traditional machine learning requires:

1. Collect labeled data
2. Define a loss function
3. Optimize weights via gradient descent (see Backpropagation)

ICL skips all of this. The model’s weights remain frozen; “learning” happens through attention over the prompt. The examples never change the model — they only change what the model conditions on.

Formal Framework

Here is the same idea in notation. Let $x_1, y_1, \ldots, x_k, y_k$ be demonstration examples and $x_{test}$ be a new input. The model computes:

$$P(y_{test} | x_1, y_1, \ldots, x_k, y_k, x_{test})$$

You do not need to memorize this. It just says: the model predicts the next output given everything in the prompt — the demonstrations create a task pattern that the model conditions on.

How Many Examples?

Setting	Examples	Use Case
Zero-shot	0	Task described in natural language
One-shot	1	Single example + new input
Few-shot	2-32	Multiple examples

More examples generally improve accuracy, but with diminishing returns.

Interactive Visualization

See how adding examples affects model predictions:

In-Context Learning Demo

Examples:2

Prompt:

Input: "The movie was terrible" → negative

Input: "I loved every minute" → positive

Input: "The acting was brilliant" → ?

Model Prediction

positive(81% confidence)

Probability Distribution

positive81%

negative19%

Observation: More examples → higher confidence. The model "learns" the sentiment task from context alone.

What Makes ICL Work?

Nobody fully knows — this is an active research question. Several hypotheses:

1. Task Recognition: the model recognizes the task from the examples and retrieves behavior it already learned during pre-training
2. Implicit Fine-Tuning: attention over the demonstrations acts like a form of gradient descent, without touching the weights
3. Bayesian Inference: the model infers which task distribution the demonstrations came from
4. Induction Heads: specific attention patterns inside the Transformer copy patterns from earlier in the context

These are not mutually exclusive; ICL may be all of them at once, in different situations.

Emergent Ability

ICL appears only at scale:

• Small models: cannot do ICL effectively
• GPT-3 (175B): strong ICL ability emerges
• Larger models: increasingly robust ICL

This “phase transition” makes ICL an emergent capability — you cannot see it coming by studying small models. See Scaling Laws for the broader story of what scale buys you.

Best Practices

Practice	Effect
Diverse examples	Better generalization
Consistent format	Clearer task signal
Similar examples to test	Improved accuracy
Clear separators	Reduces confusion

Limitations

• Context window: limited by maximum sequence length — every demonstration costs tokens
• Order sensitivity: performance can vary with example ordering
• Recency bias: the model may weight recent examples more heavily
• Task complexity: struggles with multi-step reasoning on its own — that is where Chain-of-Thought comes in

Common Confusion

• ICL vs. fine-tuning: fine-tuning updates the model’s weights with gradient descent; ICL leaves the weights frozen and only changes the prompt. “Learning” in ICL is conditioning, not training.
• ICL vs. Chain-of-Thought: ICL is the general ability to learn from in-prompt examples. Chain-of-Thought is a specific style of examples that include reasoning steps.
• Zero-shot is still ICL-adjacent: even with zero examples, describing a task in plain language and getting correct behavior relies on the same conditioning machinery.

Where To Go Next

• Read Chain-of-Thought for the prompting technique that fixes ICL’s weakness on multi-step reasoning.
• Read GPT for the paper that made few-shot in-context learning famous.
• Read Scaling Laws to see why abilities like ICL show up only in large models.
• Read Transformer and Attention Is All You Need to understand the attention mechanism that makes conditioning on demonstrations possible.