# In-Context Learning

> How large language models learn from examples in the prompt without weight updates

- Type: Concept
- By: Brown et al.
- Year: 2020
- Topics: Deep learning, NLP, Language models, Emergent abilities, Foundation models
- Page: https://aipedia.org/in-context-learning

## Resources

- [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165): Paper, Brown et al., 2020
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://arxiv.org/abs/2202.12837): Paper, Min et al., 2022
- [In-context Learning and Induction Heads](https://arxiv.org/abs/2209.11895): Paper, Olsson et al., 2022

**In-Context Learning (ICL)** is the ability of large language models to pick up a new task from a few examples written directly in the prompt — with no gradient updates and no fine-tuning. You show the model what you want, and it does it.

> If you are unsure how language models are trained in the first place, read [Pre-training](https://aipedia.org/pre-training) and [GPT](https://aipedia.org/gpt) first. This page is about what a *frozen* model can do at inference time.

## Why Students Should Care

- ICL is the reason prompting works at all. Every "few-shot prompt" you write relies on it.
- It broke a core assumption of machine learning: that learning a new task requires updating weights.
- It is the foundation for [Chain-of-Thought](https://aipedia.org/chain-of-thought) prompting and most practical LLM usage.

## The Phenomenon

Give the model a few input-output examples, and it generalizes to a new input:

```
Input: "The movie was terrible" → Sentiment: negative
Input: "I loved every minute" → Sentiment: positive
Input: "It was a waste of time" → Sentiment:
```

The model completes with "negative" — having learned the task from context alone. Nobody trained it on this exact format; it inferred the pattern from two examples.

## Why It's Surprising

Traditional machine learning requires:

1. Collect labeled data
2. Define a loss function
3. Optimize weights via gradient descent (see [Backpropagation](https://aipedia.org/backpropagation))

ICL skips all of this. The model's weights remain frozen; "learning" happens through attention over the prompt. The examples never change the model — they only change what the model conditions on.

## Formal Framework

Here is the same idea in notation. Let $x_1, y_1, \ldots, x_k, y_k$ be demonstration examples and $x_{test}$ be a new input. The model computes:

$$
P(y_{test} | x_1, y_1, \ldots, x_k, y_k, x_{test})
$$

You do not need to memorize this. It just says: **the model predicts the next output given everything in the prompt — the demonstrations create a task pattern that the model conditions on.**

## How Many Examples?

| Setting | Examples | Use Case |
|---------|----------|----------|
| Zero-shot | 0 | Task described in natural language |
| One-shot | 1 | Single example + new input |
| Few-shot | 2-32 | Multiple examples |

More examples generally improve accuracy, but with diminishing returns.

## Interactive Visualization

See how adding examples affects model predictions:

*Interactive demo: https://aipedia.org/in-context-learning*

## What Makes ICL Work?

Nobody fully knows — this is an active research question. Several hypotheses:

1. **Task Recognition**: the model recognizes the task from the examples and retrieves behavior it already learned during pre-training
2. **Implicit Fine-Tuning**: attention over the demonstrations acts like a form of gradient descent, without touching the weights
3. **Bayesian Inference**: the model infers which task distribution the demonstrations came from
4. **Induction Heads**: specific attention patterns inside the [Transformer](https://aipedia.org/transformer) copy patterns from earlier in the context

These are not mutually exclusive; ICL may be all of them at once, in different situations.

## Emergent Ability

ICL appears only at scale:

- Small models: cannot do ICL effectively
- GPT-3 (175B): strong ICL ability emerges
- Larger models: increasingly robust ICL

This "phase transition" makes ICL an **emergent capability** — you cannot see it coming by studying small models. See [Scaling Laws](https://aipedia.org/scaling-laws) for the broader story of what scale buys you.

## Best Practices

| Practice | Effect |
|----------|--------|
| Diverse examples | Better generalization |
| Consistent format | Clearer task signal |
| Similar examples to test | Improved accuracy |
| Clear separators | Reduces confusion |

## Limitations

- **Context window**: limited by maximum sequence length — every demonstration costs tokens
- **Order sensitivity**: performance can vary with example ordering
- **Recency bias**: the model may weight recent examples more heavily
- **Task complexity**: struggles with multi-step reasoning on its own — that is where [Chain-of-Thought](https://aipedia.org/chain-of-thought) comes in

## Common Confusion

- **ICL vs. fine-tuning**: fine-tuning updates the model's weights with gradient descent; ICL leaves the weights frozen and only changes the prompt. "Learning" in ICL is conditioning, not training.
- **ICL vs. Chain-of-Thought**: ICL is the general ability to learn from in-prompt examples. [Chain-of-Thought](https://aipedia.org/chain-of-thought) is a specific style of examples that include reasoning steps.
- **Zero-shot is still ICL-adjacent**: even with zero examples, describing a task in plain language and getting correct behavior relies on the same conditioning machinery.

## Where To Go Next

- Read [Chain-of-Thought](https://aipedia.org/chain-of-thought) for the prompting technique that fixes ICL's weakness on multi-step reasoning.
- Read [GPT](https://aipedia.org/gpt) for the paper that made few-shot in-context learning famous.
- Read [Scaling Laws](https://aipedia.org/scaling-laws) to see why abilities like ICL show up only in large models.
- Read [Transformer](https://aipedia.org/transformer) and [Attention Is All You Need](https://aipedia.org/attention-is-all-you-need) to understand the attention mechanism that makes conditioning on demonstrations possible.