# Pre-training

> The stage where a model learns broad patterns from a very large dataset

- Type: Concept
- Topics: Deep learning, Training, Foundation models
- Page: https://aipedia.org/pre-training

## Resources

- [Improving Language Understanding by Generative Pre-Training](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf): Paper, Radford et al., 2018
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805): Paper, Devlin et al., 2018
- [Why Does Unsupervised Pre-training Help Deep Learning?](https://jmlr.org/papers/v11/erhan10a.html): Paper, Erhan et al., 2010
- [Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556): Paper, Hoffmann et al., 2022 (Chinchilla)

**Pre-training** is the first major stage in building a modern foundation model. During this stage, the model is exposed to a huge amount of data and learns broad patterns in language, code, images, or other modalities before it is adapted to a narrower task.

> Good follow-up pages: [GPT](https://aipedia.org/gpt), [BERT](https://aipedia.org/bert), [Transformer](https://aipedia.org/transformer), and [Scaling Laws](https://aipedia.org/scaling-laws).

## A Simple Example

Suppose you want a model that can answer biology questions. You could train it only on biology examples, but that would be limiting.

Instead, pre-training teaches the model broad skills first:

- how sentences are structured
- how facts are expressed in text
- how code, math, or diagrams tend to look

After that broad stage, the model can be adapted more efficiently to specific tasks.

## Why Pre-training Matters

Pre-training does three important things:

1. It gives the model general-purpose representations
2. It reduces the amount of labeled task-specific data you need later
3. It creates a reusable base model for many downstream uses

That is why so much of modern AI development is really about building and adapting pre-trained models.

## Common Objectives

Most pre-training uses **self-supervised learning**, which means the training signal comes from the data itself.

### Causal Language Modeling

Predict the next token from the previous ones:

$$
P(x) = \prod_{i=1}^n P(x_i | x_{<i})
$$

This is the objective used by GPT-style models.

### Masked Language Modeling

Hide some tokens and ask the model to fill them in from context.

This is the objective used by BERT-style models.

## The Pre-training Pipeline

1. **Data curation**: collect, filter, deduplicate, and clean large corpora
2. **Tokenization**: convert raw text or other inputs into model-ready units
3. **Training**: optimize the model across many steps on large compute clusters
4. **Evaluation**: track loss and benchmark behavior

## What Makes Pre-training Hard

- huge compute requirements
- messy internet-scale data
- difficult decisions about data mixture and quality
- strong interactions between model size, tokens, and compute budget

That is where [Scaling Laws](https://aipedia.org/scaling-laws) become useful.

## Pre-training vs Fine-Tuning

| Stage | Goal |
|-------|------|
| Pre-training | Learn broad patterns from massive data |
| Fine-tuning | Adapt the model to a narrower behavior or task |
| Preference tuning / alignment | Make behavior more useful to people |

## What To Remember

- Pre-training is the broad learning stage, not the final product
- It usually uses self-supervised objectives
- Most modern AI systems start with a strong pre-trained base model