GPT: Generative Pre-Training

GPT (Generative Pre-trained Transformer) is a family of language models built on one deceptively simple task: predict the next token. Starting with GPT-1 (2018) from OpenAI, the series demonstrated that scaling Transformer decoders creates increasingly capable general-purpose models.

If this page feels too fast, read Transformer first. GPT is a decoder-only Transformer trained for generation; its sibling BERT is an encoder-only Transformer trained for understanding.

Why Students Should Care

GPT is the direct ancestor of ChatGPT, Claude, and most modern LLMs — they are all decoder-only next-token predictors.
The GPT series is the clearest demonstration of the scaling hypothesis: same recipe, more parameters and data, qualitatively new abilities.
Ideas you will meet everywhere — prompting, few-shot learning, foundation models — were established by these papers.

A Quick Example

Read this and predict the next word:

The capital of France is

You said “Paris.” To do that reliably across billions of sentences, a model has to absorb grammar, facts, and even reasoning patterns. That is the whole training signal: guess the next token, get corrected, repeat.

Core Idea: Next Token Prediction

Formally, GPT models the probability of a sequence as a chain of next-token predictions:

P(x_1, x_2, ..., x_n) = \prod_{i=1}^{n} P(x_i | x_1, ..., x_{i-1})

You do not need to memorize the equation. The important idea is: each token is predicted from everything before it, and nothing after it. This simple objective — predicting what comes next — turns out to encode rich understanding of language, facts, and reasoning.

The Architecture

GPT uses the Transformer decoder with causal (masked) self-attention. “Causal” means each token can only attend to earlier tokens — the future is hidden:

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right)V

where $M$ is a causal mask preventing attention to future tokens. Without the mask, predicting the next token would be trivial cheating — the answer would be visible.

Model	Year	Parameters	Context	Training Data
GPT-1	2018	117M	512	BookCorpus
GPT-2	2019	1.5B	1024	WebText (40GB)
GPT-3	2020	175B	2048	570GB filtered
GPT-4	2023	~1.7T*	8K-128K	Unknown

*Estimated, not officially disclosed

Interactive Demo

Explore autoregressive generation and causal attention:

GPT: Autoregressive Generation

The

quick

brown

fox

How GPT works: Given a sequence of tokens, GPT predicts the most likely next token. It can only attend to previous tokens (causal masking), making generation natural: each new token is sampled from the predicted distribution, then appended to the context.

GPT-1

117M

GPT-2

1.5B

GPT-3

175B

GPT-4

~1.7T

GPT-2: The Scaling Revelation

GPT-2 demonstrated emergent capabilities from scale — abilities nobody explicitly trained for:

Zero-shot task performance without fine-tuning
Coherent long-form text generation
Basic reasoning and arithmetic
Translation and summarization (without training for it)

The paper’s key insight: “Language models are unsupervised multitask learners.” Learning to predict text well means implicitly learning the tasks that text describes.

GPT-3: In-Context Learning

GPT-3 introduced few-shot prompting: show the model a couple of examples inside the prompt, and it picks up the pattern.

Translate English to French:
sea otter => loutre de mer
peppermint => menthe poivrée
cheese =>

The model learns new tasks from examples in the prompt, without any gradient updates — its weights never change. This emerged purely from scale. See In-Context Learning for more.

Training Objective

Training minimizes the standard language modeling loss — how surprised the model is by each true next token:

\mathcal{L} = -\sum_{i=1}^{n} \log P(x_i | x_1, ..., x_{i-1}; \theta)

GPT-3 additionally uses:

Sparse attention patterns for efficiency
Model parallelism across many GPUs
Careful data deduplication

Why Decoder-Only?

BERT uses encoders (bidirectional), GPT uses decoders (causal). Why choose one or the other?

Aspect	Encoder (BERT)	Decoder (GPT)
Training	Masked LM	Next token
Generation	Cannot generate	Natural generation
Understanding	Both directions	Left context only
Use case	Classification, QA	Generation, chat

Modern LLMs (GPT-4, Claude) use decoder-only architectures because generation is the primary interface.

Scaling Laws

GPT-3 revealed that performance improves predictably with size:

L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N}

Loss decreases as a power law with model size $N$ , data size $D$ , and compute $C$ . The takeaway: you can forecast how good a bigger model will be before you train it. See Scaling Laws for the full story.

From GPT to ChatGPT

A raw language model completes text; it does not naturally follow instructions. The path to conversational AI:

GPT-3: raw language model
InstructGPT: fine-tuned to follow instructions (RLHF)
ChatGPT: optimized for dialogue

RLHF (Reinforcement Learning from Human Feedback) aligns the model with human preferences — see RLHF.

Historical Impact

GPT established:

Scaling hypothesis: bigger models → better capabilities
Emergent abilities: capabilities appearing at scale
Prompt engineering: programming via natural language
Foundation models: one model, many tasks

Common Confusion

GPT is the model family; ChatGPT is a product built on GPT models plus instruction tuning and RLHF.
Decoder-only does not mean worse at understanding. GPT models understand text too — they just build that understanding from left context only.
In-context learning is not fine-tuning. Few-shot prompting changes no weights; fine-tuning does.

Where To Go Next

Read Transformer for the underlying architecture
Read BERT for the encoder-only alternative
Read Scaling Laws for why bigger keeps getting better
Read In-Context Learning for how prompting works as learning
Read RLHF for how raw GPT became a helpful assistant

Key Papers

Improving Language Understanding by Generative Pre-Training – Radford et al., 2018
https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
Language Models are Unsupervised Multitask Learners (GPT-2) – Radford et al., 2019
https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
Language Models are Few-Shot Learners (GPT-3) – Brown et al., 2020
https://arxiv.org/abs/2005.14165
Training language models to follow instructions with human feedback (InstructGPT) – Ouyang et al., 2022
https://arxiv.org/abs/2203.02155