# BERT: Bidirectional Transformers

> Pre-training deep bidirectional representations for NLP

- Type: Paper
- By: Devlin et al.
- Year: 2018
- Topics: Architecture, Transformers, Deep learning, NLP, Pre training
- Page: https://aipedia.org/bert

## Resources

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805): Paper, Devlin et al., 2018
- [google-research/bert](https://github.com/google-research/bert): Code, Official code and models
- [The Illustrated BERT, ELMo, and co.](https://jalammar.github.io/illustrated-bert/): Article, Jay Alammar
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692): Paper, Liu et al., 2019
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942): Paper, Lan et al., 2019

**BERT** (Bidirectional Encoder Representations from Transformers) is a language model that reads a sentence in both directions at once. Published by Devlin et al. at Google in 2018, it achieved state-of-the-art results on 11 NLP tasks and became one of the most cited AI papers ever.

> If this page feels too fast, read [Transformer](https://aipedia.org/transformer) first. BERT is an **encoder-only** Transformer trained for understanding; its sibling [GPT](https://aipedia.org/gpt) is a **decoder-only** Transformer trained for generation.

## Why Students Should Care

- BERT popularized the **pre-train then fine-tune** recipe that most of modern NLP still follows.
- It shows why **bidirectional context** matters: understanding a word often requires the words after it, not just before.
- BERT-style encoders still power search engines, classifiers, and embedding models today.

## A Quick Example

Fill in the blank:

`The man went to the [BLANK] to buy milk.`

You can guess "store" because you read the whole sentence — including "to buy milk," which comes **after** the blank. A left-to-right model only sees "The man went to the" and has far less to work with. BERT is trained on exactly this kind of fill-in-the-blank task, so every word gets to use context from both sides.

## The Key Insight

Previous language models were either left-to-right (GPT) or used shallow concatenation of left-to-right and right-to-left models (ELMo). BERT's breakthrough: **mask tokens and predict them using both left and right context simultaneously**.

$$
P(w_i | w_1, ..., w_{i-1}, w_{i+1}, ..., w_n)
$$

You do not need to memorize the notation. The important idea is: **predict a hidden word from everything around it**, so every token "sees" the entire sentence when building its representation.

## Pre-training Objectives

BERT is pre-trained on two tasks that need no human labels — the text itself is the supervision.

### 1. Masked Language Modeling (MLM)

Randomly mask 15% of tokens and predict them. The masked tokens are:

- 80% replaced with `[MASK]`
- 10% replaced with a random token
- 10% left unchanged

$$
\mathcal{L}_{MLM} = -\sum_{i \in \text{masked}} \log P(w_i | \mathbf{h}_i)
$$

In plain English: hide some words, make the model guess them, and penalize wrong guesses. The mix of replacements stops the model from only learning about the special `[MASK]` token, which never appears at fine-tuning time.

### 2. Next Sentence Prediction (NSP)

Given sentence A and sentence B, predict whether B actually follows A in the original text:

```
[CLS] The cat sat on the mat [SEP] It was comfortable [SEP]
Label: IsNext
```

This teaches the model relationships **between** sentences, useful for tasks like question answering.

## Architecture

BERT uses the **Transformer encoder** (no decoder), stacked deep:

| Model | Layers | Hidden | Heads | Parameters |
|-------|--------|--------|-------|------------|
| BERT-Base | 12 | 768 | 12 | 110M |
| BERT-Large | 24 | 1024 | 16 | 340M |

## Interactive Demo

Explore how BERT processes tokens bidirectionally and learns masked predictions:

*Interactive demo: https://aipedia.org/bert*

## Input Representation

Before any attention happens, BERT turns each token into a vector by adding three embeddings together:

$$
\text{Input} = \text{Token Emb} + \text{Segment Emb} + \text{Position Emb}
$$

- **Token embeddings**: which word piece this is (WordPiece vocabulary, 30K tokens)
- **Segment embeddings**: whether the token belongs to sentence A or sentence B
- **Position embeddings**: where the token sits in the sequence (learned, not sinusoidal)

## Fine-tuning

BERT popularized the "pre-train then fine-tune" paradigm:

1. **Pre-train** once on massive unlabeled text (BooksCorpus + Wikipedia)
2. **Fine-tune** cheaply on task-specific labeled data

Fine-tuning adds only a simple output layer on top of the pre-trained encoder:

- **Classification**: use the `[CLS]` token's representation
- **Token tagging**: use each token's representation
- **Question answering**: predict start/end span positions

The heavy lifting — learning what language means — happens once during pre-training and transfers to every downstream task.

## Why It Works

1. **Bidirectional context**: every token sees the full sentence
2. **Deep representations**: 12-24 Transformer layers
3. **Transfer learning**: pre-trained knowledge transfers to many tasks
4. **Attention patterns**: the model learns syntactic and semantic relationships

## Historical Impact

BERT sparked the "BERT-ology" era of follow-up models:

- **RoBERTa**: better training recipe (no NSP, more data)
- **ALBERT**: parameter sharing for efficiency
- **DistilBERT**: knowledge distillation (40% smaller, 60% faster)
- **SpanBERT**: span masking for better representations
- **ELECTRA**: replaced-token detection instead of masking

## Common Confusion

- **BERT is an encoder, GPT is a decoder.** BERT reads text in both directions but cannot naturally generate text; GPT generates text but only sees left context.
- **Masked language modeling is not the same as next-token prediction.** MLM fills in blanks anywhere in the sentence; GPT-style models always predict the next token.
- **BERT the model vs. the paradigm.** "BERT" often refers loosely to any encoder-only pre-trained Transformer (RoBERTa, ALBERT, etc.), not just the original 2018 model.

## Where To Go Next

- Read [Transformer](https://aipedia.org/transformer) for the architecture BERT is built on
- Read [GPT](https://aipedia.org/gpt) for the decoder-only, generative alternative
- Read [Attention Is All You Need](https://aipedia.org/attention-is-all-you-need) for the original Transformer paper
- Read [Pre-training](https://aipedia.org/pre-training) for the broader pre-train-then-adapt paradigm
- Read [Word2Vec](https://aipedia.org/word2vec) for the earlier, static word embeddings BERT improved on