Neural Machine Translation by Jointly Learning to Align and Translate

This landmark paper by Bahdanau, Cho, and Bengio introduced the attention mechanism for neural machine translation: instead of squeezing a whole sentence into one vector, let the decoder look back at the input and pick what is relevant at each step. It is one of the most influential papers in modern AI.

If this page feels too fast, read Sequence to Sequence first — this paper fixes seq2seq’s biggest weakness. For where attention went next, see Transformer.

Why Students Should Care

• This is where attention — the mechanism behind Transformers and every modern LLM — was born.
• It is a perfect case study in research: identify a bottleneck, design a mechanism that removes it, measure the gain.
• The idea generalizes far beyond translation: “let the model choose what to look at” shows up in vision, speech, and multimodal models.

A Quick Example

Imagine translating a 50-word German sentence into English. A standard seq2seq model forces you to read the whole sentence once, memorize it as a single fixed-size summary, then write the translation purely from memory. A human translator does not work that way — they glance back at the relevant source words while writing each output word. Attention gives the model that ability to glance back.

The Bottleneck Problem

In standard seq2seq models, the encoder compresses the entire input into a fixed-size context vector $c$:

$$c = f(x_1, x_2, ..., x_T)$$

This becomes a bottleneck for long sequences — all information must squeeze through a single vector, no matter how long the input is.

The Attention Solution

Instead of a single context vector, attention computes a different context for each output step:

$$c_i = \sum_{j=1}^{T} \alpha_{ij} h_j$$

where $\alpha_{ij}$ is the attention weight that output position $i$ places on input position $j$, and $h_j$ is the encoder state for input word $j$.

The important idea is: each output word gets its own weighted mix of all input words, with the weights chosen by the model.

Computing Attention Weights

How does the model choose the weights? An alignment model scores how relevant each input position is, then a softmax turns the scores into weights that sum to 1:

$$e_{ij} = a(s_{i-1}, h_j)$$ $$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T} \exp(e_{ik})}$$

The alignment function $a$ (often a small neural network) scores how well input $j$ and output $i$ match. You do not need to memorize the formulas — the recipe is: score every input word, softmax the scores, use them as mixing weights.

Interactive Demo

Watch attention align source and target words during translation:

Attention Alignment

←Step 1/5→

Je

suis

étudiant

.

I

am

a

student

.

Source (French) → Target (English)

Current Output: "I"Show math

Je: 82%

suis: 10%

étudiant: 7%

.: 1%

With Attention

Model learns which source words are relevant for each output word

Without Attention

Entire source compressed into fixed-size vector—bottleneck

The Alignment Model

Bahdanau attention uses an additive alignment function:

$$e_{ij} = v^T \tanh(W_s s_{i-1} + W_h h_j)$$

This learnable function discovers which source words are relevant for generating each target word — no hand-built dictionary or word alignments required.

Why This Changed Everything

1. No bottleneck: each output step accesses all encoder states
2. Interpretable: attention weights show what the model “looks at”
3. Handles long sequences: performance doesn’t degrade with length
4. Generalizable: the mechanism applies far beyond translation

Results

Model	BLEU Score
RNNsearch (no attention)	26.75
RNNsearch-50 (attention)	28.45
Phrase-based SMT	33.30

Attention closed the gap with statistical MT and enabled future improvements.

Legacy

This paper’s attention mechanism became:

• The foundation of Transformers (self-attention) — see Transformer
• Used in image captioning (visual attention)
• Core to speech recognition (CTC-attention)
• Essential for modern LLMs

Common Confusion

• Bahdanau attention vs. self-attention. Here, the decoder attends to the encoder’s states (cross-attention between two sequences). In Transformers, tokens in one sequence attend to each other (self-attention).
• Attention vs. alignment. “Alignment” is the translation-specific framing (which source word maps to which target word); “attention” is the general mechanism computing it.
• This paper vs. Attention Is All You Need. This 2014 paper introduced attention as an add-on to RNNs; the 2017 Transformer paper removed the RNN entirely and kept only attention.

Where To Go Next

• Read Sequence to Sequence for the bottlenecked architecture this paper improved
• Read Understanding LSTMs for the RNN machinery used underneath
• Read Transformer for the architecture built purely from attention
• Read Attention Is All You Need for the paper that took that final step

Key Papers

• Neural Machine Translation by Jointly Learning to Align and Translate — Bahdanau, Cho, Bengio (2014)
  https://arxiv.org/abs/1409.0473