Sequence to Sequence Learning

Encoder-decoder architecture for mapping sequences to sequences

architecture deep-learning nlp encoder-decoder seq2seq rnn

Updated

Contents
  1. Why Students Should Care
  2. The Problem
  3. The Solution: Encoder-Decoder
  4. Interactive Demo
  5. The Architecture
  6. Encoder (LSTM)
  7. Decoder (LSTM)
  8. Key Innovations
  9. 1. Reversing the Input Sequence
  10. 2. Deep LSTMs
  11. 3. Beam Search Decoding
  12. Training
  13. The Bottleneck Problem
  14. From Seq2Seq to Attention
  15. Applications
  16. Evolution
  17. Why It Mattered
  18. Code Example
  19. Common Confusion
  20. Where To Go Next
  21. Key Papers

Sequence-to-Sequence (Seq2Seq) is a neural architecture that turns one sequence into another: read the input with an encoder, then write the output with a decoder. Introduced by Sutskever et al. at Google in 2014, it enabled neural machine translation and became the foundation for modern language models.

This page assumes basic familiarity with recurrent networks — read Understanding LSTMs first if needed. Seq2Seq’s main weakness led directly to Bahdanau Attention and then the Transformer.

Why Students Should Care

  • The encoder-decoder split introduced here is still the mental model behind translation, summarization, and chat systems.
  • Seq2Seq is the first step in the chain that leads to attention and Transformers — you cannot appreciate what attention fixed without knowing this design.
  • Practical ideas born here — teacher forcing, beam search — are still used to train and run today’s language models.

The Problem

Traditional neural networks require fixed-size inputs and outputs. But many tasks have variable lengths on both sides:

  • Translation: “Hello” (1 word) → “Bonjour” (1 word)
  • Translation: “How are you?” (3 words) → “Comment allez-vous?” (2 words)

There is no fixed slot count you can design a plain feed-forward network around. Seq2Seq needed a way to map any-length input to any-length output.

The Solution: Encoder-Decoder

Split the problem into two parts:

  1. Encoder: read the entire input sequence and compress it into a fixed-size vector (the “thought vector” or “context”)
  2. Decoder: generate the output sequence, one token at a time, starting from this context vector
Input→Encoderc→DecoderOutput\text{Input} \xrightarrow{\text{Encoder}} \mathbf{c} \xrightarrow{\text{Decoder}} \text{Output}

Think of it as: read the whole sentence, form a thought, then say the thought in the other language.

Interactive Demo

Watch how Seq2Seq encodes a sentence and decodes its translation:

Seq2Seq: Encoder-Decoder

1. Encode
2. Context
3. Decode
Encoder (LSTM)
The
→
h1
cat
→
h2
sat
→
h3
<EOS>
→
h4
Context
"Thought Vector"
Decoder (LSTM)
s1
→
<SOS>
s2
→
Le
s3
→
chat
s4
→
assis
s5
→
<EOS>
Press "Start" to see how Seq2Seq translates "The cat sat" to "Le chat assis".
Input Processing
Often reversed ("sat cat The") to reduce distance between corresponding words
The Bottleneck
Fixed-size context limits long sequences → Led to attention mechanisms

The Architecture

Encoder (LSTM)

The encoder is an LSTM that processes the input sequence x1,x2,...,xTx_1, x_2, ..., x_T one token at a time, updating its hidden state:

ht=LSTM(xt,ht−1)h_t = \text{LSTM}(x_t, h_{t-1})

The final hidden state hTh_T becomes the context vector c\mathbf{c} — a summary of everything it read.

Decoder (LSTM)

The decoder is another LSTM that generates the output sequence y1,y2,...,yT′y_1, y_2, ..., y_{T'}, feeding each generated word back in as input for the next step:

st=LSTM(yt−1,st−1)s_t = \text{LSTM}(y_{t-1}, s_{t-1}) P(yt∣y<t,c)=softmax(Ws⋅st)P(y_t | y_{<t}, \mathbf{c}) = \text{softmax}(W_s \cdot s_t)

The decoder is initialized with s0=cs_0 = \mathbf{c} — the “thought” is its starting point.

Key Innovations

1. Reversing the Input Sequence

Reversing the source sentence improved translation significantly:

Original: "A B C" → "X Y Z"
Reversed: "C B A" → "X Y Z"

Why it helps: the first source word ends up right next to the first target word, so the RNN has a shorter path between corresponding words at the start of the sentence.

2. Deep LSTMs

Using 4-layer LSTMs significantly outperformed shallow networks:

DepthBLEU Score
1 layer25.9
2 layers29.6
4 layers34.8

3. Beam Search Decoding

Greedy decoding (taking the single most likely word at each step) can lock in an early mistake. Beam search instead keeps the top-k candidate sequences at every step:

Beam size = 3:
Step 1: [The, A, This]
Step 2: [The cat, The dog, A cat, ...]
...select top 3 sequences by total probability

Training

Training uses teacher forcing: at each step, feed the decoder the ground-truth previous token yt−1∗y_{t-1}^* rather than the model’s own (possibly wrong) prediction:

L=−∑t=1T′log⁡P(yt∗∣y1∗,...,yt−1∗,c)\mathcal{L} = -\sum_{t=1}^{T'} \log P(y_t^* | y_1^*, ..., y_{t-1}^*, \mathbf{c})

The takeaway: during training the model always sees the correct history, which keeps learning stable; at inference it must rely on its own outputs.

The Bottleneck Problem

The fixed-size context vector c\mathbf{c} must encode the entire input, and that is the design’s Achilles’ heel:

  • Works well for short sequences
  • Degrades for long sequences (information gets compressed away)
  • Led directly to the invention of attention mechanisms

From Seq2Seq to Attention

The limitation of a single context vector motivated Bahdanau attention (2014):

ct=∑iαtihi\mathbf{c}_t = \sum_i \alpha_{ti} h_i

Instead of one fixed summary, each decoder step gets its own weighted mix of all encoder states — the decoder can look back at whichever input words it needs. This solved the bottleneck.

Applications

Seq2Seq enabled a huge range of “sequence in, sequence out” tasks:

TaskInputOutput
Translation”Hello world""Bonjour le monde”
SummarizationLong articleShort summary
DialogueUser messageResponse
Code generationDescriptionCode
Speech recognitionAudio featuresText

Evolution

YearModelInnovation
2014Seq2SeqEncoder-decoder RNNs
2014BahdanauAttention mechanism
2015LuongSimplified attention variants
2017TransformerSelf-attention, no recurrence
2018+BERT, GPTPre-trained transformers

Why It Mattered

Seq2Seq established:

  • End-to-end learning: no hand-crafted features or alignment
  • Encoder-decoder paradigm: used by transformers today
  • Variable-length I/O: fundamental for language tasks
  • Transfer learning path: pre-trained encoders/decoders

Code Example

A minimal Seq2Seq loop with teacher forcing:

class Seq2Seq(nn.Module):
    def __init__(self, encoder, decoder):
        self.encoder = encoder
        self.decoder = decoder

    def forward(self, src, trg):
        # Encode: get context from final hidden state
        _, (hidden, cell) = self.encoder(src)

        # Decode: generate output sequence
        outputs = []
        input = trg[0]  # <SOS> token
        for t in range(1, len(trg)):
            output, (hidden, cell) = self.decoder(input, hidden, cell)
            outputs.append(output)
            input = trg[t]  # Teacher forcing

        return torch.stack(outputs)

Common Confusion

  • Seq2Seq is an architecture pattern, not one specific model. The 2014 paper used LSTMs, but “encoder-decoder” describes Transformers too.
  • The context vector is not attention. Vanilla Seq2Seq has exactly one fixed summary vector; attention (a later addition) replaces it with a per-step weighted mix.
  • Teacher forcing happens only during training. At inference time the decoder must consume its own predictions.

Where To Go Next

  • Read Understanding LSTMs for the recurrent cells inside encoder and decoder
  • Read Bahdanau Attention for the fix to the bottleneck problem
  • Read Transformer for the architecture that replaced recurrence entirely
  • Read GPT for modern decoder-only sequence generation

Key Papers

Found an error or want to contribute? Edit this page on GitHub

↑↓ to navigate ↵ to open esc to close