Pointer Networks

Pointer Networks fix a blind spot in sequence-to-sequence models: a standard seq2seq model can only output words from a fixed, pre-decided vocabulary. A Pointer Network instead outputs pointers back into its own input — “the answer is element 3, then element 7, then element 1” — so the set of possible outputs grows and shrinks with the input.

This page builds on Seq2Seq and Bahdanau Attention. If either is unfamiliar, read those first.

Why Students Should Care

• Pointer Networks introduced the idea of attention as output, not just as an internal routing mechanism — a small twist with a long legacy.
• They are the standard tool when a neural network must select items from its input: copying rare words, choosing spans, ordering elements.
• Their “copy from the input” idea lives on in summarization models, extractive QA, and the copy mechanisms inside modern text generators.

The Problem, Concretely

Suppose you want a neural network to compute the convex hull of a set of points: given 10 points, output the ones on the outer boundary, in order. Now give it 50 points. The “vocabulary” of possible answers just changed — it is the input points themselves, and there are 50 of them now instead of 10.

A standard seq2seq model produces outputs from a fixed dictionary:

$$P(y_i | y_1, ..., y_{i-1}, x) = \text{softmax}(W \cdot h_i)$$

The matrix $W$ has one row per vocabulary item, fixed at training time. There is no way for this output layer to say “point number 37” when the model was built for at most 10 points. For convex hull, the output is indices into the input — vocabulary size equals input size, and that changes every example.

The Pointer Mechanism

The fix is elegant: you already have a mechanism that scores every input position at every decoding step — attention. Instead of using attention to build a context vector and then predicting from a fixed vocabulary, use the attention weights themselves as the output distribution:

$$u_j^i = v^T \tanh(W_1 e_j + W_2 d_i)$$ $$P(y_i | y_1, ..., y_{i-1}, x) = \text{softmax}(u^i)$$

Here $u_j^i$ is a score for “how much should output step $i$ point at input position $j$”. Softmax over these scores gives a probability distribution over input positions — the attention weights $\alpha_{ij}$ directly become the output probabilities.

The takeaway: a Pointer Network is a seq2seq model whose output layer is an attention mechanism over the input. Nothing else changes.

Architecture

1. Encoder: Process input sequence $(x_1, ..., x_n)$ to get representations $(e_1, ..., e_n)$
2. Decoder: At each step, produce hidden state $d_i$
3. Pointer: Compute attention over encoder states, output highest-attention position

Interactive Demo

Watch a Pointer Network solve convex hull by pointing to input coordinates:

Pointer Networks

New PointsFind Hull

Pointer Output

5

6

0

4

1

→

Output is a sequence of pointers to input positions

Variable Output Size

Output length depends on input—impossible with fixed vocabulary

Attention as Output

Attention weights become the output distribution over inputs

Applications

All of these share the same shape: the answer is a selection or ordering of the input elements.

Convex Hull

Given points, output the subset forming the convex hull. Output size varies with input geometry.

Delaunay Triangulation

Given points, output triangles. Number of triangles depends on point configuration.

Traveling Salesman Problem

Approximate TSP by learning to output city visitation order.

Sorting

Learn to sort sequences by outputting indices in sorted order.

Why Pointers Matter

Standard Seq2Seq	Pointer Network
Fixed vocabulary	Input-dependent vocabulary
Can’t reference input	Output references input
Fixed output size	Variable output size

Key Equations

For reference, the full model. Encoder (bidirectional LSTM):

$$e_j = [\overrightarrow{h}_j; \overleftarrow{h}_j]$$

Decoder with attention:

$$d_i = \text{LSTM}(d_{i-1}, [y_{i-1}; c_{i-1}])$$

Pointer distribution:

$$P(y_i = j) = \frac{\exp(u_j^i)}{\sum_k \exp(u_k^i)}$$

You do not need to memorize these. They are a standard attention-based seq2seq model — the only novelty is that the last softmax ranges over input positions instead of a vocabulary.

Influence

Pointer Networks introduced the idea of using attention as output, which influenced:

• Copy mechanisms in text generation
• Pointer-generator networks for summarization
• Graph neural network outputs

Common Confusion

• Pointer Networks vs. attention: ordinary attention (Bahdanau Attention) is used internally to build a context vector; Pointer Networks repurpose the attention distribution as the final output.
• Pointing vs. generating: a pure Pointer Network can only select input elements — it cannot produce a token that is not in the input. Pointer-generator hybrids mix both abilities.
• Pointer Networks vs. Order Matters: same lead author, related problems. Pointer Networks solve variable output vocabularies; Order Matters studies how element ordering affects models operating on sets (and uses pointers in its Write phase).

Where To Go Next

• Read Order Matters for the follow-up work on set-structured inputs and outputs.
• Read Bahdanau Attention for the attention mechanism being repurposed here.
• Read Seq2Seq for the encoder-decoder backbone.
• Read Attention Is All You Need to see where attention-centric architectures went next.

Key Paper

• Pointer Networks — Vinyals, Fortunato, Jaitly (2015)
  https://arxiv.org/abs/1506.03134