# Pointer Networks

> Neural architecture that outputs pointers to input positions, enabling variable-size outputs

- Type: Paper
- By: Vinyals, Fortunato & Jaitly
- Year: 2015
- Topics: Deep learning, Attention, Seq2seq, Combinatorial
- Page: https://aipedia.org/pointer-networks

## Resources

- [Pointer Networks](https://arxiv.org/abs/1506.03134): Paper, Vinyals, Fortunato & Jaitly, 2015

**Pointer Networks** fix a blind spot in sequence-to-sequence models: a standard seq2seq model can only output words from a fixed, pre-decided vocabulary. A Pointer Network instead outputs *pointers back into its own input* — "the answer is element 3, then element 7, then element 1" — so the set of possible outputs grows and shrinks with the input.

> This page builds on [Seq2Seq](https://aipedia.org/seq2seq) and [Bahdanau Attention](https://aipedia.org/bahdanau-attention). If either is unfamiliar, read those first.

## Why Students Should Care

- Pointer Networks introduced the idea of **attention as output**, not just as an internal routing mechanism — a small twist with a long legacy.
- They are the standard tool when a neural network must *select* items from its input: copying rare words, choosing spans, ordering elements.
- Their "copy from the input" idea lives on in summarization models, extractive QA, and the copy mechanisms inside modern text generators.

## The Problem, Concretely

Suppose you want a neural network to compute the **convex hull** of a set of points: given 10 points, output the ones on the outer boundary, in order. Now give it 50 points. The "vocabulary" of possible answers just changed — it is the input points themselves, and there are 50 of them now instead of 10.

A standard seq2seq model produces outputs from a fixed dictionary:

$$
P(y_i | y_1, ..., y_{i-1}, x) = \text{softmax}(W \cdot h_i)
$$

The matrix $W$ has one row per vocabulary item, fixed at training time. There is no way for this output layer to say "point number 37" when the model was built for at most 10 points. For convex hull, the output is indices into the input — vocabulary size equals input size, and that changes every example.

## The Pointer Mechanism

The fix is elegant: you already have a mechanism that scores every input position at every decoding step — **attention**. Instead of using attention to build a context vector and then predicting from a fixed vocabulary, use the attention weights themselves *as the output distribution*:

$$
u_j^i = v^T \tanh(W_1 e_j + W_2 d_i)
$$

$$
P(y_i | y_1, ..., y_{i-1}, x) = \text{softmax}(u^i)
$$

Here $u_j^i$ is a score for "how much should output step $i$ point at input position $j$". Softmax over these scores gives a probability distribution over input positions — the attention weights $\alpha_{ij}$ directly become the output probabilities.

The takeaway: **a Pointer Network is a seq2seq model whose output layer is an attention mechanism over the input.** Nothing else changes.

## Architecture

1. **Encoder**: Process input sequence $(x_1, ..., x_n)$ to get representations $(e_1, ..., e_n)$
2. **Decoder**: At each step, produce hidden state $d_i$
3. **Pointer**: Compute attention over encoder states, output highest-attention position

## Interactive Demo

Watch a Pointer Network solve convex hull by pointing to input coordinates:

*Interactive demo: https://aipedia.org/pointer-networks*

## Applications

All of these share the same shape: the answer is a selection or ordering of the input elements.

### Convex Hull

Given points, output the subset forming the convex hull. Output size varies with input geometry.

### Delaunay Triangulation

Given points, output triangles. Number of triangles depends on point configuration.

### Traveling Salesman Problem

Approximate TSP by learning to output city visitation order.

### Sorting

Learn to sort sequences by outputting indices in sorted order.

## Why Pointers Matter

| Standard Seq2Seq | Pointer Network |
|-----------------|-----------------|
| Fixed vocabulary | Input-dependent vocabulary |
| Can't reference input | Output references input |
| Fixed output size | Variable output size |

## Key Equations

For reference, the full model. Encoder (bidirectional LSTM):

$$
e_j = [\overrightarrow{h}_j; \overleftarrow{h}_j]
$$

Decoder with attention:

$$
d_i = \text{LSTM}(d_{i-1}, [y_{i-1}; c_{i-1}])
$$

Pointer distribution:

$$
P(y_i = j) = \frac{\exp(u_j^i)}{\sum_k \exp(u_k^i)}
$$

You do not need to memorize these. They are a standard attention-based seq2seq model — the only novelty is that the last softmax ranges over input positions instead of a vocabulary.

## Influence

Pointer Networks introduced the idea of using attention as output, which influenced:

- **Copy mechanisms** in text generation
- **Pointer-generator networks** for summarization
- **Graph neural network outputs**

## Common Confusion

- **Pointer Networks vs. attention**: ordinary attention ([Bahdanau Attention](https://aipedia.org/bahdanau-attention)) is used *internally* to build a context vector; Pointer Networks repurpose the attention distribution as the *final output*.
- **Pointing vs. generating**: a pure Pointer Network can only select input elements — it cannot produce a token that is not in the input. Pointer-generator hybrids mix both abilities.
- **Pointer Networks vs. [Order Matters](https://aipedia.org/order-matters)**: same lead author, related problems. Pointer Networks solve *variable output vocabularies*; Order Matters studies how *element ordering* affects models operating on sets (and uses pointers in its Write phase).

## Where To Go Next

- Read [Order Matters](https://aipedia.org/order-matters) for the follow-up work on set-structured inputs and outputs.
- Read [Bahdanau Attention](https://aipedia.org/bahdanau-attention) for the attention mechanism being repurposed here.
- Read [Seq2Seq](https://aipedia.org/seq2seq) for the encoder-decoder backbone.
- Read [Attention Is All You Need](https://aipedia.org/attention-is-all-you-need) to see where attention-centric architectures went next.