Word2Vec: Word Embeddings

Word2Vec is a method for turning words into vectors of numbers so that words with similar meanings end up close together. Introduced by Mikolov et al. at Google in 2013, it revolutionized NLP — the famous equation “king - man + woman = queen” showed that these vectors capture meaningful relationships.

This page is a good starting point for NLP — no Transformer knowledge needed. For what replaced static word vectors, see BERT.

Why Students Should Care

Embeddings are everywhere. Every modern language model starts by mapping tokens to vectors — Word2Vec is where that idea took off.
It is the cleanest demonstration that meaning can emerge from prediction: train on a simple guessing game, get semantic structure for free.
The same trick was later applied to graphs, products, users, and more — “X2Vec” became a pattern.

A Quick Example

How would you represent the word “cat” as numbers? The naive answer is a one-hot vector: a huge vector of zeros with a single 1 in the “cat” slot. But then “cat” and “dog” are exactly as different from each other as “cat” and “refrigerator” — every pair of one-hot vectors is equally far apart. There is no notion of similarity at all.

Word2Vec’s answer: learn a short, dense vector for each word from how it is used in text. Words that appear in similar contexts (“cat” and “dog” both appear near “pet”, “fur”, “vet”) end up with similar vectors.

The Key Insight

Instead of one-hot vectors (sparse, no similarity), learn dense embeddings where:

Similar words are close: vec(“cat”) ≈ vec(“dog”)
Relationships are linear: vec(“king”) - vec(“man”) + vec(“woman”) ≈ vec(“queen”)

Two Architectures

Both variants train embeddings by playing a prediction game over a sliding window of text — they just play it in opposite directions.

Skip-gram

Given a word, predict the surrounding context words:

P(w_{context} | w_{center}) = \frac{\exp(v_{w_c}^T v_{w_t})}{\sum_{w \in V} \exp(v_w^T v_{w_t})}

Objective: maximize the probability of the context words given the center word. The dot product $v_{w_c}^T v_{w_t}$ measures how compatible two word vectors are — the softmax turns those scores into probabilities.

CBOW (Continuous Bag of Words)

Given the context words, predict the center word:

P(w_{center} | w_{context}) = \frac{\exp(v_{w_t}^T \bar{v}_{context})}{\sum_{w \in V} \exp(v_w^T \bar{v}_{context})}

where $\bar{v}_{context}$ is the average of the context word vectors.

You do not need to memorize either formula. The important idea is: words are pushed toward vectors that make their real contexts predictable.

Interactive Demo

Explore word embeddings and vector arithmetic:

Word2Vec: Word Embeddings

royalty

gender

animal

place

Vector Analogies:

How it works:

The relationship between king and man is captured as a vector. Adding this same vector to woman gives us queen.

vec(king) - vec(man) + vec(woman) ≈ vec(queen)

Skip-gram Training

Context:thequickbrownfoxjumps

Given "brown", predict: "the", "quick", "fox", "jumps"

Training with Negative Sampling

There is a practical problem: the softmax above sums over the entire vocabulary (hundreds of thousands of words) for every training example. That is far too expensive. Negative sampling approximates it:

\log \sigma(v_{w_O}^T v_{w_I}) + \sum_{i=1}^{k} \mathbb{E}_{w_i \sim P_n(w)} [\log \sigma(-v_{w_i}^T v_{w_I})]

Instead of normalizing over all words, contrast each positive pair against just $k$ random “negative” words. In plain English: make the true context word score high, and a handful of random words score low.

The Training Process

Sentence: "The quick brown fox jumps"
Window size: 2

For center word "brown":
  Positive pairs: (brown, quick), (brown, fox)
  Negative samples: (brown, computer), (brown, elephant), ...

Objective: Push positive pairs together, negative pairs apart

Why Linear Relationships?

Why does vector arithmetic like king - man + woman work at all? The optimization objective creates a structure where:

\vec{v}_{king} - \vec{v}_{man} \approx \vec{v}_{queen} - \vec{v}_{woman}

This emerges because words appearing in similar contexts get similar vectors, and gender/royalty patterns are consistent across the corpus. Nobody programmed this in — it is a side effect of the prediction objective.

Hyperparameters

Parameter	Typical Value	Effect
Embedding dimension	100-300	Higher = more capacity
Window size	5-10	Larger = more syntactic
Negative samples	5-20	More = better for rare words
Min word count	5	Filter rare words
Subsampling	1e-3 to 1e-5	Downsample frequent words

Word2Vec vs. Later Methods

Method	Year	Key Difference
Word2Vec	2013	Static embeddings, one vector per word
GloVe	2014	Global co-occurrence statistics
FastText	2016	Subword embeddings (handles OOV)
ELMo	2018	Context-dependent embeddings
BERT	2018	Deep bidirectional context

Limitations

One vector per word: “bank” (river) = “bank” (financial) — the vector must average both meanings
No morphology: “run”, “running”, “runs” are unrelated
Fixed vocabulary: out-of-vocabulary words get no embedding
Shallow context: just neighboring words, not deep semantics

These limitations led to contextual embeddings (ELMo, BERT), where a word’s vector changes depending on the sentence it appears in.

Historical Impact

Word2Vec:

Made NLP research accessible (fast to train)
Introduced the embedding paradigm
Enabled transfer learning in NLP
Demonstrated emergent structure in learned representations
Inspired similar approaches for graphs, products, etc.

Common Confusion

Word2Vec embeddings are static; BERT embeddings are contextual. Word2Vec gives each word one fixed vector; BERT computes a fresh vector for each occurrence based on the sentence.
Skip-gram and CBOW are two training setups, not two models. Both produce the same kind of word vectors — they differ only in which side of the window is predicted.
Word2Vec is not a language model. It learns representations from a prediction game, but it is not designed to generate text.

Where To Go Next

Read BERT for contextual embeddings that fixed Word2Vec’s one-vector-per-word limit
Read Transformer for the architecture behind contextual embeddings
Read Seq2Seq for how word vectors feed into sequence models
Read Pre-training for the broader idea of learning from unlabeled data first

Key Papers

Efficient Estimation of Word Representations in Vector Space – Mikolov et al., 2013
https://arxiv.org/abs/1301.3781
Distributed Representations of Words and Phrases and their Compositionality – Mikolov et al., 2013
https://arxiv.org/abs/1310.4546
GloVe: Global Vectors for Word Representation – Pennington et al., 2014
https://nlp.stanford.edu/pubs/glove.pdf
Enriching Word Vectors with Subword Information (FastText) – Bojanowski et al., 2016
https://arxiv.org/abs/1607.04606