Day 171 — Word Embeddings: Word2Vec and GloVe

• Understand why raw text is unusable by neural networks and what embeddings solve

• Explore the architecture of Word2Vec (Skip-gram and CBOW) and GloVe

• Implement embedding lookups, similarity search, and semantic arithmetic in Python

• Connect word vectors to production NLP systems at scale

Every search engine, recommendation system, and large language model you have ever used runs on one foundational idea: words can be represented as points in high-dimensional space, and words with similar meanings cluster near each other. Without this, a model cannot know that “bank” and “financial institution” refer to similar concepts, or that “Paris” relates to “France” the same way “Tokyo” relates to “Japan.”

Word embeddings are the bridge between human language and mathematical computation. They are the reason modern NLP works at all.

Neural networks process tensors — grids of floating point numbers. The letter “A” means nothing to a matrix multiplication. One-hot encoding is the naive fix: represent 50,000 vocabulary words as a 50,000-dimensional vector where only one position is 1. It works, but it is catastrophically wasteful and semantically blind. “Dog” and “canine” have zero overlap in one-hot space even though they mean the same thing.

Word2Vec, introduced by Google researchers in 2013, learns embeddings by training a shallow neural network on a deceptively simple task: given a word, predict its neighbors, or given neighbors, predict the center word.

Skip-gram — Takes a center word, predicts surrounding context words. Works well for infrequent words.

CBOW (Continuous Bag of Words) — Takes surrounding context words, predicts the center word. Faster, better for frequent words.

The key insight is that the network never actually cares about the prediction task. What matters is the weight matrix it builds while solving that task — a 300-dimensional vector per word that encodes semantic relationships the model learned purely from co-occurrence patterns in billions of text examples.

The “aha” moment. After training on billions of words, the geometry that emerges is astonishing. King minus man plus woman equals queen. Paris minus France plus Germany equals Berlin. These are not programmed rules. The geometry emerges entirely from statistics.