Embeddings — The Simplest Way to Turn Meaning Into Coordinates

In one line

Embedding is turning text into vectors so that they can be compared by distance. Even with TF-IDF alone, without a neural network, you can observe the whole skeleton, namely vectorize → normalize → dot product, and all the ways that skeleton breaks down.

Why this was needed

Say two reports come in about a customer-support search box. One is that searching for "payment is slow" does not bring up the "handling payment delays" document, and the other is that searching for the error code E4012 brings up an unrelated document first. The first is a failure caused by different surface characters, and the second is a failure caused by fetching something that only means something similar when the characters had to match exactly. To tell the two failures apart and fix them, you first have to know how "similar" was defined as a number.

Once text is turned into vectors, you can compute similarity with a dot product or an angle. The simplest vector is a list of word occurrence counts. The problem is that common words dominate. A word that appears in every document is no help in telling documents apart, yet its value is the largest.

TF-IDF tackles this problem head on. It multiplies a word's frequency within a document (TF) by the inverse of its document frequency (IDF). A word that appears evenly across many documents has a low IDF and is pushed down, and a word that appears only in particular documents has a high IDF and is emphasized.

How it works

The smoothed IDF is usually defined like this. It is the same formula that scikit-learn's TfidfTransformer uses by default.

idf(t) = ln((1 + N) / (1 + df(t))) + 1

Adding 1 to the denominator and the numerator is like pretending there is one extra document that contains every term exactly once, so that the division does not break even for a term whose df is 0. Adding 1 at the end keeps the weight of a term that appears in every document from becoming 0 and vanishing entirely. The numbers give you a feel for it. With 30 documents, a term in only one document is about 3.741, a term in three documents about 3.048, a term in ten documents about 2.036, and a term in every document exactly 1. A rare term is treated as three to four times heavier than a common one.

After building the vectors, you do L2 normalization. If you divide each row by its own length so that the length becomes 1, the dot product of two vectors becomes exactly the cosine similarity, and the effect of document length disappears. Without normalization, a long document has a large dot product with almost every query simply because it has many words.

The hashing trick assigns each term to one of a fixed number of buckets with a hash function instead of maintaining a vocabulary dictionary. However much the vocabulary grows, the dimension stays fixed and you do not have to carry a dictionary around. The price is twofold. If different terms land in the same bucket they cannot be told apart, and because a hash is one-way you cannot recover the original term from a bucket number. scikit-learn's HashingVectorizer uses 2 to the power of 20 buckets by default and alternates signs according to the hash value, so collisions do not pile up on one side but cancel each other. If you make the bucket count small, such as 1024, collisions become numerous enough to see, and the price shows up as nearest neighbors changing.

A neural embedding has the same skeleton. The only difference is how the vector is made. Instead of word counts, a trained model turns a sentence into a dense vector of several hundred dimensions, so "slow" and "high latency" end up close. The rest of the procedure, normalizing and comparing with a dot product, stays the same.

What goes wrong in the field

Forgetting to normalize. The symptom is that a few particular documents appear near the top for most unrelated queries. They are usually the longest documents. If you see the same document id repeating in the result list, check the row lengths first.

The query becomes a zero vector. If all the words in the query are outside the index vocabulary, the query vector is all zeros. Dividing by its length here divides 0 by 0, so every value becomes nan, and numpy leaves a single warning line and carries on with the computation. If you sort with scores that contain nan, the order becomes meaningless, and no error is raised. Filter out a query with length 0 before normalization and return "no results".

The index and the query follow different rules. If you lowercased at indexing time but not at query time, or recomputed the IDF every time a query came in, the two vectors live in different spaces. Scores do come out, so it is hard to notice. With neural embeddings this problem is bigger. If the model changes, vectors can no longer be compared, so you must re-index everything, and some models are trained to put different prefixes on queries and documents (for example query: and passage: in the E5 family), so leaving them out silently lowers quality.

The hash changes from run to run. Python's built-in hash() mixes in a random value that differs per process for strings. If you decide buckets with it, yesterday's index and today's incoming queries use different buckets. You must use a reproducible hash (hashlib.md5 and so on).

Carrying a score threshold over unchanged. A TF-IDF cosine of 0.3 and a neural-embedding cosine of 0.3 do not mean the same thing. The score distribution differs from model to model, so when you change the model or the weighting scheme, re-decide the threshold with an evaluation set.

How to check

Once you have built the matrix, check the invariants before you trust the computation.

import numpy as np

X = np.load("tfidf.npy")
norms = np.linalg.norm(X, axis=1)
print(np.allclose(norms, 1.0))        # every row has length 1
print(np.isnan(X).any())               # no nan anywhere
S = X @ X.T
print(np.allclose(np.diag(S), 1.0))    # self-similarity is 1
np.fill_diagonal(S, -1)
nn = S.argmax(axis=1)
print(np.bincount(nn).max())           # one document as everyone's neighbour?

The last line is especially useful. Collect each document's nearest neighbor and count how many times one document becomes "someone's closest neighbor". If one document is picked unusually often, it is a sign that there is a problem with normalization or weighting.

You measure the loss from the hashing trick with two numbers. Comparing the number of distinct terms with the number of buckets actually used shows how many collisions there are, and counting how many nearest neighbors agree between the exact matrix and the hashed matrix shows how much those collisions changed the search results. And when you check search results, do not look only at the score; look at why that document came up. If you print the terms the query and the document share and their weights, the cause usually becomes obvious at a glance.

What you will do in the next lab

The module has two labs. The next one implements the BPE tokenizer from the previous reading from the ground up, and the one after it works out this reading's content by hand. You tokenize 30 documents and build the document frequency, IDF and TF-IDF matrix yourself with numpy, run nearest-neighbor and query search with cosine similarity, and finally build a 1024-dimensional matrix with the hashing trick and measure how much the nearest neighbors change.