Hashed Bag-of-Words Embeddings

Before neural embeddings, search used bag-of-words vectors: count each word. The hashing trick maps each word to one of a fixed number of slots with a hash function, so the vector size stays fixed (here 64) no matter how many words exist. It captures shared words, not meaning, but it uses exactly the same vector maths as real embeddings.

A bag-of-words vector ignores word order and grammar and just records which words appear and how often, like emptying a sentence into a bag and counting what is inside. A plain bag-of-words needs one slot per word in the vocabulary, which means tens of thousands of slots and a vocabulary that must be built in advance. The hashing trick avoids both problems: a hash function turns any word into a big number, and number % 64 picks one of 64 slots. New words need no vocabulary update; they simply land in some slot.

Building one yourself shows the whole embedding pipeline (text in, fixed-length normalized vector out, cosine to compare) without any library or model download. It also makes the weakness of word matching visible, which is the best way to understand why neural embeddings were such a big step.

1. Tokenize: lower-case the text and split it into words with a regular expression.
2. Remove stop words such as "the", "is", "my", which appear everywhere and carry little meaning.
3. Hash each word with MD5 (stable across runs) and take the remainder modulo 64 to choose a slot.
4. Add ±1 to that slot. Another bit of the hash picks the sign, so when two different words collide in one slot they tend to cancel rather than pile up.
5. Normalize the vector to length 1 so that the dot product is the cosine.

import hashlib, math, re

DIM = 64
STOP = {"the", "is", "a", "my", "i", "to", "of", "for", "in", "on", "and", "it"}

def embed(text):
    v = [0.0] * DIM
    for word in re.findall(r"[a-z0-9]+", text.lower()):
        if word in STOP:
            continue
        h = int(hashlib.md5(word.encode()).hexdigest(), 16)
        v[h % DIM] += 1.0 if (h >> 8) % 2 == 0 else -1.0     # signed hashing reduces collisions' effect
    n = math.sqrt(sum(x * x for x in v)) or 1.0
    return [x / n for x in v]

def cos(a, b):
    return sum(x * y for x, y in zip(a, b))                  # vectors are already normalized

q = embed("refund not received for cancelled order")
for text in ["When will I get my refund for a cancelled order?",
             "Order cancelled but refund not received yet",
             "Track my delivery",
             "Where is my money back?"]:
    print(f"{cos(q, embed(text)):.3f}  {text}")
print("non-zero slots in query vector:", sum(1 for x in q if x))

0.548  When will I get my refund for a cancelled order?
0.894  Order cancelled but refund not received yet
0.000  Track my delivery
0.000  Where is my money back?
non-zero slots in query vector: 5

The last sentence means the same as the query but scores 0, because it shares no words. That is exactly the gap a neural embedding model fills: it would put "money back" close to "refund".

The second sentence (0.894) wins because it contains all five content words of the query (plus "but" and "yet"); the first scores 0.548 because it shares only three (refund, cancelled, order). "Track my delivery" shares nothing and scores 0.

• Bag-of-words counts words and ignores order; the hashing trick keeps the vector size fixed.
• It uses the same normalize-and-cosine maths as neural embeddings.
• It matches shared words, not meaning, so synonyms score 0.
• Keyword methods still win on exact IDs and codes; hybrid search uses both.