From Tokens to Embeddings

A model cannot read text. The tokenizer turns text into token IDs, and the first layer looks up each ID in an embedding table: one row of numbers per token in the vocabulary. Real models use vocabularies of 100,000+ tokens and vectors of thousands of numbers. We use 6 tokens and 4 numbers.

"refund my order" ──tokenizer──► [3, 1, 4] ──embedding table──► 3 vectors of size d ──► transformer layers

A transformer is the neural network design behind almost every modern LLM. It is a stack of identical layers; each layer takes one vector per token and returns an improved vector per token. Neural networks only do arithmetic, so the first job is to turn each token into numbers that arithmetic can work with.

Why not just use the token ID itself, like 3 for "refund"? Because the ID is an arbitrary label: 3 is not "closer" to 4 in meaning than to 1000. An embedding gives each token a vector of d_model numbers (the model width, 4096 in an 8B Llama), and training adjusts these numbers so that tokens used in similar ways end up with similar vectors. "refund" and "return" drift close together; "refund" and "monsoon" do not. Each number is a learned feature, though real features are not neatly human-readable.

Mathematically, the lookup is the same as multiplying a one-hot vector (all zeros, a single 1 at the token's ID) by the embedding matrix, which simply selects one row. That is why it is called an embedding table or matrix: vocabulary size × d_model numbers, learned like every other weight. For a 128,000-token vocabulary and width 4096, that is about 524 million parameters in this table alone.

import random
rng = random.Random(42)
VOCAB = ["<pad>", "my", "is", "refund", "order", "late"]
d_model = 4
# The embedding table is learned during training; here it is random
table = [[round(rng.uniform(-1, 1), 2) for _ in range(d_model)] for _ in VOCAB]

text = "refund my order"
ids = [VOCAB.index(w) for w in text.split()]
print("token ids:", ids)
for w, i in zip(text.split(), ids):
    print(f"{w:7} -> row {i}: {table[i]}")
print("table shape:", len(table), "x", d_model)

token ids: [3, 1, 4]
refund  -> row 3: [-0.95, -0.6, 0.3, 0.09]
my      -> row 1: [0.47, 0.35, 0.78, -0.83]
order   -> row 4: [-0.56, 0.18, 0.62, -0.99]
table shape: 6 x 4

Each word becomes its ID, and each ID selects one row of the 6 × 4 table. The numbers here are random because nothing has been trained; in a real model they are the result of training on trillions of tokens. These vectors are only the starting point: the layers above will change them so that each token's vector also reflects its context.

Misconception: "the embedding of a word is its meaning". The input embedding of "bank" is the same in "river bank" and "bank account"; it cannot know which one is meant. The difference appears only after attention layers mix in the surrounding tokens, producing contextual vectors (see Self-Attention Step by Step).

• Tokens → IDs → rows of a learned embedding table (vocabulary × d_model).
• Similar usage leads to similar vectors after training.
• Input embeddings are context-free; layers make them contextual.
• Token embeddings inside an LLM are different from sentence embeddings in a vector DB.