Careeroza

Next Token Prediction

One Token at a Time

An LLM does one thing again and again: given the text so far, it gives a probability for every possible next token, picks one, adds it to the text, and repeats. A whole essay is just this loop run a few hundred times.

"Your order is"            → model → { "on": 0.41, "delayed": 0.22, "delivered": 0.20, ... }
"Your order is on"         → model → { "the": 0.93, "its": 0.05, ... }
"Your order is on the"     → model → { "way": 0.88, "route": 0.06, ... }
"Your order is on the way" → model → { ".": 0.71, "and": 0.15, ... }

The Loop in Code

Example

Here a small hard-coded probability table is a stand-in for the model. The loop always takes the most likely token (called greedy decoding):

# stand-in for a model: next-token probabilities for a given last token
table = {
    "<start>":   {"Your": 0.6, "The": 0.4},
    "Your":      {"order": 0.7, "refund": 0.3},
    "order":     {"is": 0.8, "was": 0.2},
    "is":        {"on": 0.41, "delayed": 0.22, "delivered": 0.20, "cancelled": 0.17},
    "on":        {"the": 0.93, "its": 0.07},
    "the":       {"way": 0.88, "route": 0.12},
    "way":       {".": 0.71, "and": 0.29},
}

tokens = ["<start>"]
while tokens[-1] in table and len(tokens) < 10:
    probs = table[tokens[-1]]
    best = max(probs, key=probs.get)
    print(f"after {tokens[-1]!r:10} -> pick {best!r} (p={probs[best]})")
    tokens.append(best)

print("\nGenerated:", " ".join(tokens[1:]))

Output

after '<start>'  -> pick 'Your' (p=0.6)
after 'Your'     -> pick 'order' (p=0.7)
after 'order'    -> pick 'is' (p=0.8)
after 'is'       -> pick 'on' (p=0.41)
after 'on'       -> pick 'the' (p=0.93)
after 'the'      -> pick 'way' (p=0.88)
after 'way'      -> pick '.' (p=0.71)

Generated: Your order is on the way .

Note: A real LLM looks at all previous tokens (not just the last one) and chooses from about 100,000+ possible tokens. But the loop is exactly the same.