Decoder-Only Models and Causal Masking · Transformers Under the Hood in Generative AI | Generative AI & LLMs

Three Families

Type	Sees	Examples	Good at
Encoder-only	all tokens, both directions	BERT, many embedding models	classification, embeddings, search
Encoder-decoder	encoder: all input; decoder: past output	T5, original Transformer	translation, summarisation
Decoder-only	only earlier tokens (causal)	GPT, Claude, Llama, Mistral, Gemma	text generation, chat, code

The Causal Mask

A decoder-only model is trained to predict the next token, so a token must not see the tokens after it. Before the softmax, scores for future positions are set to minus infinity, which makes their weight exactly 0.

Example

import math
def softmax(r):
    m = max(x for x in r if x != float("-inf"))
    e = [0.0 if x == float("-inf") else math.exp(x - m) for x in r]
    return [x / sum(e) for x in e]

tokens = ["Your", "refund", "is", "processing"]
scores = [[0.5, 0.1, 0.3, 0.9], [0.2, 0.8, 0.4, 0.1], [0.3, 0.9, 0.2, 0.6], [0.1, 0.7, 0.5, 0.4]]
masked = [[s if j <= i else float("-inf") for j, s in enumerate(row)] for i, row in enumerate(scores)]
for t, row in zip(tokens, masked):
    print(f"{t:>10} " + " ".join(f"{w:5.2f}" for w in softmax(row)))

Output

      Your  1.00  0.00  0.00  0.00
    refund  0.35  0.65  0.00  0.00
        is  0.27  0.49  0.24  0.00
processing  0.18  0.32  0.26  0.24

Generation and the KV Cache

prompt tokens ──► all layers ──► probabilities for next token ──► pick one (sampling)
      ▲                                                                   │
      └──────────── append the new token, repeat until a stop token ◄─────┘

During generation, the keys and values of earlier tokens do not change, so servers keep them in a KV cache and compute only the new token each step. That is why the first token takes longest (the whole prompt is processed) and later tokens come at a steady speed.