Careeroza

From Scores to Probabilities

Logits and Softmax

Inside the model, each possible next token gets a raw score called a logit. The softmax function turns scores into probabilities that add up to 1: p_i = exp(s_i) / Σ exp(s_j). Bigger scores get much bigger probabilities because of the exponential.

Softmax in Plain Python

Example

Scores for the next word after "Your order is":

import math

def softmax(scores):
    m = max(scores)                       # subtract max for numerical safety
    exps = [math.exp(s - m) for s in scores]
    total = sum(exps)
    return [e / total for e in exps]

tokens = ["on", "delayed", "delivered", "cancelled", "banana"]
logits = [3.1, 2.5, 2.4, 2.2, -1.0]

for t, pr in zip(tokens, softmax(logits)):
    print(f"{t:10} logit={logits[tokens.index(t)]:5}  p={pr:.3f}")
print("sum =", round(sum(softmax(logits)), 6))

Output

on         logit=  3.1  p=0.405
delayed    logit=  2.5  p=0.222
delivered  logit=  2.4  p=0.201
cancelled  logit=  2.2  p=0.165
banana     logit= -1.0  p=0.007
sum = 1.0