Careeroza

Positional Encoding

Why Order Needs Help

Self-attention looks at all tokens at once, so by itself it cannot tell "dog bites man" from "man bites dog". The model adds position information to each token vector. The original Transformer (2017) used fixed sine and cosine waves; most modern LLMs use RoPE (rotary position embeddings), which rotates the query and key vectors by an angle that depends on the position.

PE(pos, 2i)   = sin(pos / 10000^(2i / d))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d))

Example

Example

import math
def positional_encoding(pos, d):
    return [round(math.sin(pos / 10000 ** (i / d)) if i % 2 == 0 else math.cos(pos / 10000 ** ((i - 1) / d)), 3)
            for i in range(d)]

for pos in range(4):
    print(f"position {pos}: {positional_encoding(pos, 4)}")

token_vec = [-0.95, -0.6, 0.3, 0.09]           # embedding of "refund" from the previous page
pos_vec = positional_encoding(0, 4)
print("refund at position 0:", [round(a + b, 3) for a, b in zip(token_vec, pos_vec)])

Output

position 0: [0.0, 1.0, 0.0, 1.0]
position 1: [0.841, 0.54, 0.01, 1.0]
position 2: [0.909, -0.416, 0.02, 1.0]
position 3: [0.141, -0.99, 0.03, 1.0]
refund at position 0: [-0.95, 0.4, 0.3, 1.09]

Context Length

Position handling is one reason context windows have limits. Models are trained up to some length; techniques like RoPE scaling let them work on longer inputs, but quality on very long contexts must still be tested.