What is a Token
Tokens, Not Words
Models do not read characters or words. They read tokens: chunks of text such as order, ing, the, ₹. In English one token is roughly 3–4 characters or about ¾ of a word. Everything is measured in tokens: the context limit, speed and price.
| Text | Possible tokens |
|---|---|
order | order (1 token, common word) |
unbelievably | un believ ably (3 tokens) |
SE-10423 | SE - 104 23 (IDs and numbers split a lot) |
Note: The exact split depends on the model's tokenizer. The splits above are illustrations; use the provider's token counter for real numbers.
Words, Characters and Bytes
Example
Modern tokenizers start from UTF-8 bytes. English letters are 1 byte each, but Hindi or Tamil letters are 3 bytes each, which is why Indian-language text usually needs more tokens (and costs more) than the same message in English:
texts = {
"English": "Your order is on the way",
"Hindi": "आपका ऑर्डर रास्ते में है",
"Tamil": "உங்கள் ஆர்டர் வழியில் உள்ளது",
}
print(f"{'lang':8} {'words':>5} {'chars':>5} {'bytes':>5}")
for lang, t in texts.items():
print(f"{lang:8} {len(t.split()):>5} {len(t):>5} {len(t.encode('utf-8')):>5}")Output
lang words chars bytes English 6 24 24 Hindi 5 24 64 Tamil 4 28 78
Tip: Rough rule for English: tokens ≈ characters / 4. For Indian languages, code and numbers, the ratio is worse, so always measure with the real tokenizer before estimating a budget.