LLM Fundamentals
What is a token, and why do tokens matter?
A token is the unit an LLM reads and writes: a word, part of a word, or a symbol. In English one token is roughly ¾ of a word; Hindi and other Indic scripts often need more tokens per word. Tokens matter because price, speed and the context limit are all counted in tokens, and output tokens cost several times more than input tokens.
Tip: For exact counts with Claude, use the token counting endpoint (client.messages.count_tokens(...)); other vendors' tokenizers give wrong numbers.
What is the context window?
The maximum number of tokens the model can handle in one request: system prompt + tools + conversation history + retrieved documents + the output. Anything beyond it must be cut, summarised or retrieved on demand. A bigger window does not mean the model uses every part equally well, so relevant context still beats more context.
What are temperature, top-k and top-p?
They control how the next token is sampled from the model's probability distribution.
- Temperature divides the logits before softmax: low = sharper, more predictable; high = flatter, more varied.
- Top-k keeps only the k most likely tokens.
- Top-p (nucleus) keeps the smallest set of tokens whose probabilities add up to p.
Note: On current Claude models (Opus 5.5, Sonnet 5.5, Haiku 5.5) these parameters are rejected by the API. You steer output with the prompt, examples, structured outputs and output_config={"effort": ...}. The concepts still matter for open models and interviews.
Why do LLMs hallucinate, and how do you reduce it?
The model is trained to produce likely-sounding text, not to check facts. When it lacks the information, a fluent wrong answer is often more likely than "I don't know".
- Ground answers in retrieved documents (RAG) and ask for citations.
- Explicitly allow "I don't know" and give an escalation path.
- Use tools for live data and calculations.
- Verify output after generation (citations, numbers, IDs) and measure a faithfulness metric in evals.
Explain self-attention in simple words.
Each token creates a query ("what am I looking for?"), a key ("what do I contain?") and a value ("what I pass on"). A token's query is compared with every key (dot product ÷ √d), softmax turns the scores into weights, and the output is the weighted sum of the values. So "it" in "the order was late, refund it" can pull information from "order". Multi-head attention runs several of these in parallel; decoder-only models add a causal mask so tokens only see earlier tokens.
Attention(Q, K, V) = softmax(Q · Kᵀ / √d_k) · V