RAG

Retrieval-Augmented Generation: at question time, retrieve relevant chunks from your documents and put them in the prompt, so the model answers from them.

• Knowledge stays fresh: update the index, not the model.
• Answers can cite sources, which users and auditors can check.
• Access control is possible per document and per user.
• Much cheaper and faster to build than fine-tuning; fine-tuning is better for style and behaviour, not facts.

question → embed → vector/hybrid search → top-k chunks → prompt (docs + question) → LLM → answer with citations

• Split on structure first (headings, sections, paragraphs), then by size: typically 200-800 tokens.
• Add 10-20% overlap so facts at boundaries are not cut in half.
• Keep metadata with each chunk: document title, section, URL, date, access group.
• Small chunks match precisely but lose context; use parent-document retrieval or contextual retrieval to get both.
• Tune size and overlap on retrieval metrics (recall@k), not by guess.

1. Look at the retrieved chunks for failing questions. Was the right chunk retrieved? Measure recall@k.
2. If not: fix retrieval (hybrid BM25 + vectors, better chunking, query rewriting, re-ranking, metadata filters).
3. If yes but the answer is wrong: fix the prompt (answer only from docs, cite, say "I don't know"), reduce noisy chunks, or use a stronger model.
4. Check the source documents themselves: outdated or conflicting policies are common.
5. Add each failure to the golden set so it stays fixed.

Hybrid search runs keyword search (BM25) and vector search and merges the results, usually with reciprocal rank fusion: score = Σ 1/(k + rank), k ≈ 60. Keywords catch exact IDs and names; vectors catch meaning. A re-ranker (cross-encoder) then reads query and candidate together and re-scores the top 20-50, keeping the best 3-5 for the prompt. Together they are the most common upgrades over basic RAG.