Evaluation and Safety

• Build a golden dataset of real questions with expected facts, expected sources and tags (including edge cases and attacks).
• Measure each stage: retrieval (recall@k, MRR), answers (exact match / must-include facts / LLM judge for faithfulness and helpfulness), tools (right tool, right arguments).
• Run it in CI on every prompt or model change; block regressions.
• In production: thumbs up/down, escalation rate, resolution rate, and weekly human review.

A strong model grades outputs against a rubric (for example "faithful: every claim is in the docs"). It scales human-like grading cheaply.

• Position bias and verbosity bias: swap orders, tell the judge length is not quality.
• It must be calibrated against human labels before you trust it.
• Prefer binary criteria with a short reason first, and structured output for the grade.

Text that tries to override your instructions: directly from the user ("ignore previous instructions...") or indirectly from content the model reads (web pages, reviews, emails, tool results). It cannot be fully prevented with prompts, so defend in layers:

• least-privilege tools and human approval for risky actions (the real boundary)
• instructions in the system prompt; untrusted content wrapped in tags and described as data
• detection heuristics or a classifier on inputs and retrieved docs
• validate outputs and tool arguments; never execute model text as SQL, shell or HTML

Mask PII (phone, email, Aadhaar, PAN, card, UPI IDs) before sending text to the LLM and before logging; keep a private mapping if you must restore values. Follow the DPDP Act 2023: clear notice and consent for the purpose, collect only what is needed, set retention limits and delete on time, support access/correction/erasure requests, secure data and report breaches. Do not reuse customer chats for training or evals without consent and anonymisation.