Careeroza

Citations and Unknown Answers

Trust Needs Sources

Citations let users (and your code) check where an answer came from. Two simple rules make a RAG bot much safer:

  1. Every claim cites a document ID, and code checks that cited IDs were really in the context.
  2. No good context → no answer. If the best retrieval score is low, do not call the model at all, or tell it to reply "I don't know" and offer a human agent.

Note: Claude also has a built-in citations feature: send documents as document content blocks with "citations": {"enabled": true} and the response text comes with exact cited passages.

Checking Citations and Deciding to Answer

Example

The answers below are stand-ins for model replies:

import re

def should_answer(scores, min_best=0.55):
    return bool(scores) and max(scores) >= min_best

def check_citations(answer, context_ids):
    cited = {int(x) for x in re.findall(r"\[(\d+)\]", answer)}
    return {"cited": sorted(cited), "unknown_ids": sorted(cited - context_ids), "has_citation": bool(cited)}

context_ids = {1, 2}
answers = [
    "COD refunds go to your bank account or wallet [2], within 5-7 working days [1].",
    "Refunds are instant for all orders [4].",
    "You will get your refund soon.",
]
for a in answers:
    print(check_citations(a, context_ids), "<-", a[:45])

print("\nretrieval scores [0.83, 0.79] -> answer?", should_answer([0.83, 0.79]))
print("retrieval scores [0.31, 0.22] -> answer?", should_answer([0.31, 0.22]),
      "-> reply: I don't know that yet. Let me connect you to an agent.")

Output

{'cited': [1, 2], 'unknown_ids': [], 'has_citation': True} <- COD refunds go to your bank account or wallet
{'cited': [4], 'unknown_ids': [4], 'has_citation': True} <- Refunds are instant for all orders [4].
{'cited': [], 'unknown_ids': [], 'has_citation': False} <- You will get your refund soon.

retrieval scores [0.83, 0.79] -> answer? True
retrieval scores [0.31, 0.22] -> answer? False -> reply: I don't know that yet. Let me connect you to an agent.

The second answer cites document 4, which was never in the context: a sign of hallucination, so the app should not show it.