Retrieval Metrics · Evaluation in Generative AI | Generative AI & LLMs

Precision, Recall and MRR

Metric	Question it answers	Formula
precision@k	How much of the top k is relevant?	relevant in top k ÷ k
recall@k	How much of the relevant set did we find in the top k?	relevant in top k ÷ all relevant
hit rate@k	Did at least one relevant doc appear?	1 if any, else 0
MRR	How high is the first relevant doc?	mean of 1 ÷ rank of first relevant

Example

def precision_at_k(retrieved, relevant, k):
    return sum(d in relevant for d in retrieved[:k]) / k

def recall_at_k(retrieved, relevant, k):
    return sum(d in relevant for d in retrieved[:k]) / len(relevant)

def reciprocal_rank(retrieved, relevant):
    for rank, d in enumerate(retrieved, start=1):
        if d in relevant:
            return 1 / rank
    return 0.0

runs = [  # (question, retrieved ids in order, relevant ids)
    ("earbuds return?",   ["returns", "warranty", "cod"],      {"returns", "warranty"}),
    ("refund for COD?",   ["cod", "shipping", "refunds"],      {"refunds"}),
    ("express delivery?", ["returns", "cod", "warranty"],      {"shipping"}),
]
k = 3
for q, got, rel in runs:
    print(f"{q:18} P@3={precision_at_k(got, rel, k):.2f} R@3={recall_at_k(got, rel, k):.2f} "
          f"RR={reciprocal_rank(got, rel):.2f}")
mrr = sum(reciprocal_rank(got, rel) for _, got, rel in runs) / len(runs)
print(f"MRR = {mrr:.3f}")

Output

earbuds return?    P@3=0.67 R@3=1.00 RR=1.00
refund for COD?    P@3=0.33 R@3=1.00 RR=0.33
express delivery?  P@3=0.00 R@3=0.00 RR=0.00
MRR = 0.444

Reading the Numbers

For RAG, recall@k matters most: if the right chunk is not in the top k, the model cannot use it. The third question has recall 0, so fix retrieval (add BM25, better chunking) before touching the prompt.