Careeroza

Golden Datasets

What Goes In

  • Input: real user questions (anonymised), including typos and Hinglish.
  • Expected output: an exact answer, or the facts the answer must contain.
  • Expected sources: which documents should be retrieved (for retrieval metrics).
  • Tags: category, difficulty, language, "should refuse", "needs tool".
  • Edge cases: out-of-scope questions, prompt injections, missing info.
{"id": "ret-007", "question": "earbuds return kar sakte hai?", "must_include": ["7", "replacement"], "expected_docs": ["returns"], "tags": ["returns", "hinglish"]}
{"id": "oos-002", "question": "Who will win the IPL this year?", "must_include": [], "expected_docs": [], "tags": ["out_of_scope", "should_refuse"]}

Example

Example

Check that the golden set covers every category before trusting its score.

from collections import Counter

golden = [
    {"id": "ret-001", "tags": ["returns"]}, {"id": "ret-007", "tags": ["returns", "hinglish"]},
    {"id": "ref-003", "tags": ["refunds"]}, {"id": "ord-010", "tags": ["orders", "needs_tool"]},
    {"id": "oos-002", "tags": ["out_of_scope", "should_refuse"]},
    {"id": "inj-001", "tags": ["injection", "should_refuse"]},
]
REQUIRED = ["returns", "refunds", "orders", "shipping", "out_of_scope", "injection", "hinglish"]

counts = Counter(t for row in golden for t in row["tags"])
for tag in REQUIRED:
    flag = "" if counts[tag] >= 1 else "   <-- missing, add cases"
    print(f"{tag:13} {counts[tag]}{flag}")

Output

returns       2
refunds       1
orders        1
shipping      0   <-- missing, add cases
out_of_scope  1
injection     1
hinglish      1

Tip

Tip: Start with 30-50 cases written by the team in one afternoon. Then add every bad production conversation as a new case. A golden set that grows from real failures is worth more than a big synthetic one.