Real World Example

• GitHub Copilot-style coding tools measure whether generated code passes hidden unit tests, not whether it "looks right".
• Education apps (Khan Academy, Duolingo style) have teachers grade samples against rubrics such as "does not give away the answer".
• Support platforms (Freshworks, Zoho style) track resolution rate and escalation rate per bot version.

The common thread: each team picks a metric that is close to the real outcome it cares about. Passing hidden tests is closer to "the code works" than how the code looks. "Did not give away the answer" is the actual teaching goal. Resolution rate is what the support business pays for. A good eval system links offline metrics (fast, run before release) to these online outcomes (slow, measured in production) and checks that they move together.

ShopEase combines everything from this module into one nightly job. Each golden case is run through the real pipeline (not a simplified copy), and three numbers are computed: recall@5 for retrieval (only for cases that have expected documents), must_include for the key facts, and faithful from the rubric judge. The report is compared with the previous night, and any drop is listed with the exact cases, so an engineer can open them the next morning.

golden.jsonl (214 cases) ──► real RAG pipeline ──► answer + doc ids
                                                   │
                     ┌─────────────────────────────┼──────────────────────┐
                     ▼                             ▼                      ▼
               recall@5 (search)        must_include (facts)     judge: faithful
                     └──────────────► nightly report + diff vs yesterday ◄┘

import json
from shopease.assistant import answer_with_sources      # the real RAG pipeline
from shopease.evals import judge, recall_at_k

cases = [json.loads(line) for line in open("evals/golden.jsonl", encoding="utf-8")]
report = {"recall@5": 0, "faithful": 0, "must_include": 0}
for c in cases:
    ans, doc_ids = answer_with_sources(c["question"])
    if c["expected_docs"]:
        report["recall@5"] += recall_at_k(doc_ids, set(c["expected_docs"]), 5)
    report["must_include"] += all(f.lower() in ans.lower() for f in c["must_include"])
    report["faithful"] += judge(c["question"], doc_ids, ans).faithful
n = len(cases)
n_ret = sum(1 for c in cases if c["expected_docs"])          # recall only counts cases with expected docs
scores = {k: round(v / (n_ret if k == "recall@5" else n), 3) for k, v in report.items()}
print(scores, "cases:", n)

python evals/nightly.py

{'recall@5': 0.912, 'faithful': 0.957, 'must_include': 0.893} cases: 214
compared with last night: must_include -0.019 (4 cases), see evals/report-diff.html

Note one detail in the code: recall@5 is divided only by the cases that have expected_docs. Out-of-scope questions have no relevant documents, and counting them would quietly pull the recall number down. Small details like this decide whether a metric can be trusted.

• Pick metrics close to the real outcome (tests pass, resolution rate).
• Run the real pipeline on the golden set nightly and in CI.
• Combine retrieval, fact and judge metrics; diff against the last run.
• Check that offline metrics track online results.