Careeroza

Why Evals Matter

From Vibes to Numbers

An eval is a repeatable test that scores your LLM app on fixed inputs. Without evals, every prompt change is a guess: it may fix one answer and quietly break ten others. Model upgrades, new documents and new tools all need the same check.

KindWhenExample
Offline evalsbefore every releaserun 200 golden questions, score answers and retrieval
Online metricsin productionthumbs up/down, escalation to human rate, resolution rate
Human reviewweekly samplesupport leads grade 50 random conversations
A/B testsbig changesprompt v3 for 10% of users, compare resolution rate

Evaluate Each Part

question ──► retrieval ──► prompt ──► model answer ──► tool actions
               │                          │                 │
         recall@k, MRR            faithfulness,       correct tool + args,
                                  correctness          no forbidden actions

Tip: When an answer is wrong, first check retrieval. If the right document was not retrieved, no prompt change will help.