Careeroza

Regression Testing Prompts

Treat Prompts Like Code

Store prompts in version control with an ID (support_v7). On every change, run the golden set against the old and new version and block the release if the score drops or any must-pass case fails.

Example

Example

Compare two prompt versions on the same cases. fake_model is a stand-in that behaves differently per prompt version.

CASES = [("ret-001", "earbuds return window?", ["7", "replacement"]),
         ("ref-003", "refund time for UPI?", ["5-7"]),
         ("oos-002", "who wins the IPL?", ["only help with ShopEase"])]

def fake_model(prompt_version, question):  # stand-in for the model
    answers = {
        "v6": {"earbuds return window?": "Earbuds have a 7 day replacement.",
               "refund time for UPI?": "Refunds take 5-7 working days.",
               "who wins the IPL?": "Probably Mumbai Indians!"},
        "v7": {"earbuds return window?": "Earbuds have a 7 day replacement.",
               "refund time for UPI?": "Refunds usually take a few days.",
               "who wins the IPL?": "Sorry, I can only help with ShopEase orders and products."},
    }
    return answers[prompt_version][question]

def run(version):
    return {cid: all(f.lower() in fake_model(version, q).lower() for f in facts) for cid, q, facts in CASES}

old, new = run("v6"), run("v7")
for cid in old:
    change = "" if old[cid] == new[cid] else ("  FIXED" if new[cid] else "  REGRESSION")
    print(f"{cid}: v6={'pass' if old[cid] else 'fail'} v7={'pass' if new[cid] else 'fail'}{change}")
print(f"score v6={sum(old.values())}/3 v7={sum(new.values())}/3")
print("release blocked" if any(old[c] and not new[c] for c in old) else "release ok")

Output

ret-001: v6=pass v7=pass
ref-003: v6=pass v7=fail  REGRESSION
oos-002: v6=fail v7=pass  FIXED
score v6=2/3 v7=2/3
release blocked

Run Evals in CI

# .github/workflows/evals.yml
name: llm-evals
on: [pull_request]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: pytest evals/ -q
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Tip: Keep the CI eval set small (50-100 cases) so it is fast and cheap; run the full set nightly. Tools like promptfoo, DeepEval, Ragas, LangSmith and Langfuse help organise this.