Careeroza

Prompt Versioning

Prompts Change, So Version Them

A one-word change in a prompt can change the behaviour of your whole product. Treat prompts like code:

  • Store prompts in files in Git (or a prompt registry), each with a version: support_reply@v3.
  • Log the prompt version with every request, so you can trace a bad answer back to the prompt that produced it.
  • Before switching versions, run both on the same test set and compare (evaluation is covered in the advance level).
  • Roll out gradually: send 10% of traffic to the new version first.

A Small Prompt Registry with Gradual Rollout

Example

Each user is put into a stable bucket using a hash of their ID, so the same user always sees the same version during the experiment:

import hashlib

PROMPTS = {
    "support_reply": {
        "v2": "You are ShopEase support. Be polite. Answer in 3 sentences.",
        "v3": "You are ShopEase support. Be warm and brief (max 60 words). Offer one clear next step.",
    }
}
ROLLOUT = {"support_reply": {"stable": "v2", "candidate": "v3", "percent": 10}}

def bucket(user_id):
    return int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 100

def get_prompt(name, user_id):
    r = ROLLOUT[name]
    version = r["candidate"] if bucket(user_id) < r["percent"] else r["stable"]
    text = PROMPTS[name][version]
    fingerprint = hashlib.sha256(text.encode()).hexdigest()[:8]
    return version, fingerprint, text

counts = {"v2": 0, "v3": 0}
for i in range(1000):
    v, _, _ = get_prompt("support_reply", f"user-{i}")
    counts[v] += 1
print("traffic split over 1000 users:", counts)

v, fp, text = get_prompt("support_reply", "user-42")
print(f"user-42 -> {v} (sha {fp}): {text}")

Output

traffic split over 1000 users: {'v2': 905, 'v3': 95}
user-42 -> v2 (sha c13670e6): You are ShopEase support. Be polite. Answer in 3 sentences.