Careeroza
Monitoring in Production
What to Watch
| Signal | Example alert |
|---|---|
| p95 latency and time to first token | p95 > 8 s for 10 minutes |
| Error rate by type (429, 5xx, timeouts) | > 2% of requests |
| Cost per hour and per user | > 2× yesterday's hourly average |
| Quality: thumbs down, escalations, "I don't know" rate | thumbs-down rate doubles after a deploy |
| Safety: blocked injections, PII detections | spike from one user or IP |
Example
Example
Turn a window of request logs into the numbers a dashboard shows.
LOGS = [ # (latency_ms, status, usd, feedback)
(1800, 200, 0.004, "up"), (2100, 200, 0.005, None), (950, 200, 0.002, None),
(7600, 200, 0.011, "down"), (2300, 200, 0.004, None), (30000, 504, 0.0, None),
(1500, 429, 0.0, None), (2600, 200, 0.006, "up"), (1900, 200, 0.004, None), (2200, 200, 0.005, None),
]
def percentile(values, p):
s = sorted(values)
return s[min(len(s) - 1, round(p / 100 * (len(s) - 1)))]
ok = [l for l in LOGS if l[1] == 200]
lat = [l[0] for l in ok]
errors = len(LOGS) - len(ok)
votes = [l[3] for l in LOGS if l[3]]
print(f"requests {len(LOGS)} | error rate {errors / len(LOGS):.0%}")
print(f"latency p50 {percentile(lat, 50)} ms | p95 {percentile(lat, 95)} ms")
print(f"cost USD {sum(l[2] for l in LOGS):.3f} | thumbs down {votes.count('down')}/{len(votes)}")
if errors / len(LOGS) > 0.02:
print("ALERT: error rate above 2%")Output
requests 10 | error rate 20% latency p50 2200 ms | p95 7600 ms cost USD 0.041 | thumbs down 1/3 ALERT: error rate above 2%
Tip
Tip: Log the prompt version and model with every request. When a metric moves, you can see at once whether it started with a deploy, a prompt change or a provider issue.