Latency and Cost Control
Where Time and Money Go
| Factor | Effect | Lever |
|---|---|---|
| Output tokens | the slowest and most expensive part (output costs 5× input) | ask for short answers, set max_tokens |
| Model size | bigger = slower and pricier | route easy requests to Haiku or Sonnet |
| Input tokens | long prompts and history cost on every call | trim history, retrieve fewer chunks, cache the prefix |
| Effort / thinking | more thinking = better on hard tasks, slower | output_config={"effort": "low"} for simple tasks |
| Perceived latency | users wait for the first word | stream the response |
Why Output Tokens Dominate
To control latency and cost you need a mental model of how a request is processed. It has two phases. In prefill, the model reads all the input tokens at once, in parallel; this is fast per token. In decoding, it generates the output one token at a time, and each new token needs a full pass through the model. So a 3,000-token prompt may be read in a fraction of a second, while a 900-token answer takes several seconds. That is why output tokens are priced higher (5× input on current Claude models) and why "be brief" is the strongest latency lever you have.
| Term | Meaning |
|---|---|
| TTFT (time to first token) | how long until the first word appears; mostly prefill + queueing |
| Tokens per second | decoding speed; decides how long the full answer takes |
| Total latency | TTFT + output tokens ÷ tokens per second (+ tool and retrieval time) |
| p50 / p95 latency | median and slow-tail latency; users remember the p95 |
A worked feel: if TTFT is 0.6 s and the model produces about 80 tokens per second, a 160-token answer takes about 0.6 + 2 = 2.6 s, and a 800-token answer about 10.6 s. Cutting the answer length cuts the wait almost proportionally. Streaming does not make the total shorter, but the user starts reading after 0.6 s, which feels much faster.
Example
Example
A cost report from request logs, grouped by route. Prices per 1M tokens (input/output): Opus 5.5 $4/$20, Sonnet 5.5 $2/$10, Haiku 5.5 $0.10/$0.50. Prices change; check the pricing page.
PRICES = {"claude-opus-5-5": (4.00, 20.00), "claude-sonnet-5-5": (2.00, 10.00), "claude-haiku-5-5": (0.10, 0.50)}
LOGS = [ # (route, model, input_tokens, output_tokens, latency_ms)
("support_answer", "claude-sonnet-5-5", 3200, 180, 2100),
("support_answer", "claude-sonnet-5-5", 2900, 240, 2600),
("intent", "claude-haiku-5-5", 350, 4, 380),
("intent", "claude-haiku-5-5", 410, 4, 410),
("product_copy", "claude-opus-5-5", 1200, 900, 9800),
]
def cost(model, i, o):
pin, pout = PRICES[model]
return (i * pin + o * pout) / 1_000_000
report = {}
for route, model, i, o, ms in LOGS:
r = report.setdefault(route, {"calls": 0, "usd": 0.0, "ms": 0})
r["calls"] += 1; r["usd"] += cost(model, i, o); r["ms"] += ms
print(f"{'route':15}{'calls':>6}{'USD/call':>11}{'avg ms':>8}{'USD per 1M calls':>18}")
for route, r in sorted(report.items()):
per_call = r["usd"] / r["calls"]
print(f"{route:15}{r['calls']:>6}{per_call:>11.5f}{r['ms'] // r['calls']:>8}{per_call * 1_000_000:>18,.0f}")Output
route calls USD/call avg ms USD per 1M calls intent 2 0.00004 395 40 product_copy 1 0.02280 9800 22,800 support_answer 2 0.00820 2350 8,200
Tip
Tip: Track cost per route and per successful outcome (for example per resolved ticket), not only the total bill. That shows where a cheaper model or a shorter prompt pays off.
In the report, product_copy costs about 570 times more per call than intent and is the slowest, because it uses the largest model and writes 900 output tokens. The intent route is almost free: a small model, a short prompt and a 4-token answer. This is the typical shape of an LLM bill: a few routes with long outputs on big models dominate it.
Note: Misconceptions: "the cheapest model is always the cheapest system". If a small model fails and the user retries twice, or the ticket escalates to a human, the outcome costs more. And "shorter prompts are always better": cutting needed context lowers quality; trim history and duplicate text first, and use caching for the fixed part.
- Prefill (reading input) is fast; decoding (writing output) is slow and pricier.
- Biggest levers: shorter outputs, smaller models for easy routes, less input, caching, streaming.
- Measure TTFT and p95 latency, and cost per route and per successful outcome.