LLMOps and Production
How do you reduce the cost of an LLM app?
- Route easy tasks to small models (Haiku 5.5 at $0.10/$0.50 per 1M tokens vs Opus 5.5 at $4/$20; prices change).
- Prompt caching for the fixed prefix (cache reads cost about 10% of input price).
- Batches API for non-urgent jobs (50% off).
- Shorter outputs (
max_tokens, "answer in under 80 words"), lower effort for simple tasks. - Fewer, better chunks in RAG; trim or summarise chat history; semantic cache for FAQs.
How does prompt caching work, and what breaks it?
The provider caches the processed prompt prefix up to a cache_control breakpoint (render order: tools → system → messages). A later request with the byte-identical prefix reads it cheaply and faster; the default lifetime is 5 minutes. It breaks when anything in the prefix changes: a timestamp or user name in the system prompt, a different tool list or order, unsorted JSON, or a model switch. Check usage.cache_read_input_tokens to confirm hits.
How do you handle rate limits and provider errors?
Retry 429, 529 and 5xx with exponential backoff and jitter, honouring the retry-after header; never retry 400/401/403. The Anthropic SDK already retries these (2 retries by default, configurable with max_retries). Add client-side rate limiting (token bucket), timeouts, a fallback model, request queues for spikes, and a friendly message when everything fails.