Prompt Caching and Batching

Most requests repeat a long, fixed prefix: tools, system prompt, policy docs. Prompt caching stores that prefix on the provider side. Later requests with the exact same prefix read it at about 10% of the input price; writing it costs about 1.25× once. The default cache lifetime is 5 minutes (a 1-hour option exists).

Why is this possible? During prefill, the model computes internal values (the key/value cache) for every input token, and those values depend only on the tokens before them. So if two requests start with the same 20,000 tokens, the work for those tokens is identical. The provider can store it and skip it next time. This is also why the match must be an exact prefix: change one token early in the prompt and every value after it changes, so nothing after that point can be reused. It is like a teacher who has already read the first 50 pages of a long file: if the file is the same, they continue from page 51; if page 3 changed, they must reread from page 3.

1. Put the fixed parts first: tools, then the system prompt and policy docs. Changing parts (the user's question, history) go last.
2. Mark the end of the fixed part with cache_control.
3. The first request writes the cache (slightly more expensive); later requests with the same prefix read it at about a tenth of the price, and with lower latency.
4. Each read refreshes the lifetime, so steady traffic keeps the cache warm. Very short prefixes are not cached (the minimum length depends on the model).

response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1000,
    system=[{
        "type": "text",
        "text": SHOPEASE_POLICIES,                 # ~20,000 tokens, identical on every call
        "cache_control": {"type": "ephemeral"},
    }],
    messages=[{"role": "user", "content": question}],
)
u = response.usage
print(u.cache_creation_input_tokens, u.cache_read_input_tokens, u.input_tokens)

python ask.py  # second call within 5 minutes

0 20184 42

price_in = 2.00 / 1_000_000
prefix, question, requests, hit_rate = 20_000, 150, 10_000, 0.95

no_cache = requests * (prefix + question) * price_in
hits = requests * hit_rate
misses = requests - hits
with_cache = (misses * prefix * 1.25 + hits * prefix * 0.10 + requests * question) * price_in
print(f"input cost without caching: USD {no_cache:8.2f}/day")
print(f"input cost with caching   : USD {with_cache:8.2f}/day")
print(f"saving                    : {1 - with_cache / no_cache:.0%}")

input cost without caching: USD   403.00/day
input cost with caching   : USD    66.00/day
saving                    : 84%

The input bill falls from USD 403 to USD 66 a day, an 84% saving. The 5% misses still pay the 1.25× write price, and the 150-token questions are never cached, but the 95% hits dominate. The sample usage line above shows the same effect for one request: 20,184 tokens read from cache and only 42 billed as normal input.

• No timestamps, request IDs or user names in the system prompt; put changing data in the messages.
• Keep the tool list and its order the same on every call.
• Serialise JSON with sort_keys=True if it goes into the prompt.
• Check cache_read_input_tokens in your logs; if it is always 0, something in the prefix changes.

For work that does not need an instant answer (nightly product descriptions, labelling, evals), the Message Batches API processes up to 100,000 requests per batch at 50% of the normal price; most batches finish within an hour.

for result in client.messages.batches.results(batch_id):
    if result.result.type == "succeeded":
        msg = result.result.message
        save_description(result.custom_id, next(b.text for b in msg.content if b.type == "text"))
    else:
        retry_later(result.custom_id, result.result.type)     # errored / canceled / expired

Batching trades latency for price: the provider can schedule your requests when it has spare capacity, so it charges less. Use it whenever nobody is waiting for the answer. Results come back in any order, so match them by custom_id, and plan for some requests to fail and need a retry.

• Summary: prompt caching reuses the computed prefix; keep fixed content first and byte-for-byte stable.
• Cache reads cost about 10% of input price; writes cost a bit more once.
• Batches cost 50% and suit offline jobs; match results by custom_id.
• Both can be combined: batch requests can share a cached prefix.