Choosing Models per Route

One App, Several Models

Different parts of an app need different models. A router picks the model per task, and a fallback chain tries another model when one is overloaded.

Route	Model	Why
intent classification, PII check	`claude-haiku-5-5`, effort low	tiny outputs, must be fast and cheap
support answers with RAG	`claude-sonnet-5-5`	good quality at medium cost
complex agent tasks, long product copy, judge for evals	`claude-opus-5-5`	best reasoning and writing

Why Route

Models sit on a quality-cost-latency trade-off curve: larger models are more capable on hard tasks but slower and pricier. Most apps have a mix of tasks, and many are easy. Sending everything to the largest model is like sending every parcel by air: it works, but you pay air rates for things that could go by road. Model routing means choosing the smallest model that meets the quality bar for each route, as measured by evals.

There are two common ways to decide. Static routing fixes the model per task in configuration (intent always goes to Haiku), which is simple and predictable. Dynamic routing decides per request, for example a classifier sends hard questions to a bigger model, or the app tries a small model and escalates if a check fails. Start static; add dynamic routing only where evals show a clear gain.

A fallback chain is about availability, not quality: if the first model returns 529 (overloaded) after retries, try the next one so the user still gets an answer. Choose fallbacks you have already evaluated on that route.

Example

A router with a fallback chain. fake_call is a stand-in that simulates Sonnet being overloaded.

ROUTES = {"intent": ["claude-haiku-5-5"],
          "support_answer": ["claude-sonnet-5-5", "claude-opus-5-5"],
          "agent": ["claude-opus-5-5", "claude-sonnet-5-5"]}
OVERLOADED = {"claude-sonnet-5-5"}

class Overloaded(Exception): pass

def fake_call(model, prompt):  # stand-in for the API call
    if model in OVERLOADED:
        raise Overloaded(f"529 from {model}")
    return f"[{model}] answer to: {prompt}"

def complete(route, prompt):
    errors = []
    for model in ROUTES[route]:
        try:
            return fake_call(model, prompt), errors
        except Overloaded as e:
            errors.append(str(e))
    raise RuntimeError(f"all models failed: {errors}")

for route, prompt in [("intent", "refund kab milega"), ("support_answer", "return window for shoes?")]:
    answer, errors = complete(route, prompt)
    print(f"{route:15} -> {answer}" + (f"   (fallback after {errors})" if errors else ""))

Output

intent          -> [claude-haiku-5-5] answer to: refund kab milega
support_answer  -> [claude-opus-5-5] answer to: return window for shoes?   (fallback after ['529 from claude-sonnet-5-5'])

Note

The intent route goes straight to Haiku. The support route tries Sonnet, gets a simulated 529, and falls back to Opus, and the result records which error caused the fallback so you can monitor how often it happens.

Note: Switching models changes behaviour. Run your eval set on every model in a fallback chain, and remember that caches are per model.

Route each task to the smallest model that passes its evals.
Static routing first; dynamic routing or escalation where it pays.
Fallback chains keep the app available during overloads.
Evaluate every model in the chain; prompt caches do not carry across models.