Components

Each component has a single responsibility, which is the classic design principle that makes systems testable and replaceable. The router only classifies; it never answers. Policy RAG only answers from documents; it cannot touch orders. The order agent is the only part with write tools, and those tools enforce their own limits. Because responsibilities do not overlap, you can swap one part (a better re-ranker, a new embedding model, a different model for routing) and re-run that part's evals without retesting everything.

The components fall into three layers, which is a useful way to explain any GenAI system in an interview:

• Interface layer: chat UI, API backend, streaming. Concerns: latency, auth, error messages.
• Intelligence layer: router, RAG, agent, recommendations, copywriting. Concerns: quality, grounding, tool correctness.
• Platform layer: guardrails, caching, evals, observability, secrets. Concerns: safety, cost, reliability. This layer is what turns a demo into a product, and it is usually most of the engineering work.

PRICES = {"haiku": (0.10, 0.50), "sonnet": (2.00, 10.00), "opus": (4.00, 20.00)}
# component: (model, calls per day, input tokens per call, output tokens per call, share of input cached)
PLAN = {
    "router":        ("haiku",  20_000 * 3,   400,   5, 0.0),
    "policy answers":("sonnet", 20_000 * 2, 4_000, 150, 0.8),
    "order agent":   ("sonnet", 20_000 * 1, 3_000, 200, 0.7),
    "seller copy":   ("opus",   2_000,      1_500, 600, 0.0),
}
total = 0.0
for name, (model, calls, tin, tout, cached) in PLAN.items():
    pin, pout = PRICES[model]
    input_cost = calls * tin * ((1 - cached) * pin + cached * pin * 0.1) / 1e6
    output_cost = calls * tout * pout / 1e6
    if name == "seller copy":
        input_cost, output_cost = input_cost / 2, output_cost / 2      # Batches API: 50% off
    month = 30 * (input_cost + output_cost)
    total += month
    print(f"{name:15} {model:7} USD {month:9,.0f}/month")
print(f"{'total':23} USD {total:9,.0f}/month")

router          haiku   USD        76/month
policy answers  sonnet  USD     4,488/month
order agent     sonnet  USD     2,532/month
seller copy     opus    USD       540/month
total                   USD     7,636/month

About 92% of the bill comes from the two Sonnet routes, so that is where optimisation pays: higher cache hit rates, fewer retrieved chunks, shorter answers, or testing whether Haiku passes the policy-answer evals. The router, though it runs 60,000 times a day, costs under USD 80 a month because Haiku is cheap and outputs only a few tokens. Seller copy on Opus is affordable because it runs through the Batches API at half price.

• Summary: one responsibility per component; overlap makes testing and swapping hard.
• Interface, intelligence and platform layers; the platform layer is most of the work.
• Estimate cost per component, then optimise the biggest line first.