Architecture

What We Are Building

The ShopEase AI Assistant answers policy questions from documents (RAG), looks up and acts on orders with tools, recommends products with semantic search, and writes product descriptions for sellers.

Software architecture is the high-level plan of a system: which components exist, what each is responsible for, and how they talk. For a GenAI system, the most important architectural decision is where the model is trusted and where it is not. The LLM is powerful but probabilistic: it can be wrong, manipulated or slow. A good architecture surrounds it with deterministic code that handles the parts that must always be right (permissions, money, data access, limits), and gives the model only the parts that need language understanding (reading a question, choosing a handler, writing the answer).

Think of a hospital. Doctors (the model) make judgement calls, but the pharmacy checks every prescription, the billing system computes the bill, and access to patient records is controlled by role. Nobody lets the most skilled doctor also bypass the pharmacy. Every module in this level adds one of these surrounding systems.

Architecture Diagram

┌──────────────┐   HTTPS/SSE   ┌────────────────────────── FastAPI backend (Cloud Run) ──────────────────────────┐
│ Next.js chat │ ────────────► │ guardrails in ─► router (Haiku) ─┬─► policy RAG ──► Sonnet ──► guardrails out │
│   widget     │ ◄──────────── │  (PII, injection)                ├─► order agent (tools) ──► Sonnet           │
└──────────────┘               │                                  ├─► product search (vectors + BM25)          │
                               │                                  └─► seller copy (Opus, Batches API)          │
                               └───────┬───────────────┬────────────────┬──────────────────┬──────────────────┘
                                       ▼               ▼                ▼                  ▼
                               PostgreSQL+pgvector  Orders API     semantic cache       traces / logs
                               (chunks, products)   (MCP server)   (Redis)              (Langfuse / OTel)

Design Choices

Workflow first (router + specialised handlers); only the order handler is an agent loop.
One database (PostgreSQL + pgvector) for chunks, products and full-text search keeps operations simple.
Orders are reached through an MCP server, so the support team's desktop assistant uses the same tools.
Guardrails wrap the model on both sides, so no request reaches it unmasked and no reply leaves unchecked.
Each route gets the cheapest model that passes its evals: Haiku routes, Sonnet answers, Opus writes long copy offline at batch prices.

Read the diagram from left to right as the life of one message. Every arrow is a place where something can fail (timeouts, wrong routes, empty retrieval), so every arrow also needs a fallback and a log entry. The components at the bottom are shared infrastructure: one database, one tool server, one cache and one tracing system serve all routes, which keeps operations manageable for a small team.

Trade-offs and Summary

Note: What this design gives up: a single all-powerful agent with every tool would be quicker to prototype and could handle odd mixed requests ("track SE1042 and also suggest a phone case"). The routed design handles those less gracefully, but it is cheaper, easier to test per route and much safer. A common middle ground is to send rare mixed requests to the order agent with a few extra read-only tools.

Architecture decides where the model is trusted and where code enforces the rules.
Workflow at the top, an agent only where open-ended tool use is needed.
Shared infrastructure: Postgres + pgvector, MCP tools, cache, tracing.
Guardrails on both sides of every model call; model per route by evals.