Careeroza

Conversation Memory

Memory Is Your Job

The API is stateless, so "memory" is the list of messages your backend stores (in Redis, Postgres or memory) per conversation and sends each turn. Long chats grow without limit, so you need a policy: keep the last N turns, keep what fits in a token budget, or summarize old turns.

StrategyProsCons
Send everythingSimple, nothing forgottenCost and latency grow every turn
Sliding window (last N turns / token budget)Predictable costForgets early details (name, order ID)
Summary + recent turnsRemembers key facts cheaplyExtra call; summary may drop details
Store facts separately (order ID, language)Exact and reliableYou decide what to extract

A Token-budget Window

Example

Keep the newest messages that fit the budget. The window must start with a user message, so an orphan assistant message at the start is dropped:

def tokens(text):
    return max(1, len(text) // 4)          # estimate; use count_tokens for real numbers

history = [
    {"role": "user", "content": "Hi, I'm Kavya. My order SE-10423 hasn't arrived."},
    {"role": "assistant", "content": "Sorry Kavya! It is out for delivery and should arrive by 8 pm."},
    {"role": "user", "content": "Ok. Also, do you have running socks?"},
    {"role": "assistant", "content": "Yes: DryFit ankle socks (pack of 3) for Rs 249 and CushionRun for Rs 199."},
    {"role": "user", "content": "Add the DryFit ones to my cart please."},
]

def window(messages, budget):
    kept, used = [], 0
    for m in reversed(messages):
        t = tokens(m["content"])
        if used + t > budget:
            break
        kept.insert(0, m); used += t
    while kept and kept[0]["role"] != "user":
        kept.pop(0)
    return kept, used

for budget in (100, 40, 25):
    kept, used = window(history, budget)
    print(f"budget {budget:3}: keeps {len(kept)} messages ({used} tokens), first: {kept[0]['content'][:30]!r}")

Output

budget 100: keeps 5 messages (63 tokens), first: "Hi, I'm Kavya. My order SE-104"
budget  40: keeps 3 messages (36 tokens), first: 'Ok. Also, do you have running '
budget  25: keeps 1 messages (9 tokens), first: 'Add the DryFit ones to my cart'

With a small budget the bot forgets the customer's name and order ID. That is why the next topic adds a summary.