Careeroza

Summarizing History

Summary Plus Recent Turns

When history passes a threshold, ask a cheap model to summarize the older part into a few lines of key facts, then send summary + last few turns. The summary goes into the system prompt (or a first user message) so the model still knows the customer's name, order and what was already tried.

Note: Claude also offers server-side compaction (a beta feature) that summarizes long conversations automatically. Building it yourself first shows what it does.

Compacting a Conversation

Example

fake_summarize is a stand-in for a call to a small model (it just pulls out names, order IDs and requests):

import re

history = [
    ("user", "Hi, I'm Kavya. My order SE-10423 hasn't arrived."),
    ("assistant", "Sorry Kavya! It is out for delivery and should arrive by 8 pm."),
    ("user", "Ok. Also, do you have running socks?"),
    ("assistant", "Yes: DryFit ankle socks for Rs 249 and CushionRun for Rs 199."),
    ("user", "Add the DryFit ones to my cart please."),
    ("assistant", "Done! DryFit ankle socks are in your cart."),
    ("user", "When will the shoes come again?"),
]

def fake_summarize(turns):
    """Stand-in for a cheap model call that writes a short summary of key facts."""
    text = " ".join(t for _, t in turns)
    facts = []
    if m := re.search(r"I'm (\w+)", text): facts.append(f"Customer name: {m.group(1)}")
    facts += [f"Order discussed: {o}" for o in sorted(set(re.findall(r"SE-\d+", text)))]
    if "cart" in text: facts.append("Added DryFit ankle socks to cart")
    return "\n".join(facts)

def compact(history, keep_last=3, max_turns=5):
    if len(history) <= max_turns:
        return None, history
    old, recent = history[:-keep_last], history[-keep_last:]
    assert recent[0][0] == "user", "messages sent to the API must start with a user turn"
    return fake_summarize(old), recent

summary, recent = compact(history)
system = "You are ShopEase support.\n<conversation_summary>\n" + summary + "\n</conversation_summary>"
print(system)
print("recent turns sent:", recent)

Output

You are ShopEase support.
<conversation_summary>
Customer name: Kavya
Order discussed: SE-10423
</conversation_summary>
recent turns sent: [('user', 'Add the DryFit ones to my cart please.'), ('assistant', 'Done! DryFit ankle socks are in your cart.'), ('user', 'When will the shoes come again?')]

With a Real Model

def summarize(turns: list[dict]) -> str:
    transcript = "\n".join(f"{m['role']}: {m['content']}" for m in turns)
    r = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=300,
        system="Summarize this support chat as short bullet facts: customer name, order IDs, problem, "
               "what was already done, open requests. No greetings.",
        messages=[{"role": "user", "content": f"<transcript>\n{transcript}\n</transcript>"}],
    )
    return next(b.text for b in r.content if b.type == "text")