Distillation

Distillation uses a large, accurate teacher model to label or answer many examples, then trains (or few-shot prompts) a small, cheap student model on those outputs. You get most of the quality at a fraction of the cost and latency.

10,000 real tickets ──► teacher (large model, Batches API) ──► labels + confidence
                                                              │ filter, spot-check by humans
                                                              ▼
                                         train.jsonl ──► fine-tune small model ──► evaluate vs teacher

Large models are expensive because they are general: they can write code, poems and legal summaries. A ShopEase ticket router needs one narrow skill. A small model has enough capacity for that one skill; what it lacks is good training data. The teacher supplies that data cheaply, because labelling 10,000 tickets with a large model costs far less than paying people to do it, and it is fast.

It is like a senior doctor writing detailed case notes that junior doctors study. The juniors do not become senior doctors, but for common cases they make the same call, faster and at lower cost. The student is only as good as the teacher on that task, and usually a bit worse.

• Response distillation (what this page shows): train the student on the teacher's final outputs. Works with any API model.
• Logit distillation: train the student to match the teacher's full probability distribution over tokens. Richer signal, but needs access to the teacher's internals, so it is used mainly with open models.

TICKETS = ["refund kab milega", "order late by 4 days", "size exchange please",
           "item damaged in box", "where is my parcel", "refund not credited"]

def teacher(text):  # stand-in for a large model's label + confidence
    t = text.lower()
    if "refund" in t: return "refund_status", 0.97
    if "late" in t or "where" in t: return "delivery_delay", 0.93
    if "exchange" in t: return "exchange", 0.95
    return "damaged_item", 0.61

def student(text):  # stand-in for the small model after training
    t = text.lower()
    return "refund_status" if "refund" in t else "delivery_delay" if ("late" in t or "where" in t) else "exchange"

labelled = [(t, *teacher(t)) for t in TICKETS]
train = [(t, lab) for t, lab, conf in labelled if conf >= 0.9]
print("kept for training:", len(train), "of", len(TICKETS))
agree = sum(student(t) == lab for t, lab, _ in labelled)
print(f"student agrees with teacher on {agree}/{len(labelled)} tickets")
print("send to human review:", [t for t, _, conf in labelled if conf < 0.9])

kept for training: 5 of 6
student agrees with teacher on 5/6 tickets
send to human review: ['item damaged in box']

Five of six labels are confident enough to train on. The low-confidence one ("item damaged in box") goes to a human, which is where your labelling budget is best spent. The student agrees with the teacher on 5/6, missing exactly the class it never saw in training. That is a typical failure: rare classes need extra examples.

Use the Message Batches API for the teacher step: it is asynchronous and costs 50% of normal prices.

import anthropic
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

client = anthropic.Anthropic()
batch = client.messages.batches.create(requests=[
    Request(custom_id=f"ticket-{i}", params=MessageCreateParamsNonStreaming(
        model="claude-opus-5-5",
        max_tokens=100,                        # room for any thinking + the label
        output_config={"effort": "low"},       # short classification task
        system="Label the ShopEase ticket with one of: refund_status, delivery_delay, exchange, damaged_item. Output only the label.",
        messages=[{"role": "user", "content": text}],
    ))
    for i, text in enumerate(tickets)
])
print(batch.id, batch.processing_status)

python label_batch.py

msgbatch_01HkcTjaV5uDC8jWR4ZsDV8d in_progress

• Distillation = a large teacher labels data; a small student learns from it.
• It trades a little quality for big savings in cost and latency on narrow tasks.
• Filter by confidence, send unsure cases to humans, watch rare classes.
• Use the Batches API for cheap, asynchronous teacher labelling.