Preparing a Fine-tuning Dataset in JSONL

Training data is usually JSONL: one JSON object per line, each a full example conversation in the chat format the model will see at inference.

{"messages": [{"role": "system", "content": "Classify the ShopEase ticket intent."}, {"role": "user", "content": "parcel not delivered yet, its been 6 days"}, {"role": "assistant", "content": "delivery_delay"}]}
{"messages": [{"role": "system", "content": "Classify the ShopEase ticket intent."}, {"role": "user", "content": "paise wapas kab aayenge?"}, {"role": "assistant", "content": "refund_status"}]}

Why this shape? In supervised fine-tuning (SFT), the model learns to produce the assistant message given everything before it. So each example must look exactly like a real request at serving time: the same system prompt, the same style of user input. If you train with one system prompt and serve with another, you lose part of the benefit. Usually the training loss is computed only on the assistant tokens; the system and user text are context, not targets.

JSONL (JSON Lines) is used instead of one big JSON array because it can be streamed line by line, appended to, split and shuffled easily, and one broken line does not break the whole file.

A fine-tuned model copies the patterns in its data, including the mistakes. If two labellers tag "paise wapas kab aayenge?" as refund_status and payment_issue, the model learns that the boundary is random. A few hundred clean, consistent examples usually beat thousands of noisy ones. Think of it like preparing a student with solved papers: if a third of the answer key is wrong, more papers make them worse, not better.

The data should also match the real distribution of inputs. If 30% of real tickets are Hinglish but your dataset is all formal English, the model will be weakest exactly where users need it.

• Real, varied inputs (typos, Hinglish, short and long), not only clean examples.
• Consistent labels: write a labelling guide; two people should agree.
• Remove duplicates and PII before training.
• Keep a held-out validation set (10-20%) and never train on your eval set.
• Balanced classes, or at least enough examples of the rare ones.

import io, json, random
from collections import Counter

raw = [("parcel not delivered yet, its been 6 days", "delivery_delay"),
       ("paise wapas kab aayenge?", "refund_status"),
       ("Parcel not delivered yet, its been 6 days", "delivery_delay"),   # duplicate (case)
       ("wrong size shoes received", "exchange"),
       ("my number is 9876543210 call me", "callback"),
       ("refund not received for SE1042", "refund_status"),
       ("", "exchange")]                                                  # empty input
SYSTEM = "Classify the ShopEase ticket intent."
LABELS = {"delivery_delay", "refund_status", "exchange", "callback"}

seen, rows, dropped = set(), [], Counter()
for text, label in raw:
    key = text.strip().lower()
    if not key: dropped["empty"] += 1; continue
    if key in seen: dropped["duplicate"] += 1; continue
    if any(ch.isdigit() for ch in text) and len([c for c in text if c.isdigit()]) == 10:
        dropped["contains phone number"] += 1; continue
    assert label in LABELS, label
    seen.add(key)
    rows.append({"messages": [{"role": "system", "content": SYSTEM},
                              {"role": "user", "content": text.strip()},
                              {"role": "assistant", "content": label}]})

random.Random(42).shuffle(rows)
cut = max(1, int(len(rows) * 0.8))
train, val = rows[:cut], rows[cut:]
buf = io.StringIO()
for r in train:
    buf.write(json.dumps(r, ensure_ascii=False) + "\n")
print("kept:", len(rows), "| dropped:", dict(sorted(dropped.items())))
print("train:", len(train), "val:", len(val))
print("labels:", dict(sorted(Counter(r["messages"][2]["content"] for r in rows).items())))
print("first line:", buf.getvalue().splitlines()[0][:80] + "...")

kept: 4 | dropped: {'contains phone number': 1, 'duplicate': 1, 'empty': 1}
train: 3 val: 1
labels: {'delivery_delay': 1, 'exchange': 1, 'refund_status': 2}
first line: {"messages": [{"role": "system", "content": "Classify the ShopEase ticket intent...

Of seven raw rows, three are dropped: an empty input, a case-only duplicate, and a row with a 10-digit phone number (PII). The four kept rows are shuffled with a fixed seed and split 3/1 into train and validation. The label counts line is a quick class balance check you should always print.

• SFT data is JSONL, one complete conversation per line, matching the serving format.
• Consistency and realism beat volume.
• Clean: drop empties, duplicates and PII; check label balance.
• Deduplicate, then split; keep a held-out set you never train on.