Careeroza

Training Data and Pre-training

Learning by Reading

Pre-training means showing the model trillions of tokens of text (web pages, books, code, articles) and asking it to predict each next token. Each wrong guess slightly adjusts its internal numbers (parameters). After enough text, the model has absorbed grammar, facts, coding patterns and styles.

  • No human labels are needed: the next word in the text is the answer. This is called self-supervised learning.
  • It is very expensive: thousands of GPUs for weeks or months.
  • The model only knows what was in its data up to a date called the training cut-off.

Training a Tiny Model by Counting

Example

The simplest possible "language model" just counts which word follows which in the training text, then turns counts into probabilities. Real pre-training learns far richer patterns, but the goal is the same: estimate P(next token | text so far).

from collections import Counter, defaultdict

corpus = """your order is on the way .
your order is delayed due to rain .
your refund is processed .
your order is on the way to your city ."""

counts = defaultdict(Counter)
words = corpus.split()
for a, b in zip(words, words[1:]):
    counts[a][b] += 1

def next_probs(word):
    c = counts[word]
    total = sum(c.values())
    return {w: round(n / total, 2) for w, n in c.most_common()}

print("after 'your':", next_probs("your"))
print("after 'is':  ", next_probs("is"))
print("after 'the': ", next_probs("the"))

Output

after 'your': {'order': 0.6, 'refund': 0.2, 'city': 0.2}
after 'is':   {'on': 0.5, 'delayed': 0.25, 'processed': 0.25}
after 'the':  {'way': 1.0}