Pre-training Fine-tuning and RLHF · How LLMs Work in Generative AI | Generative AI & LLMs

Pre-training Fine-tuning and RLHF

Three Stages in Simple Words

Stage	Data	What the model learns	Result
Pre-training	Trillions of tokens of general text	Language, facts, code, reasoning patterns	A "base model" that continues any text
Fine-tuning (instruction tuning)	Thousands of example conversations (question → good answer)	To follow instructions and chat	An assistant model
RLHF / preference training	Humans (or AI) rank two answers: which is better?	To be helpful, honest and harmless	The polished chat model you call through the API

RLHF = Reinforcement Learning from Human Feedback. Anthropic also uses Constitutional AI, where the model is trained against a written set of principles with AI feedback.

Fine-tuning Shifts the Probabilities

Example

Using the counting model again: first train on general text, then add a small amount of support-chat data. The next-word probabilities move toward the support style. Fine-tuning a real LLM works on the same principle: a small, focused dataset nudges a big pre-trained model.

from collections import Counter, defaultdict

def train(text, counts):
    w = text.split()
    for a, b in zip(w, w[1:]):
        counts[a][b] += 1

def top(counts, word):
    c = counts[word]
    total = sum(c.values())
    return [(w, round(n / total, 2)) for w, n in c.most_common(3)]

counts = defaultdict(Counter)
general = "sorry I am late . sorry I forgot . sorry to hear that . sorry I am busy ."
train(general, counts)
print("base model, after 'sorry':      ", top(counts, "sorry"))

support = "sorry for the delay . sorry for the trouble . sorry for the inconvenience ."
for _ in range(2):            # repeat the small dataset (like training epochs)
    train(support, counts)
print("fine-tuned model, after 'sorry':", top(counts, "sorry"))

Output

base model, after 'sorry':       [('I', 0.75), ('to', 0.25)]
fine-tuned model, after 'sorry': [('for', 0.6), ('I', 0.3), ('to', 0.1)]

Tip: As an app developer you rarely fine-tune. Prompting and RAG (giving the model your documents) solve most problems more cheaply. The advance level compares these options.