LoRA and QLoRA Intuition

A 7-billion-parameter model has 7 billion numbers. Full fine-tuning updates all of them and keeps extra optimizer state for each, which needs many large GPUs. LoRA (Low-Rank Adaptation) freezes the original weights and trains two small matrices A and B per layer. The update is W + B × A.

W  (4096 × 4096, frozen)   +   B (4096 × r)  ×  A (r × 4096)   (trained, r = 8 or 16)

Each layer of a transformer contains big weight matrices: grids of numbers that transform one vector into another. Fine-tuning learns a change ΔW to add to each one. LoRA rests on one observation from research: the change needed to adapt a model to a narrow task is simple. It does not need a full 4096 × 4096 grid of independent numbers; it can be described by a few directions. In maths terms the change has low rank.

What does rank mean here? Multiplying a tall thin matrix B (4096 × r) by a short wide one A (r × 4096) gives a full-size 4096 × 4096 matrix, but every row of it is a combination of only r basic patterns. With r = 8 you describe the whole change with 8 "knobs" per layer instead of 16.7 million free numbers. It is like describing a photo filter with a few settings (brightness, warmth, contrast) instead of repainting every pixel by hand.

1. Freeze all original weights W; they are never updated.
2. Add small matrices A and B next to selected layers. B starts at zero, so at first the model behaves exactly like the base model.
3. Train only A and B on your examples.
4. Save just the adapter (megabytes). At serving time, use W + B × A, or merge it into W once.

d = 4096
full = d * d
for r in (8, 16, 64):
    lora = d * r + r * d
    print(f"rank {r:>2}: {lora:>9,} trainable vs {full:,} full  ({lora / full:.2%})")

# Tiny demo: W (3x3, frozen) + B (3x1) @ A (1x3)  -> a rank-1 change to every weight
W = [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]]
B = [[0.5], [0.0], [-0.5]]
A = [[0.2, 0.4, 0.0]]
delta = [[B[i][0] * A[0][j] for j in range(3)] for i in range(3)]
W_new = [[round(W[i][j] + delta[i][j], 2) for j in range(3)] for i in range(3)]
print("updated W:", W_new)
print("numbers trained:", len(B) + len(A[0]), "instead of", 9)

rank  8:    65,536 trainable vs 16,777,216 full  (0.39%)
rank 16:   131,072 trainable vs 16,777,216 full  (0.78%)
rank 64:   524,288 trainable vs 16,777,216 full  (3.12%)
updated W: [[1.1, 0.2, 0.0], [0.0, 1.0, 0.0], [-0.1, -0.2, 1.0]]
numbers trained: 6 instead of 9

For one 4096 × 4096 matrix, rank 8 trains 65,536 numbers, under 0.4% of the full 16.7 million. In the tiny demo, 6 trained numbers change several weights at once: rows 1 and 3 move in opposite directions (because B has +0.5 and −0.5) in the same pattern given by A. That shared pattern is what "rank 1" means.

QLoRA loads the frozen base model in 4-bit precision and trains LoRA adapters on top. That is how people fine-tune 7B-8B models on a single 16-24 GB GPU (or a free Colab/Kaggle GPU for small models).

def weights_gb(params_billion, bits):
    return params_billion * 1e9 * bits / 8 / 1e9

for name, params in [("Llama 3.1 8B", 8), ("Qwen2.5 1.5B", 1.5)]:
    print(f"{name:13} fp16 {weights_gb(params, 16):5.1f} GB | 4-bit {weights_gb(params, 4):4.1f} GB (weights only)")

Llama 3.1 8B  fp16  16.0 GB | 4-bit  4.0 GB (weights only)
Qwen2.5 1.5B  fp16   3.0 GB | 4-bit  0.8 GB (weights only)

Each parameter stored in 16-bit takes 2 bytes, so 8 billion parameters need 16 GB just to load; 4-bit storage cuts that to 4 GB. Training also needs memory for activations and the adapter's optimizer state, so real usage is higher than these numbers. Quantisation means storing numbers with fewer bits, accepting a small loss of precision; QLoRA keeps the adapters in higher precision so training stays stable.

from peft import LoraConfig
from trl import SFTConfig, SFTTrainer
from datasets import load_dataset

dataset = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})
peft_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                         target_modules="all-linear", task_type="CAUSAL_LM")
trainer = SFTTrainer(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
    peft_config=peft_config,
    args=SFTConfig(output_dir="shopease-intent-lora", num_train_epochs=3,
                   per_device_train_batch_size=8, learning_rate=2e-4),
)
trainer.train()
trainer.save_model()      # saves only the small adapter (a few MB)

• LoRA trains small low-rank matrices next to frozen weights: under 1% of the parameters.
• QLoRA loads the frozen base in 4-bit so big models fit on one GPU.
• Adapters are small, cheap to store and easy to swap.