Three Ways to Customize an LLM
The Options
Think of the model as a smart new employee. Prompting is giving clear instructions on day one. RAG is handing them the company handbook to look things up. Fine-tuning is months of training so the habits are built in.
| Prompting | RAG | Fine-tuning | |
|---|---|---|---|
| What changes | the input text | the input text (plus retrieved docs) | the model weights |
| Adds new knowledge | only what fits in the prompt | yes, and it stays up to date | poorly; facts get mixed up and go stale |
| Changes style / format / behaviour | yes | a little | yes, very consistently |
| Setup effort | minutes | days | weeks (data, training, evals) |
| Cost to change | edit the prompt | re-index documents | retrain |
| Explains its sources | no | yes (citations) | no |
What Actually Changes
The three methods differ in where the customisation lives. A trained model is a fixed set of numbers, its weights (parameters), learned during pre-training. Everything it "knows" and every habit it has comes from those weights plus the text you send in the request.
- Prompting leaves the weights alone and changes the input: instructions, examples (few-shot), output format. It is called in-context learning, because the model adapts to the task from the context alone, for that one request.
- RAG also leaves the weights alone. It adds retrieved documents to the input at request time, so the model can quote facts it never saw in training. Update a document and the next answer changes.
- Fine-tuning continues training on your examples and changes the weights. The new behaviour is present in every request without being in the prompt, but changing it means training again.
A useful rule of thumb: RAG is for knowledge, fine-tuning is for behaviour. Fine-tuning teaches a model how to answer (format, tone, a narrow task like intent labels) much better than what is true. Facts learned through fine-tuning are blended into the weights, cannot be cited, and go stale when your return policy changes next month. Tools (function calling) are a fourth lever for live data, like order status, that should never be memorised at all.
The Usual Order
- Write a good prompt with examples. Build an eval set.
- Add RAG when answers depend on your own or changing data (policies, catalogue, orders).
- Fine-tune only when evals show prompting + RAG cannot reach the quality, cost or latency you need, and you have hundreds to thousands of good examples.
Note
Note: Fine-tuning and RAG are not rivals. A fine-tuned small model can still use RAG. Hosted Claude models are customised with prompts, RAG and tools; open models (Llama, Mistral, Qwen, Gemma) are fine-tuned with Hugging Face tools, and some providers (for example OpenAI and Google) offer hosted fine-tuning.
Misconception: "fine-tuning on our documents will make the model know our company". It usually produces a model that sounds like your documents but still hallucinates their details. If the goal is correct, current facts, use RAG.
Summary
- Prompting and RAG change the input; fine-tuning changes the weights.
- RAG for knowledge (especially changing knowledge), fine-tuning for behaviour and narrow tasks, tools for live data.
- Go in order: prompt + evals → RAG/tools → fine-tune only if evals demand it.
- The methods combine; they are not either/or.