When to Use Which

Decision Guide

Your need	Best first choice
Answer from policy docs that change every month	RAG
Always reply in a fixed JSON format	prompting + structured outputs
Use live data (order status, stock)	tools / function calling
Classify 5 million tickets a day at very low cost	fine-tune (or distil into) a small model
Speak in your brand voice consistently	prompting with examples first; fine-tune if still inconsistent
Domain language the base model does not know (rare Indic dialects, internal codes)	fine-tuning, plus RAG for facts

How to Reason About It

The table is easier to remember if you ask four questions about the requirement, in this order:

Is the problem missing knowledge or wrong behaviour? Missing or changing facts point to RAG or tools. Wrong format, tone or task skill points to prompting, then fine-tuning.
How often does the information change? Order status changes every minute (tools), policies every month (RAG), brand voice almost never (prompt or fine-tune).
What is the volume and latency budget? At millions of calls a day, a small specialised model can be 10-40× cheaper than a large general one. At a few thousand calls a day, the engineering cost of training is rarely worth it.
Do you have the data? Fine-tuning needs hundreds to thousands of high-quality, consistent examples, plus an eval set. Without them, any fine-tune is guesswork.

An analogy: if a new cashier keeps quoting old prices, you give them the updated price list (RAG), not a six-week training course (fine-tuning). If they are rude to customers, a price list will not help; that is a behaviour problem.

Example

A small rule-based helper that encodes the guide. Interviewers like this kind of structured reasoning.

def recommend(need):
    picks = []
    if need.get("private_or_changing_data"):
        picks.append("RAG")
    if need.get("live_actions_or_data"):
        picks.append("tools")
    if need.get("strict_format"):
        picks.append("structured outputs")
    if need.get("requests_per_day", 0) > 1_000_000 and need.get("narrow_task"):
        picks.append("fine-tune a small model")
    if need.get("examples", 0) < 500 and "fine-tune a small model" in picks:
        picks.append("(first collect more labelled examples)")
    return picks or ["prompting with few-shot examples"]

cases = {
    "policy Q&A bot":        {"private_or_changing_data": True},
    "order tracking":        {"live_actions_or_data": True, "strict_format": True},
    "ticket intent tagging": {"requests_per_day": 3_000_000, "narrow_task": True, "examples": 200},
    "product title rewrite": {},
}
for name, need in cases.items():
    print(f"{name:22} -> {', '.join(recommend(need))}")

Output

policy Q&A bot         -> RAG
order tracking         -> tools, structured outputs
ticket intent tagging  -> fine-tune a small model, (first collect more labelled examples)
product title rewrite  -> prompting with few-shot examples

The intent-tagging case shows the important nuance: the volume justifies a fine-tuned small model, but 200 examples is not enough, so the helper says to collect data first. The product title case has no special need, so plain prompting wins.

Note: Common mistake: choosing fine-tuning first because it sounds more advanced in an interview or a pitch. Interviewers are usually checking whether you would start with the cheapest approach that meets the eval target and justify each step up with numbers.

Exercise

Exercise: Duolingo-style app: the bot must explain grammar mistakes in Hinglish, in a friendly fixed tone, for 2 lakh learners a day. Which approach would you start with, and what would make you switch?

Show answer

Start with prompting: a system prompt describing the tone + 5-10 few-shot examples in Hinglish,
plus an eval set of ~100 learner sentences graded by teachers.
Switch to fine-tuning (or distillation into a smaller model) only if
  - the tone stays inconsistent on the eval set, or
  - cost/latency at 2 lakh users/day is too high with a large model.
Grammar rules themselves can come from RAG over a curated grammar guide.

Summary

First ask: missing knowledge or wrong behaviour?
Change frequency decides between tools (live), RAG (changing) and weights (stable).
Volume and latency can justify a small fine-tuned or distilled model.
No good dataset, no fine-tuning.