Cover graphic for 'RAG vs Fine-Tuning vs Prompting: What Your Problem Needs': three labeled badges for Prompting, RAG, and Fine-tuning, each connected to the distinct problem it solves — unclear ask, missing facts, and wrong style, beside the headline Three fixes, not one.
AI Concepts

RAG vs Fine-Tuning vs Prompting: What Your Problem Needs

In 2024, the legal AI company Harvey worked with OpenAI to build something ambitious: a custom-trained model fed the equivalent of 10 billion tokens of U.S. case law, plus a purpose-built embedding model trained on more than 20 billion tokens of legal text, so it could tell one court’s reasoning from another’s. In blind tests, lawyers preferred its output to plain GPT-4 97% of the time.

That’s a genuine fine-tuning success story, and it’s a useful one to sit with, because of what happened next. By 2026, Harvey’s own documentation describes its production system rather differently: a mix of off-the-shelf frontier models from Anthropic, OpenAI, and Google, cascading through retrieval systems that pull in public and user-provided data, coordinated by careful orchestration rather than one single custom-trained brain. The company that pioneered a heavyweight legal fine-tune ended up leaning most of its weight on retrieval, orchestration, and prompting instead.

That’s not a story about fine-tuning failing. It’s a story about three different tools solving three different problems, and a sophisticated team routing each problem to the tool actually built for it, then adjusting as the underlying models got better. Most teams asking “should we use RAG or fine-tuning?” are actually asking the wrong question. It’s rarely either-or, and it’s almost never the first thing you should try.

This article is the framework for figuring out which one your problem actually needs, in what order, and why. No hype, no “fine-tuning is dead” absolutism, no vendor pitch. Just a clear way to diagnose the actual failure you’re dealing with.

Quick answer: Start with prompting, since it’s free and immediate. Add retrieval-augmented generation (RAG) when the model is missing information it was never trained on, current facts, private data, anything that changes. Only consider fine-tuning when you’ve genuinely tried both of those and the model still gets the underlying behavior wrong in a way that’s structural rather than informational, and even then, check whether a narrow, high-volume, stable task is what’s actually driving the need. Most serious systems end up combining more than one of these, not picking a single winner.

Here’s what you’ll walk away knowing:

Three different failures, not one multiple-choice question

Before comparing mechanisms, it’s worth being precise about what’s actually going wrong when an AI tool disappoints you, because the three fixes target three genuinely different failures.

The model has the knowledge and the skill, but isn’t doing what you want. It’s answering in the wrong tone, the wrong format, ignoring a constraint you gave it, or being needlessly verbose. This is an instruction problem. Prompting fixes it.

The model is missing information it needs. It doesn’t know your return policy, your product’s pricing this quarter, or what happened in the news this morning, because that information either didn’t exist during training or was never public. This is a knowledge problem. RAG fixes it.

The model has the knowledge and the instructions, but the underlying behavior itself needs to change at a structural level, for a task performed so often, so narrowly, and so predictably that baking the pattern directly into the model’s weights is worth the cost. This is a rare, specific kind of problem, and fine-tuning is the only one of the three that actually touches it.

Confusing these three is expensive. Fine-tuning a model to “know” your company’s current pricing is like sending an employee to a six-month training course to memorize a price list that changes monthly. It will work, technically, for about a month. It’s also almost never the efficient way to solve that particular problem.

What each lever actually changes

If you want the deep mechanics of any one of these, we’ve covered them individually elsewhere: what RAG actually is and how it works, and how large language models predict text in the first place. Here’s the short version of each, side by side.

Prompting changes nothing about the model. It changes what you ask it and how you ask it, through instructions, examples, formatting rules, and constraints, all supplied at the moment you make the request. It’s the fastest of the three to test and the only one that costs nothing beyond the tokens you’re already spending.

RAG also changes nothing about the model. It changes what the model is given to read before it answers, by searching an external knowledge source (documents, a database, the live web) for relevant material and attaching it to your prompt automatically. The model’s weights never move; only its reading material does.

Fine-tuning is the one that changes the model itself. Additional training adjusts the model’s internal parameters so a particular pattern, tone, or skill becomes part of how it behaves by default, without needing to be re-explained in every prompt. It’s the only one of the three that persists inside the model rather than being supplied at request time.

It helps to see the shape of each one side by side, not just described. A prompting fix looks like adding a line to your system instructions: “Always answer in under three sentences, and never mention a competitor by name.” A RAG fix looks like nothing you write by hand at all, it’s the pipeline silently inserting “Per our refund policy document, section 4.2: unopened items may be returned within 60 days” above the user’s question, every time that question comes up. A fine-tuning fix looks like a training example pair, thousands of them, showing the model the input and the exact output you want, submitted as a batch job that adjusts weights and produces a new model checkpoint you then deploy. Only the last one survives independently of what you send it at request time; the other two disappear the moment you stop supplying them.

PromptingRAGFine-tuning
What changesNothing about the model, only your instructionsNothing about the model, only what it reads before answeringThe model’s internal parameters, through further training
Typical setup costEffectively noneLow to moderate: retrieval infrastructure to build and maintainHighest: curated training data, compute, and evaluation
Time to a first resultMinutesDays to a few weeksWeeks, and that’s before evaluation
How current is the knowledgeAs current as what you paste in yourselfAs current as the source you connect it toFrozen at training time
Traceable to a sourceNoUsually, yesNo
Ongoing maintenanceLow, mostly prompt upkeepModerate: keep the index freshHigh: retrain when the base model updates or drifts

A simple sequence of questions to work through

Here’s the practical version of the framework, in the order you should actually ask it. Each question is designed to rule a lever in or out before you spend real engineering time on it.

1. Is the model capable of this, but answering wrong, badly formatted, or in the wrong voice? If yes, that’s an instruction problem. Rewrite the prompt: be more specific, add a worked example of the output you want, add explicit constraints (“never recommend a competitor,” “respond in under 100 words”). Most quality problems people blame on “the AI” are solved right here, and it’s worth genuinely exhausting this step before moving on, because it’s free.

2. Does the correct answer depend on information the model was never trained on? Private company data, anything published after its training cutoff, or anything simply too niche to have been well represented in public training data. If yes, that’s a knowledge problem, and it points at RAG: connect the model to the actual source of truth instead of hoping it memorized the right thing.

3. Have you genuinely tried both of the above, and the model still gets something structurally wrong, not because it’s missing facts, but because its behavior itself needs to change? This is rare. It usually shows up as: the task is extremely narrow and repeated at very high volume, the required output format is so specific that re-explaining it in every prompt is burning real token cost, or you need meaningfully lower latency and shorter prompts than a fully-instructed general model can give you. If this genuinely describes your situation, fine-tuning starts to earn its cost.

4. Almost always: don’t stop at one. The strongest real systems stack these. A well-designed prompt defines the model’s behavior and guardrails, RAG supplies the facts it doesn’t have, and fine-tuning, when it’s used at all, is a narrow, targeted layer added on top of both, not a replacement for either.

A decision flowchart showing three questions in sequence: does the model just need better instructions, then use prompting; does it need facts it wasn't trained on, then use RAG; is the task narrow, high-volume, and stable, then fine-tuning starts to earn its cost; otherwise combine all three

Work through the questions in order. Each one rules a lever in or out before you spend real engineering time on it.

How to actually test which lever you need, instead of guessing

The diagnostic questions above will point you in a direction, but you don’t have to take them on faith. You can test which lever you actually need in an afternoon, with no infrastructure beyond the API you’re already calling.

Build a small, honest test set first. Pull 20 to 30 real questions or tasks your users have actually asked, not the easy ones you’d pick to make a demo look good. Include the ones that currently embarrass the system. This set is the thing you’ll re-run after every change, so its quality matters more than its size.

Run it against a careful prompt, with nothing else changed. No retrieval, no fine-tuning, just your best system instructions and a few good examples. Score each answer and, more importantly, categorize each failure: wrong tone or format (instruction problem), confidently wrong or missing fact (knowledge problem), or wrong in some way that isn’t about tone or facts at all (a genuine behavioral gap). Most teams find the first pass is dominated by the first category, which is good news, since it’s the cheapest to fix.

Manually paste the correct answer in and re-run the failures. This is the single most useful five minutes in this whole process. For each question the model got factually wrong, paste the actual correct information directly into the prompt yourself, right above the question, and ask again. If it gets it right once the fact is sitting in front of it, that’s not a fine-tuning problem, no matter how tempting a custom model sounds. It’s a retrieval problem, and RAG’s whole job is automating exactly what you just did by hand.

Only after that, evaluate whether anything left over is structural. Once instructions are sharp and retrieval is supplying the right facts, whatever failures remain, and are narrow, high-volume, and stable enough to be worth solving permanently, are your actual candidates for fine-tuning. In most projects, this remaining category turns out to be small or empty, which is exactly what you’d expect given that only 9% of enterprise deployments rely primarily on fine-tuning in production.

When prompting alone genuinely gets you there

It’s easy to underrate prompting because it feels too simple to be a “real” solution. In practice, well-engineered prompting, especially when combined with tool calling (letting the model query a live API rather than relying on memorized facts), covers a surprising amount of ground on its own.

Klarna’s AI customer service assistant is a good real-world anchor here. Built on OpenAI’s models and launched in February 2024, it handled two-thirds of Klarna’s customer service chats in its first month, doing the equivalent work of 700 full-time agents, cutting average resolution time from 11 minutes to under 2, and reaching customer satisfaction on par with human agents. The heavy lifting wasn’t a custom-trained model. It was extensive prompt and system-instruction design layered on top of tool access to live account data, such as transaction history and payment schedules, through internal APIs. That’s prompting and tool-calling doing genuinely serious work at real scale, not a toy example.

Reach for prompting first, and take it seriously, when:

  • The model already has the general knowledge or skill; it just needs clearer instructions, examples, or constraints
  • You need to iterate fast, ship this week, and adjust based on real usage
  • The task benefits from live, structured data (account balances, order status) more than from unstructured document search, which points toward giving the model tool access rather than a retrieval pipeline
  • Budget and team size don’t support standing up new infrastructure right now

The limits show up fast, though. Prompting has no memory of its own between sessions unless you build that separately, every fact you want the model to use has to be supplied by you at request time, and very long, fact-heavy prompts get slow and expensive quickly. That’s the ceiling RAG is built to raise.

When the problem is what the model doesn’t know

If you’ve tightened the prompt and the model is still confidently wrong about facts, or worse, confidently wrong about facts you can trace to a specific document it should have used, you’re not looking at an instructions problem anymore. You’re looking at a knowledge problem, and that’s what retrieval-augmented generation was built to solve: search a knowledge source for relevant material, attach it to the prompt automatically, then let the model write its answer grounded in what it just read.

Harvey’s system, again, is instructive here. Its production pipeline runs “RAG systems that incorporate public or user-provided data” as a core layer, not a bolt-on, specifically because legal answers need to be traceable to an actual case or clause, not just statistically plausible. That traceability, being able to point at the exact passage an answer came from, is one of RAG’s biggest practical advantages over both prompting and fine-tuning, neither of which can show its work in the same way.

Reach for RAG when:

  • The answer lives in a specific document, database, or knowledge base the model was never trained on
  • The information changes regularly (pricing, policies, inventory, this week’s news) and staying current matters more than baking in a permanent behavior
  • You need to show users or auditors exactly where an answer came from
  • Your knowledge base is too large or too sensitive to paste into every prompt, even with a large context window

One caveat worth being direct about, because it’s a genuinely common misconception in 2026: a bigger context window is not a replacement for RAG. Frontier models now advertise context windows up to 1 million tokens, and it’s tempting to think you can just paste your entire knowledge base into every request instead of building retrieval. Two problems with that: cost and latency scale with every token you send, on every single request, and research on long-context recall has repeatedly found that models retrieve information less reliably when it’s buried in the middle of a very long prompt, even when the model’s advertised limit is far larger than what you’re sending. RAG keeps each individual request small, cheap, and focused on the handful of passages that actually matter, which is a different and usually better trade than “send everything and hope.”

When fine-tuning actually earns its cost

Fine-tuning is the most misunderstood of the three, partly because it’s the most technically impressive-sounding and partly because the ground under it has genuinely shifted in the last two years.

Here’s the honest 2026 state of play: OpenAI announced in May 2026 that it’s winding down its general-purpose fine-tuning platform, with new organizations already unable to start fine-tuning jobs, and existing customers losing the ability to create new ones by January 2027. Anthropic has never opened fine-tuning for its frontier Claude models through its own API at all; its fine-tuning offering is limited to the older, smaller Claude 3 Haiku model, and only through Amazon Bedrock. That’s a deliberate choice, not an oversight: bigger context windows, better retrieval, and steadily more capable base models have eaten into the specific use cases fine-tuning used to be the only answer for.

None of that means fine-tuning is pointless. It means the set of problems it’s genuinely the right tool for has narrowed to something more specific: tasks that are narrow, extremely high-volume, and stable enough that the pattern won’t change week to week. GitHub Copilot Enterprise is a clean example still in active use: it fine-tunes a model on a customer’s own codebase using LoRA (Low-Rank Adaptation), a technique that trains only a small set of additional parameters rather than the full model. That’s exactly fine-tuning’s sweet spot: one company’s code style and internal API patterns, repeated across millions of completions a day, where shaving even a small amount of prompt length and latency compounds into a real, measurable saving.

LoRA and similar parameter-efficient methods are worth understanding specifically because they’ve made the fine-tuning that does still make sense meaningfully cheaper than it used to be. Rather than retraining every one of a model’s parameters, the original 2021 LoRA paper by Hu et al. showed that injecting small, trainable low-rank matrices into a frozen pretrained model can match much of full fine-tuning’s benefit while training a tiny fraction of the parameters, with no added cost at inference time because the small matrices merge back into the base model’s weights. It’s the reason fine-tuning didn’t disappear even as it became a smaller slice of the overall picture.

Reach for fine-tuning, carefully, when:

  • The task is narrow and repeated at genuinely high volume, not a handful of edge cases
  • The required behavior is stable and won’t need retraining every few weeks
  • You’ve already tried prompting and RAG, and the gap is about how the model behaves, not what it knows
  • Shorter prompts and lower latency at scale are worth real money to you, and you can measure that

And go in with your eyes open about the costs that don’t show up in a training-cost estimate. OpenAI’s own guidance sets 10 examples as the technical minimum for a supervised fine-tuning job, with meaningful improvements typically starting around 50 to 100 well-crafted examples, and quality in, quality out applies ruthlessly here. You also inherit an ongoing maintenance obligation that prompting and RAG don’t carry in the same way: a fine-tuned model is tied to a specific base model, and when that base model is eventually deprecated, as OpenAI’s own deprecation policy makes explicit, your fine-tuned version is deprecated with it. Retraining on top of the new base isn’t optional; it’s the price of staying on a supported model at all.

Prompt caching: the cost argument for fine-tuning just got weaker

There’s a fourth lever worth knowing about, not because it’s a replacement for any of the three above, but because it quietly undercuts one of fine-tuning’s classic justifications: shorter, cheaper requests.

Both major providers now offer prompt caching, which stores the unchanged part of a long prompt, your system instructions, a lengthy set of examples, a chunk of retrieved documents, so repeat requests that reuse that same prefix skip reprocessing it from scratch. Anthropic’s documentation describes cache reads costing roughly 90% less than the equivalent input tokens, and OpenAI applies a similar automatic discount to any request that repeats a cached prefix of 1,024 tokens or more, with no code changes required beyond structuring your prompt sensibly.

Why this matters for the fine-tuning decision specifically: one of the strongest arguments for fine-tuning used to be “it lets us send a shorter prompt every time, which saves money at scale.” Caching chips away at exactly that argument, for a fraction of the engineering effort, with none of the retraining obligation. It doesn’t replace fine-tuning for the narrow behavioral cases described above, but if cost is your primary motivation, it’s worth ruling out caching before you rule in a training pipeline.

They’re not mutually exclusive, and the best systems don’t treat them that way

If there’s one habit worth taking from Harvey’s story, it’s this: the company didn’t pick a lane and stay in it. It fine-tuned a specialized embedding model to make retrieval better, ran RAG on top of that improved retrieval, and wrapped the whole thing in carefully engineered prompts and multi-model orchestration, adjusting the mix as better general-purpose models made some of the earlier heavy custom work less necessary.

That’s the pattern worth internalizing more than any single case study. Prompting is essentially always present, since even a RAG or fine-tuned system needs a prompt telling it what to do with what it’s been given. RAG gets layered on top when facts are the gap. Fine-tuning, when it’s used at all, is typically the narrowest and last layer, applied to one specific, well-understood piece of the problem rather than the whole system.

A worked example: one problem, three lenses

Concrete beats abstract, so here’s how this plays out on an actual project: you’re building a customer support assistant for a mid-sized software company.

Start with prompting. Give the model a clear system prompt: your brand voice, your escalation rules, formatting constraints, and a handful of example exchanges. For general questions the model already understands, “how do I reset my password,” “what’s a good default setting for X,” this alone often performs well. Ship it, watch real conversations, and see where it actually breaks.

Add RAG when the gaps are factual. You’ll notice a pattern quickly: it doesn’t know this month’s pricing tiers, it doesn’t know about the bug fixed in Tuesday’s release, it can’t answer a question that’s genuinely specific to your product. That’s your signal. Connect it to your actual help center articles, release notes, and policy documents through retrieval, the same pattern behind “chat with your documentation” tools generally. Now its factual answers are grounded in your real, current content instead of a guess.

Consider fine-tuning only if volume and stability justify it. Maybe you’ve got two years of resolved support tickets and you want every single response to follow an extremely specific structure, tone, and internal shorthand your team uses, at a volume where re-explaining that structure in every prompt is a measurable cost. That’s a legitimate fine-tuning case. If you don’t have that volume, or the format changes often as the product evolves, skip it. A well-maintained prompt plus RAG solves the vast majority of support use cases without it, which lines up with why only 9% of enterprise production deployments in Menlo Ventures’ 2024 survey relied primarily on fine-tuning, versus 51% using RAG.

The same sequence holds up outside customer support, too. Picture a coding assistant instead: prompting handles “follow this team’s style guide and comment conventions,” RAG handles “look up how this specific internal library’s function is supposed to be called before suggesting code that uses it,” matching the pattern behind coding assistants like GitHub Copilot pulling in relevant documentation rather than guessing from stale training data. Fine-tuning only enters the picture at genuine scale, which is exactly why GitHub reserves it for Copilot Enterprise customers fine-tuning on their own codebase, not for the general-purpose product everyone else uses. Same three questions, same order, a completely different domain.

Cost and complexity, compared honestly

A visual comparison of prompting, RAG, and fine-tuning across setup cost, time to results, and how current the knowledge stays, showing prompting as lowest effort and fine-tuning as highest effort and cost

Cost and effort climb sharply from left to right. Most problems are fully solved before you reach the right-hand column.

The honest summary: prompting is nearly free and should be your default starting point for everything. RAG asks for real but bounded engineering investment, building and maintaining a retrieval pipeline, in exchange for keeping answers current and traceable. Fine-tuning asks for the most: curated data, compute, evaluation, and a standing commitment to retrain whenever the base model underneath it changes. That last cost is easy to underestimate until the first deprecation notice arrives.

Who you actually need on the team for each

This is the part that gets skipped in most technical explainers, and it’s often the deciding factor for a small team or a business reader deciding what to greenlight.

Prompting needs someone who understands the task deeply and can write clearly, which is often a product manager, a subject-matter expert, or a support-team lead, not necessarily an engineer at all. The skill is closer to writing a very good instruction manual than to programming.

RAG needs someone who can build and maintain a small piece of infrastructure: a way to chunk and index documents, keep that index updated as source material changes, and wire retrieval into the request pipeline. That’s typically a backend or data engineer, though a growing number of no-code and low-code platforms have made this more approachable than it was even a couple of years ago.

Fine-tuning needs the most specialized team: someone who can curate a genuinely high-quality training set (not just a large one), run and evaluate training jobs, and, critically, own the ongoing task of retraining when the base model changes. This is usually a machine learning engineer or applied scientist, and it’s the main reason fine-tuning remains the rarest of the three in production, not because the training step itself is exotic, but because the surrounding discipline is.

If your team doesn’t have anyone who fits the third description, that’s not a reason to force fine-tuning to work anyway. It’s useful information about which of the first two options you should be leaning on harder.

Where people get this wrong

A handful of mistakes come up often enough to name directly.

Reaching for fine-tuning before exhausting the free option. It’s the most technically interesting of the three, so it’s tempting to reach for it first. It’s also almost always the wrong order. Prompting costs nothing to test; skipping it wastes the cheapest, fastest diagnostic step you have.

Treating RAG as a hallucination cure. RAG substantially reduces the rate of fabricated answers by grounding responses in real documents, but it doesn’t eliminate the problem: a model can still misread a retrieved passage or blend two sources incorrectly. If you haven’t read it yet, our breakdown of why AI hallucinations happen goes deeper on exactly what’s happening when a grounded system still gets something wrong.

Assuming a huge context window makes RAG unnecessary. As covered above, cost, latency, and recall reliability all argue against just pasting your whole knowledge base into every prompt, even when the model technically allows it.

Expecting fine-tuning to teach new facts reliably. Fine-tuning is much better at teaching a model a pattern, tone, or skill than it is at making it reliably recall specific new facts on demand; for that job, retrieval remains the more dependable tool, because it hands the model the fact directly rather than hoping it was absorbed during training.

Skipping the maintenance math on fine-tuning. A fine-tuned model isn’t a one-time cost. It’s tied to a base model that will eventually be deprecated, which means retraining is a recurring line item, not a step you take once and forget.

Assuming a more capable base model already knows your business. Every new model generation gets better at reasoning, writing, and general knowledge, and it’s easy to mistake that for the model knowing your company’s specific policies, prices, or proprietary data. It doesn’t, and never will, no matter how capable the underlying model gets, because that information simply wasn’t part of its training. That gap only closes when you deliberately supply it, through RAG in almost every case.

The bottom line

Prompting, RAG, and fine-tuning aren’t three competing answers to the same question. They’re three different tools built for three different failures: bad instructions, missing knowledge, and behavior that needs to change at a structural level. Work through them in that order, be honest about which failure you’re actually looking at, and don’t be surprised when the real answer, like Harvey’s, like most serious production systems, turns out to be more than one of them stacked together.

The next time someone asks whether you should “just fine-tune it” or “just add RAG,” you now have a sharper question to ask back: what, specifically, is going wrong, missing information, or a behavior baked in wrong? That one question does most of the work of picking the right tool.

Frequently asked questions

Can I use RAG and fine-tuning together?

Yes, and a lot of serious production systems do. A common pattern is a fine-tuned or specially trained retrieval model that finds the right documents, paired with a strong general-purpose model that reads them and writes the answer, all wrapped in carefully engineered prompts. Legal AI company Harvey has described exactly this kind of layered approach in its own product documentation.

Now that context windows are a million tokens, do I still need RAG?

Usually, yes. A bigger context window means you can fit more text into one request, but you still pay per token, latency still climbs with prompt length, and research such as the 'Lost in the Middle' study has repeatedly shown that models recall information less reliably when it's buried in the middle of a very long prompt. RAG keeps each request small, cheap, and focused on the few passages that actually answer the question.

Is fine-tuning going away?

It's shrinking, not disappearing. OpenAI is winding down its general-purpose fine-tuning platform, and Anthropic still doesn't offer fine-tuning for its frontier Claude models through its own API. But it hasn't vanished: GitHub still fine-tunes Copilot Enterprise on a customer's own codebase, because that's exactly the narrow, high-volume, stable pattern fine-tuning is good at.

Which of the three is cheapest to start with?

Prompting, by a wide margin. It costs nothing beyond the API calls you're already making and you can test a new approach in minutes. RAG requires building and maintaining a retrieval system, which is a real but moderate cost. Fine-tuning requires curated training examples, compute, and ongoing retraining, which makes it the most expensive of the three to set up and keep current.

How do I tell if my chatbot's mistakes are a knowledge problem or an instruction problem?

Paste the correct answer directly into your prompt yourself, right above the question, and ask again. If the model gets it right with the answer sitting right there, it's a knowledge problem, and RAG is what you need to automate getting that answer in front of it. If it still gets it wrong, or ignores the information you gave it, that's an instruction-following problem, and better prompting is the place to start.

Do I need machine learning expertise to fine-tune a model?

Less than you'd think for parameter-efficient methods like LoRA, which many providers now offer through a managed interface rather than raw training code. But you still need a genuinely well-curated set of training examples, a way to evaluate whether the fine-tuned model actually improved, and a plan for retraining when the underlying base model is eventually retired. That operational overhead, more than the training step itself, is what makes fine-tuning the heaviest of the three options.