Fine-tuning, Prompt engineering, RAG: Choosing the right technique for your use-case

Every team building with LLMs hits the same fork in the road: do you write a better prompt, bolt on retrieval, or actually retrain the model? Vendors will tell you their approach is the answer. It isn’t. Each technique solves a different problem, and picking the wrong one is expensive in a specific way. Either you burn months fine-tuning a model that a five-line prompt would’ve fixed, or you ship a prompt-only system that hallucinates because it never had the data it needed.

So, let’s break it down properly, with real numbers, so you can figure out where you stand before you write a single line of code.

The three techniques, in one paragraph each

Prompt engineering changes nothing about the model. You’re just getting better at asking: spelling the task out properly in a system prompt, pinning down the output format, showing two or three worked examples so the model can pattern-match instead of guess (that’s few-shot prompting), or telling it to reason step by step before answering (chain-of-thought). It costs close to nothing, takes minutes to implement, and works when the model already has the knowledge you need.

RAG (retrieval-augmented generation) connects the model to an external knowledge source at query time. The model itself doesn’t change; it just gets handed the right documents before it answers. This is the fix for “the model doesn’t know this”: your internal docs, last week’s prices, a customer’s account history.

Fine-tuning retrains the model on your own examples, so it internalizes a pattern, a format, or a domain. In other words, this is the fix for “the model knows the facts but won’t behave the way I need it to,” consistently, at scale, without you re-explaining the rules in every prompt.

Where each one actually wins

Let’s think about it through a real example. A mid-market insurance company needed to classify claims into 47 internal categories, each with its own business rules. Prompt engineering topped out at 78% accuracy. The taxonomy was too specific and too far from anything a general-purpose model had seen. So that’s a fine-tuning problem, not a phrasing problem.

Compare that to a legal tech company sorting contract clauses into 80 proprietary categories. Same shape of problem. They fine-tuned an open-source model on 8,000 labelled examples. The result: faster and cheaper per call than a frontier model API, and 22 percentage points more accurate than prompt engineering alone on their benchmark.

Now flip it around. If what’s missing isn’t behaviour but knowledge, think pricing that changes weekly, a document set that grows every sprint, compliance content that has to cite a real source, fine-tuning is the wrong tool. You’d be baking in facts that go stale the moment you ship. That’s RAG’s job: pull the current version of the truth at query time instead of hoping the model memorized it correctly six months ago.

And a lot of the time, neither is needed yet. If your task is summarization, formatting, or answering questions from information the model already has, prompt engineering solves it in an afternoon. No infrastructure, no training data, no ongoing cost.

What it actually costs you is a different story. The table covers the effort side. The money side is where teams get surprised. A real retrieval pipeline runs roughly $70 to $1,000 a month once you count indexing, vector storage, and retrieval calls. Fine-tuning front-loads the cost into dataset curation and training instead.

And here’s a myth worth killing: fine-tuning doesn’t automatically mean higher inference bills. Running a smaller, fine-tuned open model can actually cut per-query costs significantly compared to repeatedly shipping bloated few-shot prompts to an expensive frontier API.

A rough sequence that holds up in practice

1. Start with prompt engineering. Build a prototype with a frontier model and a carefully written system prompt plus a few examples. Measure it against real test cases. If it clears your bar, ship it, you’re done.
2. Add RAG when freshness or knowledge scope is the bottleneck, not behavior. If the model’s answers are wrong because it doesn’t have the information, retrieval fixes that without touching the model itself.
3. Fine-tune only when you have a measured, specific gap, not a hunch. That means prompting and RAG were both tried, the failure is a consistency or behavior problem rather than a knowledge gap, and you have enough labeled examples (as a reference point, the legal tech case above used 8,000) to justify the training cost.
4. Combine RAG and Fine-Tuning for production scale. In complex enterprise scenarios, it’s rarely an either/or. You fine-tune the model so it learns your domain’s specific vocabulary, edge-case reasoning, and output structure, then feed it live context using RAG. Fine-tuning gives the model the right brain and tone; RAG gives it today’s facts.

Skipping a step is where most of the expensive mistakes happen. Teams jump to fine-tuning to fix a knowledge gap that RAG would’ve solved for a tenth of the cost, or they try to fine-tune their way out of a formatting problem that a better prompt would’ve fixed in ten minutes.

One note on the effort involved, though. That “weeks to months” figure assumes you’re building the whole pipeline yourself: curating the dataset, wiring up training, managing hosting. That’s the part teams consistently underestimate, and it’s the part we set out to collapse at Nebul. AI Studio handles the data preparation and tuning workflow directly, so the timeline stops being about infrastructure plumbing and becomes about how quickly you can get your examples in order.

Why this matters before you talk to us about fine-tuning

None of this is abstract. If you’re weighing whether fine-tuning is the right call for your use case, the answer depends entirely on which of these three problems you have: a phrasing problem, a knowledge problem, or a behavior problem. Get that diagnosis wrong, and the rest of the project is built on the wrong foundation.

That’s exactly where the next piece in this series picks up: a hands-on look at how fine-tuning actually works, and which real-world scenarios justify it.