Every few weeks a business owner tells me they want their own AI model. They have read that fine-tuning makes an AI "learn your business," and they are ready to spend real money on it. Almost every time, the honest answer is that they do not need it. The debate of fine-tuning vs prompt engineering is usually settled before it starts, and the winner is prompting.
I have built and shipped enough of these systems to say this plainly: for maybe 95% of small and mid-sized business use cases, you get where you want to go with better prompts and your own documents fed in at the right moment. Fine-tuning is a real tool, but it sits at the top of a ladder most companies never need to climb.
Let me walk you through that ladder, rung by rung, so you can see exactly where your problem sits and what it will cost you to solve it.
The Ladder: Three Rungs, Not Two
People frame this as a two-way fight, fine-tuning vs prompt engineering, as if those were the only options. There are actually three rungs, and you should climb them in order.
- Plain prompting. You write clear instructions and give the model examples of what good output looks like. No infrastructure, no training, no data pipeline.
- Prompting with your documents. You retrieve the relevant policy, product sheet, or past ticket and paste it into the prompt at answer time. This is often called retrieval, and it is where most business value lives.
- Fine-tuning. You take a base model and train it further on hundreds or thousands of your own examples so it internalizes a narrow pattern.
The rule is simple. Start at rung one. Only move up when the rung you are on genuinely cannot do the job, and you can prove it with real failing examples, not a hunch.
Rung One: Plain Prompting Solves More Than You Think
Most "we need a custom model" requests are actually "our prompt is lazy." A one-line instruction like "summarize this" gives you generic output. A prompt that says who the reader is, what tone to use, what to include, what to leave out, and shows two examples of a good summary gives you something you can ship.
Prompting well is a skill, and it is nearly free. There is no monthly training bill, no data collection project, and you can change the behavior in five minutes when your process changes. For drafting emails, classifying incoming messages, rewriting product descriptions, or answering general questions, this rung is often the whole answer.
The failure mode here is impatience. Teams try a weak prompt, get a weak result, and conclude the model is not smart enough. The model is fine. The instructions were thin.
Rung Two: Feed It Your Documents
The moment your problem depends on facts specific to you, your prices, your SOPs, your past conversations, plain prompting hits a wall. The model does not know your refund policy because it was never in the prompt.
The fix is not fine-tuning. It is retrieval: when a question comes in, you find the two or three most relevant documents and include them in the prompt. The model then answers from material it can actually see. This is how a support assistant answers "what is your warranty on this SKU" correctly. It read your warranty page a moment before answering.
This rung covers a huge share of what SMEs actually want from AI: internal knowledge assistants, customer FAQ bots, document question-and-answer. If you want a fuller picture of this pattern, I wrote about it in Chat With Your Data: What It Means for Your Business.
Cost here is modest. You need your documents organized and a retrieval step, but you are not training anything. When your policy changes, you update the document, and the next answer is already correct. That maintainability is the quiet superpower of rung two.
Rung Three: When Fine-Tuning Actually Earns Its Cost
Fine-tuning is worth it in a narrow set of cases:
- High volume, narrow task. You run the same tightly defined operation thousands of times a day, and shaving cost or latency per call matters.
- A specific style or format that is hard to describe but easy to demonstrate with many examples, like matching a house voice across thousands of outputs.
- Consistency at scale where even a good prompt drifts, and you need the behavior baked in.
Notice what is not on that list: teaching the model your latest prices, policies, or facts. Fine-tuning is bad at facts because facts change and retraining is slow and expensive. Facts belong on rung two.
The cost picture is real. Fine-tuning means collecting and cleaning a labeled dataset, running training, evaluating the result, and then doing it again every time the base model or your needs shift. You now own a maintenance burden. A fine-tuned model is not a purchase, it is a pet you feed. That is the same truth I keep coming back to about why finished software still costs money every month.
A Simple Way to Decide
Here is the decision I run with clients.
| Your situation | The right rung |
|---|---|
| Generic drafting, classifying, rewriting | Plain prompting |
| Answers depend on your own documents or data | Prompting with retrieval |
| Answers depend on facts that change often | Prompting with retrieval, never fine-tuning |
| One narrow task at very high volume, fixed for months | Consider fine-tuning |
| You want a hard-to-describe consistent style at scale | Consider fine-tuning |
If your row lands in the top three, you have your answer, and it is cheaper and faster than you expected. If it lands in the bottom two, fine-tuning is on the table, but treat it as a real project with a real owner.
The Takeaway
The fine-tuning vs prompt engineering question almost always resolves in favor of prompting, either plain or with your documents fed in. Fine-tuning is a specialist tool for narrow, high-volume, style-heavy work, and it comes with an ongoing maintenance cost that most SMEs should avoid until they have exhausted the cheaper rungs.
Climb the ladder in order. Get plain prompting right first. Add your documents when facts matter. Reach for fine-tuning only when you can prove the lower rungs failed. If you are trying to figure out which rung your specific use case sits on, that is exactly the kind of question I help founders answer as a technical partner, before anyone spends money on a model they did not need.