Quick take
Fine-tuning is the sledgehammer everyone reaches for when a screwdriver would do. Push prompting until it actually breaks. Then – and only then – consider training.
I’ve had this conversation four times in the last month. A team gets GPT-4 producing decent results, notices some inconsistency, and immediately asks: “Should we fine-tune?” The answer is almost always no. Not because fine-tuning is bad, but because they haven’t exhausted the cheaper option yet.
At a financial infrastructure company we went through this exact decision. We needed the model to produce structured financial output in a very specific format. My first instinct was fine-tuning. We ended up solving it with a tighter prompt and a JSON schema constraint. Took an afternoon instead of two weeks.
What prompting actually gives you
Prompting changes the input, not the model. That distinction matters more than it sounds. You can iterate in minutes, roll back instantly, and experiment without any infrastructure. The model stays general-purpose, which means one deployment serves multiple use cases.
A good prompt frame handles most cases:
Role: <who the model is>
Task: <what to do>
Context: <relevant background>
Constraints: <format, tone, length, hard rules>
Output format: <exact shape expected>
The ceiling is real though. Long prompts get expensive and fragile. Behavior can shift with minor rephrasing. And you’re always one API change away from regressions.
What fine-tuning actually gives you
Fine-tuning adjusts the model weights to match your examples. The output becomes more consistent without needing a long prompt to steer it. Good for: locked-down formatting, consistent voice, domain-specific terminology, or reducing token costs by eliminating verbose system prompts.
But here’s what it doesn’t do: add knowledge. If the base model doesn’t know something, fine-tuning just memorizes your training examples for that case. It’s not RAG. It’s not a knowledge base. I’ve seen teams waste weeks building training sets to teach a model facts that should have been in a retrieval pipeline.
Fine-tuning also means operational overhead. You need clean data, training runs, model versioning, evaluation against a test set, and a deployment pipeline that can handle multiple model versions. That’s real engineering work.
The comparison
| Prompting | Fine-tuning | |
|---|---|---|
| Iteration speed | Minutes | Days to weeks |
| Upfront cost | Near zero | Dataset + training compute |
| Per-request cost | Higher (longer prompts) | Lower (shorter prompts) |
| Consistency | Variable, prompt-dependent | High once trained |
| Flexibility | Easy to adjust per task | Locked to training distribution |
| Knowledge | Inject via context/RAG | Only what base model knows + examples |
| Rollback | Change the prompt | Redeploy previous model version |
| Operational burden | Low | Model versioning, eval pipelines |
| Best for | Exploration, multi-task, RAG | Consistent style/format at scale |
My decision path
This is the framework I use. Borrowed some of it from what worked at the fintech startup when we were doing NLP before it was cool (2017, pre-transformer era – we were doing word2vec and suffering).
- Start with prompting. Write a clear prompt. Test it against 20-30 real inputs. If it works 90%+ of the time, ship it.
- Add examples. Few-shot prompting fixes most consistency issues. Three good examples beat a paragraph of instructions.
- Add retrieval. If the model needs facts it doesn’t have, use RAG. Don’t try to bake knowledge into weights.
- Tighten constraints. Output schemas, stop sequences, and post-processing catch most formatting issues cheaper than training.
- Fine-tune as last resort. When you need consistent style across thousands of requests and prompt length is becoming a cost problem, now the ROI makes sense.
Most teams I’ve worked with stop at step 2 or 3. That’s not a failure. That’s efficiency.
The hybrid approach that actually works
The smartest setup I’ve seen combines all three: a fine-tuned model for the base behavior (consistent tone, format compliance), retrieval for dynamic knowledge, and a light prompt for task-specific instructions. This is what the serious teams in financial infrastructure are converging on.
The fine-tuned model handles the “how to say it” part. RAG handles the “what to say” part. The prompt handles the “what specifically to do right now” part. Each layer does what it’s good at.
Data quality is the real bottleneck
If you do fine-tune, your model is exactly as good as your training data. Garbage in, confident garbage out. Some rules I follow:
- Use real inputs from production, not synthetic examples you wrote at your desk
- Cover edge cases explicitly – the model will learn your distribution’s shape
- Remove contradictory examples (they create confused behavior, not averaged behavior)
- Keep a held-out test set and never touch it during training
- Compare the fine-tuned model against the base model on the same prompts – if the delta is marginal, you wasted your time
What matters
Prompting is the default. It’s faster, cheaper, and more flexible. Fine-tuning is a scalpel for specific consistency problems at scale. The mistake isn’t choosing wrong between them. The mistake is reaching for the expensive option before you’ve tried the cheap one.
Most of the time, a better prompt is the answer. Write a better prompt.