I keep watching developers iterate on prompts by hitting GPT-4 hundreds of times a day. Every keystroke, another API call. Every experiment, another line on the invoice. Then they act surprised when the monthly bill shows up.
This is dumb. Not because the hosted models are bad – they are great. But because you don’t need frontier-model quality to test whether your prompt template works, your parsing logic handles edge cases, or your UI renders a streamed response correctly.
Run a local model. Iterate fast. Save the API calls for when you actually need them.
The actual reasons to go local
Forget the hand-wavy “sovereignty” arguments for a moment. The practical reasons are simple:
Speed. No network round-trip. No rate limits. No waiting in a queue behind someone else’s batch job. I can test a prompt change in under a second on a MacBook with Ollama running a 7B model. That feedback loop matters when you’re doing fifty iterations in an afternoon.
Cost. Zero marginal cost per request. I ran through over a thousand prompt variations last month while building an extraction pipeline. On GPT-4, that would have been a few hundred dollars. Locally, it was electricity.
Privacy. Some of my work involves data I can’t send to a third-party API. Full stop. Local inference solves that problem without paperwork.
The trade-offs are real, so stop pretending otherwise
Local models aren’t frontier models. A 7B parameter model running on your laptop isn’t going to match GPT-4 on complex reasoning tasks. That’s fine. You aren’t using it for production quality – you’re using it for development velocity.
Where local models genuinely fall short:
- Multi-step reasoning. They lose the thread.
- Long context windows. Most local models tap out well before 128k tokens.
- Consistent formatting. They drift more on structured output tasks.
- Nuanced instruction following. Subtle prompt changes sometimes get ignored.
If your development workflow requires frontier-quality responses at every step, local models aren’t for you. But honestly, most development workflows don’t. You need a model that’s good enough to validate your integration logic, and local models clear that bar easily.
My actual setup
I keep it simple. Ollama for the runtime, a 7B model as default, and an environment variable to swap between local and remote.
func getLLMConfig() LLMConfig {
if os.Getenv("USE_LOCAL_LLM") == "true" {
return LLMConfig{
BaseURL: "http://localhost:11434",
Model: "mistral",
}
}
return LLMConfig{
BaseURL: os.Getenv("LLM_API_URL"),
Model: os.Getenv("LLM_MODEL"),
}
}
That’s it. The rest of the application doesn’t care which model it’s talking to. The interface is the same, the error handling is the same, the retry logic is the same. When I want to validate quality against the real model, I flip the variable and run my eval suite.
The workflow that actually works
- Develop locally. Prompt changes, parsing logic, UI work, error handling. All against the local model.
- Eval against remote. Before merging, run the same test cases against the production model. Compare outputs.
- Ship with confidence. The integration is tested. The quality is validated. The bill is reasonable.
The key insight: your development model and your production model don’t need to be the same. They need to share the same interface.
When to skip local entirely
Be honest about the cases where local doesn’t help:
- You’re doing few-shot prompt engineering where response quality is the variable you’re testing.
- Your feature depends on capabilities only frontier models have (vision, very long context, tool use with complex chains).
- You’re evaluating model-specific behavior like safety responses or refusal patterns.
In those cases, just use the API. The point isn’t religious purity about local inference. The point isn’t burning money on API calls when a local model would have told you the same thing.
Stop overthinking it
Install Ollama. Pull a model. Point your dev config at localhost. You will iterate faster, spend less, and keep sensitive data on your own machine. When you need the real thing, it’s one environment variable away.
This isn’t complicated. It’s just discipline.