Every product meeting I’ve been in for the past three months starts the same way: “How can we use AI for this?” Wrong question. The right question is: “What problem are we solving, and is AI the best tool for it?”
I know, I know. That sounds like something a consultant says to justify a strategy offsite. But having built AI features at a fintech company and watched a dozen teams try to do the same, the pattern is clear: teams that start with the model build demos. Teams that start with the problem build products.
The boring problem statement trick
Here’s a filter I’ve started using. Take your AI feature idea and describe it without mentioning AI, ML, LLMs, or GPT. If what’s left sounds like a useful product improvement, you probably have something. If what’s left sounds empty, you’re building a solution looking for a problem.
“We use GPT-4 to generate personalized financial insights” becomes “We show users relevant patterns in their spending.” The second version is a product. The first version is a technology demo. The second version also tells you what to measure: are the patterns actually relevant? Do users act on them?
At a fintech company, we almost fell into this trap. The initial pitch was “AI-powered transaction intelligence.” That means nothing. We reframed it as “reduce the time finance teams spend manually categorizing transactions from 4 hours to 20 minutes.” Now we had a measurable goal, a clear user, and a way to know if it worked.
Where AI actually helps
AI is good at tedious, pattern-heavy work where approximate answers are acceptable. Classification, summarization, drafting, extraction. It saves humans from the work they hate doing and are bad at doing consistently.
AI is bad at precision, accountability, and reasoning about edge cases. It’s bad at knowing what it doesn’t know. It’s bad at anything where “usually right” is worse than “always following a rule.”
The sweet spot for AI features in most products right now: draft-and-review workflows. The AI produces a first pass. The human reviews, corrects if needed, and approves. Both sides do what they’re good at.
The failure pattern: giving the model autonomy over decisions that matter. Auto-categorizing low-value transactions? Fine. Auto-approving expense reports over $10K? No. The line is about consequences. If the model gets it wrong, how bad is it? If the answer is “not great,” add a human checkpoint.
The five questions that kill bad ideas early
Before a team starts building an AI feature, I make them answer these. In writing. No hand-waving.
- What user problem is this solving? Not “what could we do with AI” but “what is painful today, and who feels that pain?”
- What does a wrong answer cost? If incorrect output has serious consequences, you need guardrails and fallbacks before you need features.
- How do we measure success? Not usage. Not engagement. What concrete outcome improves? Time saved, errors reduced, revenue impacted.
- What happens when the model is unavailable or uncertain? Your feature needs to work without AI. If it can’t, it’s too tightly coupled to a service you don’t control.
- What’s our edge over someone else calling the same API? If the answer is “our prompts,” you don’t have an edge. Proprietary data, workflow integration, and feedback loops create defensibility. Prompt engineering doesn’t.
If any answer is weak, run a time-boxed experiment instead of a full build. Two weeks, clear success criteria, then decide.
The moat isn’t the model
I wrote about this in July, but it bears repeating because I keep seeing it: access to an API isn’t a competitive advantage. OpenAI’s API is available to everyone. So is Anthropic’s. So are the open-source models.
Your moat is the system around the model. The data you collect from user interactions. The workflow integration that makes ripping you out painful. The quality metrics that let you improve faster than competitors. The fallback system that means your product works even when the model doesn’t.
At Google for Startups in Seoul, I watched a batch of companies try to build moats around technology that was commoditizing in real time. The ones that survived had distribution or data advantages. The ones that died had clever technology and nothing else. Same pattern applies here.
Ship the ugly version
The final piece of advice is the least popular: ship something simple and ugly before you ship something impressive and polished. A feature that works reliably for 80% of cases with a clear fallback for the other 20% beats a feature that works brilliantly for 95% of cases and fails catastrophically for the other 5%.
Users forgive limitations when the product is honest about them. They don’t forgive confidence that turns out to be wrong.
Build the product. Not the demo.