What the machines give back is not room to ship more. It is the chance to attempt work that used to need a company, to serve people the old cost floor priced out, and to spend a working life on the part of the work that is worth a life. A company that spends the surplus on volume has been handed a decade and bought a quarter with it.

Everything here turns on one thing. The work is how people get made. People are not how the work gets made.

What I value

Verifying over producing.

Earned autonomy over granted autonomy.

Enforced limits over declared intent.

Doing the work over approving it.

Judgment that compounds over output that repeats.

Apprenticeship over throughput.

Problems surfaced over progress reported.

Independent sources that can disagree over one that cannot be wrong.

I want the right column too. What moved is where the scarcity sits.

How I work

A process nobody can check is not automated. It is unsupervised. The check comes before the thing it protects: the test before the pipeline, the alarm before the deploy, the dry run before the announcement.

Verification rises with the consequence of being wrong. Produce enough to learn from someone real, verify enough to trust an act that repeats or cannot be taken back, and where the work decides nothing, do neither.

Autonomy is earned with evidence and taken back the same way. For machines it attaches to a step. For people it widens across a domain as their judgment earns it, and narrows at the smallest place the evidence indicts. Nobody should live permanently on probation. And evidence expires: what earned trust last year describes last year’s system, and nothing announces the change.

Nothing is done when it works. It is done when something checks it, an owner carries the outcome, and the way back has been walked at least once. A rollback nobody has rehearsed is a rumour.

Some work has no decisive check at a price anyone will pay. Mark it, keep people on it, and never let a number stand in for what cannot be measured.

Nobody is the only source of evidence that they are doing the thing, and that includes the machine that did it. You cannot feel whether the machine helped: work goes faster in the hand and slower on the clock, and nobody has trained that out of anyone.

Producing the work can be handed to a machine. Answering for it cannot. The machine wrote it in a second and somebody carries it for a year.

Deciding to kill something is not killing it. It is alive for exactly as long as its permissions still work.

Measure the system in public. Measure the person in private, and publish who did the measuring, on what evidence, and who can overturn it. Pay and promote for judgment under uncertainty, for spreading what works, and for carrying what does not.

What people are for

Every time a machine takes the making from someone, they are owed a harder question and not a smaller job. If the harder question does not come with more standing, it was a smaller job with better words.

Beginner’s work is what made the expert, and it is disappearing. Give that work to beginners anyway, because a trade that admits nobody has stopped being a trade. A company that ships more this year and can make fewer experts than last year has eaten capital and called it margin. No dashboard you already have will show it, so count the people you took from beginner to independent and publish that beside what you shipped.

Refuse work whose value depends on weakening the agency of the people it touches. Such work can be profitable and perfectly controlled. Whether its success is worth causing is the question, and it is asked before the controls and not after. Between two things you could build and defend, build the one that leaves people more able to act without you.

Anyone a decision lands on can reach a person who can explain it and change it. Not because the machine is often wrong, but because being processed by something that cannot be asked is a harm at any error rate.

If the work is handed to you rather than by you, all of that is what somebody owes you: work that can go wrong, a harder question when a machine takes the making, a name and a route when a decision goes against you. Ask for them by name. Asking is not insubordination, and if it is treated as insubordination where you are, you have your answer on your first day instead of in your third year.

Where to start

None of this needs agreeing with first. Four things are doable this week, and each hands back its own evidence.

Ship one change only after a check you did not write.

Before you let anything run on its own, walk the way back once, by hand. You will find a rollback or you will find a rumour.

Find something you declared dead last year and try its permissions. Whatever still answers was never killed.

Put a name and a route on one automated decision that lands on people, and ask the ones it went against whether they were told anything they could use.

Then stop doing them as an audit, because an audit is a thing you pass. Each one is fastened to a moment in the work and comes back when the moment does: a check you did not write, every time something ships whose failure would cost you; the way back walked, every time something is trusted to run alone; the permissions tried, every time something is retired; a name and a route, every time a decision lands on a person.

How you would know I am wrong

Every line above should be a bet, and a bet that cannot lose is not a bet. A test has to be losable by the line being wrong rather than by the world moving, so a clause that fires when the machine improves is a timer and not a bet. These five are mine. Where a line here has no test yet, that is a debt I owe and not a privilege I am claiming.

Verifying is misplaced when the work you checked went wrong as often as the work you let through.

Earning is theatre when what you trusted on sight held up as well as what you made climb for it.

Apprenticeship is sentiment when people who were never given work that could go wrong turn out able to make the same calls unaided as the people who were.

Refusing is posturing when the work you came closest to turning down, and built anyway, left the people it touched more able to act than your safer work did.

The reachable person is a courtesy when the people who had a decision go against them used it and said the wait was not worth it.

When one of these arrives, the failure counts against the line and not against you for having believed it. Drop it and say so.

How this dies

The last manifesto did not die. It won, and was hollowed out, which is worse. It spread as vocabulary while behaviour stayed home, and its heirs made it immune, so no failure could count against it. A method that explains every failure as incomplete adoption has arranged never to learn.

So if these words arrive and no decision changes, they failed there. And if most of this does not survive contact with your organization, it was wrong about your organization. A manifesto that cannot be wrong is not a manifesto. It is a liturgy.

Correct me rather than quote me.

The lead time does not disappear when you outsource it; it reaches you renamed. A quota request that takes weeks and comes back denied. A reserved-capacity contract your provider will only sign for the region where the power already exists. A new accelerator generation that is available in three regions and waitlisted in the rest. Scarce capacity lands first where the grid is already built and trickles outward from there, so the map of where you can actually run is the map of where the power already sits. You did not secure that power; you inherited its schedule, and that schedule is unforgiving in a way software schedules are not. Assume the GPUs will be there and you have assumed away a constraint nobody upstream can wave away either.

Here is the shape of the miss. A team commits a 5x ramp to the board: volumes, latency targets, a confident curve. The model works in the demo, the math closes in the spreadsheet. Then the data-residency commitment pins the workload to one region, the provider’s on-demand quota in that region is capped, the reserved-capacity deal that would lift the cap is a negotiation measured in months and multi-year terms , and the ramp you already sold slips two quarters. That is not a forecasting miss. It is a dependency you mislabeled as an assumption.

So price it like a dependency. On the roadmap, every committed ramp carries, next to its volume and latency targets:

• the region it actually lands in, and the residency rule that pins it there;
• whether that capacity is on-demand (no guarantee) or reserved (contracted), and the lead time to convert one to the other;
• the named owner of the provider relationship who can escalate a quota or open a region;
• the fallback region, with the latency and residency cost of failing over to it.

The reserved-capacity line is the nearest thing you hold to a power-procurement order: contracted capacity you claim well ahead of need, not when the quota is already capped. That is how a roadmap survives contact with reality : it names the capacity constraint before the board discovers it.

Now stop spending the scarce capacity on work that does not need it. The premium path, frontier reasoning on the newest accelerators in the few regions that have them, should be rationed by design. Most production volume is high-frequency, low-stakes work that runs fine on a smaller model, a batched job, a cached result, or last-generation hardware that is abundant precisely because everyone else moved on. Routing that volume down buys back ramp without buying more capacity, and it is the one lever here you pull directly rather than petition a vendor for.

What remains is geography and timing. Workloads can move to where capacity is cheap and available; the commitments you made to customers usually cannot move as fast. So choose regions deliberately, hold a qualified fallback for each committed workload, and treat residency rules as the hard constraint they are rather than a detail to reconcile later. The failure mode is always the same: one assumed region, one assumed quota, one assumed timeline, and nothing staged for when any of the three moves. Stage the move before the cap forces it, because by then the cap is not negotiable and your customers already have the date.

Price the same million output tokens three ways before you wire a workload to a model vendor: their API, a competitor’s API, and the same job self-hosted on an open-weight model of comparable quality, billed by the GPU-hour. You will never see the vendor’s balance sheet. You do not need it. You need a floor.

Scale buys real efficiency, so a price below your own self-host cost proves nothing; a large operator running batched, high-utilization fleets genuinely serves tokens cheaper than you can. Set the floor lower, then: the cost a well-run fleet at full utilization would carry, and compare against that. When the offered price sits below even that number, no efficiency story closes the gap. The discount is being financed by someone, and financing gets withdrawn. That spread between their price and the floor is the subsidy you will repay later, on their schedule.

This is the diligence that works, because the diligence usually prescribed does not. “Ask what funds the price, durable margin or capital that must be raised again” is a question a private vendor will never answer honestly and you cannot audit. The price-versus-floor spread you can compute yourself, repeat each quarter, and act on.

Repricing arrives as a deprecation notice

It rarely shows up as “we raised prices.” It shows up as a sunset date. The cheap model you built on is scheduled for retirement, and the supported replacement is a reasoning model that emits three to five times the tokens to answer the same prompt. The headline per-token price holds or even falls, and your cost per call still triples, because the only model you are now allowed to call thinks out loud before it answers. Token prices fell; bills did not is the same mechanism viewed from the invoice. The hit lands on your unit economics and your roadmap in the same week, when you have the least slack to absorb it.

Two clauses, written at signing

Two terms cost nothing at signing and cannot be retrofitted during an outage: a price lock paired with a minimum notice window on any model deprecation, and an explicit right to export your fine-tunes, prompt logic, and request logs in a portable form. You are not negotiating to be reassured. You are negotiating for terms that hold through the vendor’s bad quarter.

The control is a warm fallback with an owner

Comparisons and clauses narrow the risk; neither removes single-vendor exposure on a critical path. One thing does: a warm fallback, a second provider or that self-hosted model, kept able to carry the workload at degraded quality on short notice. Treat it as a build-versus-buy portfolio rather than a binary , optionality you fund on purpose, not a hedge you improvise mid-incident.

Warm means it runs. Route a thin slice of real production traffic through the alternate on a fixed cadence, weekly rather than when someone remembers, so the path stays exercised and you know its true error rate before the day you need it. The question it answers is blunt: the morning your largest model vendor posts a shutdown notice, how many days until your product runs on something else? That redundancy costs real money and competes for budget, which is the point: redundancy you can measure is a line a board can fund.

So give the line an owner. On-call carries the failover; conversion carries the dependency that makes it necessary, and will keep deepening it until someone with budget authority puts the fallback above the next feature. Until that owner exists, the vendor sets the answer to the shutdown-notice question, on the quarter that suits them.

None of that is new, which is the actual problem. “Build your own evals” is the most recycled advice in this field, and plenty of teams take it, stand up a set of real cases, and quietly stop trusting it within a quarter. The failure is never collecting the cases. It is the part everyone skips: who scores them, on every change, without going broke or going blind.

The cost trap

Pull thirty to a hundred real cases from your logs and weight them toward the unhappy paths. That part is easy. Then someone has to grade the outputs, and if that someone is a human reviewer working every model, prompt, and routing change, the set runs once a quarter and becomes a ritual instead of an instrument . Re-running a hand-graded set on every change is the cost objection that kills most owned-eval programs. It is real, and “unglamorous and cheap” does not answer it.

Tiering does. Most cases have a checkable answer: a required field, a JSON shape that must parse, a string that must or must not appear, a number within tolerance. Encode those as programmatic assertions. They are deterministic, free, never drift, and run in CI on every commit. In practice sixty to eighty percent of a real case set collapses to assertions. Only the residue needs judgment: was the summary faithful, was the refusal correct.

For that residue, use an LLM as judge, but treat the judge as versioned code, not a vibe. Pin the judge model, freeze the rubric, anchor it with a few labeled examples in the prompt. Now the expensive layer costs cents per case and runs on every change. Humans grade the grader, not the traffic: keep a small gold set you labeled by hand, and on a schedule (and whenever the judge’s model version moves) measure judge-versus-human agreement. When agreement slips, fix the rubric before you trust another run. Auditing the judge instead of the model is what makes the cheap layer safe to lean on.

The noise trap

A set of thirty cannot tell a four-point regression from sampling noise. At an eighty percent pass rate over thirty cases, the standard error on the aggregate is about seven points, so any single-digit move is meaningless. You will chase ghosts or ship regressions while admiring a stable number. The aggregate is the wrong statistic. Run the old and new model on identical cases and read the per-case flips: which cases passed before and fail now. Three clean regressions out of thirty is a signal even when the headline barely moves, because each case is compared to itself rather than to an average. Run each case a few times so model nondeterminism does not masquerade as a change. Pairing, not volume, gives a small set teeth.

What it buys

The payoff is narrow and concrete. A new model tops a public set, clears your shortlist, and your eval shows two regressions on the exact request class that quietly generates support tickets. You hold it back, with receipts, and you can tell the board why the number did not move on your terms instead of the vendor’s. That is the whole return: not a better score, an argument you can defend. Separate it from cost per correct answer , which is a different decision, and the eval set becomes the one measurement a vendor cannot tune.

Make the honest claim instead, because it is the one that survives the room. Customer-held keys do not make you subpoena-proof; a service that decrypts on request can be compelled to capture what it decrypts. What BYOK actually buys is narrower and more defensible: the customer holds the kill switch, every key use is logged on their side, and every copy at rest stays opaque without their KMS. That is revocation leverage and an audit trail, not cryptographic immunity. Sell the first as the second and the auditor stops believing your other answers too.

The fix for the questionnaire is not a longer policy. It is an architecture where the answers are forced by construction. Trace one record through it.

A customer record hits ingestion and is classified before it is stored: tagged with a data class at the boundary, the tag traveling as metadata through every downstream system. Untagged data does not pass. It is encrypted under the customer’s key, held in their KMS. When the application needs plaintext, to build a retrieval context, it requests a short-lived scoped token, decrypts in the record’s own region (data tagged EU-resident never reaches non-EU hardware), uses the plaintext for that one request, and drops it. A no-train flag rides along, enforced in the pipeline so the record is excluded from fine-tuning and eval sets by code, not by a reviewer’s good intentions.

Deletion is where most sovereignty pitches quietly contradict themselves, promising both instant crypto-shredding and a multi-store delete fan-out as if they were one mechanism. They are two, and they cover two different populations.

Everything encrypted under the customer key (primary store, snapshots, the immutable backups you cannot selectively edit) is handled by revocation. You do not run a delete job against an append-only backup; you can’t, that is what append-only means. Revoke the key version and the ciphertext is unrecoverable in place, backups included, precisely because you never needed write access to them.

Revocation does nothing for plaintext that escaped the envelope: lines in application logs, embeddings derived from decrypted text, a copy a subprocessor holds under its own key. None were encrypted under the customer key, so none shred with it. Those need an actual delete, per store, each returning a signed receipt with its timestamp and key version.

So the engineering goal is to shrink that second population until it nearly vanishes: log token references instead of plaintext, encrypt the vector index under the same customer key so embeddings die on revocation, push subprocessors onto the same custody so their copies do too. The more your deletion story is “revoke one key,” the less of it is “trust that six jobs ran.”

This is also why sovereignty is cheaper built early than retrofitted. Key custody, in-region routing, and a small escaped-plaintext surface are load-bearing assumptions. Bolt them onto a system that already copies data freely and you are rewriting your data plane under a customer’s deadline. Treat residency and custody as non-optional from the first commit , and governance becomes something you exhibit rather than a process you defend.

The deletion story is the part an auditor can actually test, and the test is one record’s deletion evidence. If you cannot say which copies revocation kills and which copies it doesn’t, neither can your buyer’s auditor, and that gap, not the questionnaire, is why the deal stalls. “We will delete your data on request” is a sentence in an MSA. The answer that clears the room has the evidence attached to it: these two stores still hold plaintext derivatives, here is each delete receipt, and everything else went dark when we revoked key version 47.

Start with what the strategy deck stacks up as defensible. Three layers, usually. The model itself, rented, and your competitor can rent the same one. The scaffolding on top of it: prompt library, routing logic, eval harness, all shaped around one vendor’s behavior and all of it breaks the morning you change providers. If the moat disappears when the vendor changes, it was never a moat. It was a dependency. That clears two of the three layers off the slide. Swap the provider in your head; whatever still works the next morning is the only candidate worth the word.

What survives is the third layer, and it is the one boards cannot tell apart from its imitation: data your own operation produces by running. Here is the mechanism, and the exact place it breaks.

Take support automation. The model drafts a resolution; a human approves before it ships. Every rejection or rewrite captures a labeled triple: the input, the output the model produced, the output the human accepted. Not a log line. A graded example of where your model was wrong and what right looked like, on your tickets, in your domain.

Now the load-bearing step, the one most decks wave through. How does that triple make a cheaper model tier handle a class it used to escalate? Two mechanisms, two different bills.

Retrieval, the few-shot route: index the accepted exemplars and inject the nearest ones into the prompt at inference. Cheap to stand up, live the moment you index a correction, but it taxes every call in tokens and latency, and the lift is capped because you are renting the base model’s in-context learning.

Distillation, the fine-tune route: train the small model on the correction set. Latency stays flat, the behavior is baked in, per-call cost drops, but you pay a training-and-eval cycle up front and re-pay it on every base-model upgrade. Which tier absorbs which class is a cost decision, not a model-quality one: retrieval for the long tail, distillation for the high-volume classes once they stop drifting.

Either way, one number tells you which you have: escalation rate on a single request class, quarter over quarter. Falling and sustained while quality holds is the loop compounding. Flat is a warehouse with a dashboard bolted to it. A logging pipeline and a compounding loop look identical in the architecture diagram and behave nothing alike in the P&L.

I cannot hand you a rival’s P&L to prove the good case, and any deck that shows you a clean before-and-after percentage is selling the illustration as the evidence. The honest test is one you run on your own numbers: name the class, name the two quarters it improved, name why a competitor on the same vendor cannot reproduce it. The answer to the last one is never the model. It is the system around it that turns each rejection into an exemplar only you hold.

Then the part the optimistic version omits: this asset depreciates. When the next base model ships, it absorbs your easy classes for free, everyone’s, not only yours, and that compresses the set of failures where your corrections still move the number. Your edge is only ever the residual: corrections illegible outside your context, your product’s quirks, your contractual edge cases, your policy language. The vendor will productize the capture loop; they already sell feedback buttons and fine-tuning APIs. What they cannot aggregate is a residual that means nothing without your business wrapped around it. The loop compounds only while you generate domain-specific corrections faster than a better base model erases the generic ones.

A beautiful demo is not a business model. It only proves the model can look useful before the business pays for edge cases. The bill arrives when the system hits real users, real load, and real failure conditions. At that point AI stops being a model-selection problem and becomes a routing problem , a fallback problem, and a repair problem. Good CTOs do not buy “smart.” They buy systems that stay cheap enough, predictable enough, and reliable enough to survive the week.

Unit economics start with routing

The wrong AI architecture sends every request to the most expensive path. That feels elegant until the invoice arrives. Mature systems route by value and by risk.

A practical routing model usually splits work into classes:

• trivial tasks that should stay cheap and local
• medium-value tasks that deserve a balanced model tier
• high-stakes tasks that justify expensive reasoning and stronger checks

This is not model worship. It is cost discipline.

The hidden cost is rarely the model line item

Teams fixate on tokens because tokens are visible. The real bill sits around the model: retries, context assembly, human correction, support escalation, and the work of proving the output is acceptable.

If a system saves one minute for a customer and creates two minutes of cleanup, it is destroying margin.

A finance-aware CTO should be able to answer these questions without hand-waving:

• what each class of request costs to serve
• where the rework happens
• what failure costs when the model is wrong
• which parts of the workflow justify premium inference

The real decision is not model choice, it is failure cost

“Best model” is usually the wrong conversation. The useful conversation is about failure cost.

A cheaper model that fails gracefully can beat a more expensive model that fails silently. A local fallback that keeps the system alive during a rate-limit event can matter more than a small quality lift in the happy path.

The CTO playbook is simple: optimize the whole system, not the benchmark screenshot.

Measure margin at the workflow level

The right unit of measure is the workflow, not the model call.

Ask:

• how much does this workflow cost end to end?
• how often does it need human repair?
• how long does it take to reach a trustworthy answer?
• what is the revenue or labor value of the result?

That is where the business truth lives. A model that looks slightly less accurate in isolation may create better margin if it is cheaper, faster, and easier to trust.

A practical threshold

If the system does not improve margin, then it needs to improve risk or speed. If it improves neither, it is a demo that escaped the lab.

AI work that survives budget review answers one of four questions:

• does it lower cost per task?
• does it reduce human labor?
• does it increase throughput?
• does it unlock new revenue with acceptable risk?

If not, the demo should stay in the demo lane.

Key Takeaways

• Route cheap work cheaply.
• Model cost is only part of the bill.
• Measure workflow margin, not call cost.
• If it does not improve margin, risk, or speed , it does not belong in production.

Most AI hiring plans are trying to fix an interface problem with resumes.

If product, platform, and applied AI are not built as one operating system, new headcount adds motion but not leverage. The constraint is usually not talent scarcity. It is system design.

Recruiting Alone Cannot Fix a Broken Stack

AI organizations often describe their issue as “we need stronger talent.” In many cases, they already have capable people. What they lack is a clear operating contract across teams.

The pattern is familiar:

• product optimizes for release velocity
• platform optimizes for reliability and control
• applied AI optimizes for model behavior and evaluation quality

Each goal is rational. The breakdown happens at the handoffs.

When interfaces are unclear, every launch becomes a negotiation. When interfaces are explicit, the same teams produce compounding output.

The Three-Layer Talent Stack

A healthy stack has three interlocking layers with distinct responsibilities:

1. Product — owns user outcomes and business success metrics.
2. Platform — owns safe defaults, deployment paths, and observability.
3. Applied AI — owns workflow behavior, retrieval/prompting/routing choices, and evaluation quality.

These are not departments in competition. They are system components with different jobs.

If product outruns platform, quality debt accumulates. If platform outruns product, infrastructure becomes generic overhead. If applied AI outruns both, you get technically impressive demos that never operationalize.

Where Organizations Usually Break

Most failures are boundary failures, not individual failures.

Common symptoms:

• no explicit owner for the model-to-product handoff
• platform operating as a ticket queue instead of an enablement layer
• applied AI measured by demo novelty instead of adoption in live workflows
• product committing features that infra cannot support safely

A concise diagnosis: org debt is usually interface debt with better branding.

Design the Stack Intentionally

The fix is not “more syncs.” The fix is explicit decision rights.

• product owns problem selection and business tradeoffs
• platform owns reliability guardrails and release safety
• applied AI owns workflow performance and evaluation integrity
• leadership owns escalation rules when tradeoffs conflict

Once this is explicit, hiring quality improves. You stop searching for mythical generalists and start hiring operators who can perform inside a coherent system.

What to Evaluate Before Adding Headcount

Before opening new roles, run this short check:

1. Are cross-team handoffs documented and current?
2. Does each layer have clear success metrics it actually controls?
3. Are escalation paths clear when speed, reliability, and quality disagree?
4. Are teams rewarded for system outcomes rather than local optimization?

If those answers are weak, fix interfaces first. New hires will scale the current operating model, good or bad.

Key Takeaways

• Strong AI organizations are designed as a system, not staffed as silos.
• Product, platform, and applied AI need explicit interfaces and decision rights.
• Boundary clarity is a bigger lever than raw headcount.
• Hiring works best after the operating contract is clear.

A lot of AI orgs look healthy in month three and brittle by year two. The model usually did not fail. The operating model did. Prototype energy is easy to create; durable coordination is not.

The question is not whether the team can ship something exciting. The question is whether the company can keep shipping after the novelty fades.

Why the prototype phase hides the real problem

In the early phase, AI teams often succeed because everyone is close to the work. Decisions are informal, context is shared, and the whole system fits in a few people’s heads. That stops scaling almost immediately.

As soon as the team grows, the same strengths turn into liabilities:

• knowledge becomes hidden
• approvals multiply
• handoffs slow down
• nobody owns the interface boundaries

What worked when the team was small no longer works when the company needs predictability.

The operating model should be explicit

A post-prototype AI org needs to define how work moves.

The model should answer:

• who owns the user problem?
• who owns the runtime?
• who owns the quality signal?
• who owns the risk boundary ?
• who can stop the release?

Without those answers, the team is improvising around gaps that will eventually become incidents or delays.

Handoffs are the hidden bottleneck

Most AI roadmaps do not fail because the team lacks ideas. They fail because each handoff adds ambiguity.

The problem shows up in predictable places:

• product asks for speed, platform asks for safety
• applied AI wants more freedom, compliance wants more proof
• leadership wants output, the system wants more control

That tension is normal. What is not normal is leaving it unresolved.

A good operating model turns tension into a documented interface, not a recurring crisis.

Scale requires less heroics, not more

The post-prototype org has to depend less on heroic behavior and more on repeatable behavior.

That usually means:

• clearer ownership
• smaller decision surfaces
• stronger eval gates
• visible rollback paths
• fewer ambiguous exceptions

This can feel slower at first, but it is the only way the org gets faster at scale.

A simple test

Ask whether the AI system can survive a senior person going on vacation for two weeks.

If the answer is “not really,” the organization is still running on hidden tribal knowledge.

If the answer is “yes, with documented ownership and a stable operating model,” the company is moving from prototype to production.

That is the real year-two test.

Key Takeaways

• Prototype energy does not scale on its own.
• The year-two problem is usually organizational, not model-related.
• Ownership, interfaces, and escalation paths matter more than the demo itself.
• A durable AI org is designed for scale before the prototype succeeds.

Slow decisions look like caution. In practice, they are hidden expense.

Decision latency belongs on the P&L. Every day a real decision sits unresolved, the business pays in delay, rework, and attention.

Why Decision Latency Matters

A team can look productive and still be dragging the business down if every meaningful decision takes too long.

Decision latency shows up as:

• stalled launches
• expired opportunities
• duplicated work
• growing frustration in the teams closest to the customer

When leaders do not measure this, they blame execution when the real problem is delay. The work may be moving. The organization is not.

What Decision Latency Looks Like in Practice

You can usually find it by asking a few questions:

• How long does a high-signal issue sit before someone decides?
• How many people need to weigh in before the first answer exists?
• How often do decisions get reopened because no one owned the original call?
• How much work is blocked waiting for alignment that never arrives?

Those are not soft questions. They are economic questions.

If a release, hiring decision , vendor decision , or architecture decision sits for weeks, the business is paying rent on uncertainty.

A useful line: ambiguous ownership is the most expensive architecture in your company.

Make It Visible

If you want leaders to care, make the metric visible.

Track:

• time from issue raised to decision made
• time from decision made to action taken
• number of escalations per decision class
• number of decisions reopened after approval

Once those numbers are in the open, patterns become hard to deny. You can see which teams move fast, which questions keep getting rerouted, and where the organization is burning time on decisions that should have been routine.

How to Reduce It

Decision latency drops when teams do four things well:

1. Define who owns each decision class .
2. Set decision boundaries before the crisis.
3. Reduce the number of people required for routine calls.
4. Make escalation fast when the decision is truly material.

This is not about making every decision unilateral. It is about making routine decisions quick and risky decisions explicit.

If the call is small, the system should move. If the call is material, the system should know exactly who has to weigh in.

Key Takeaways

• Decision latency is a real cost driver.
• Measure the time from issue to decision and from decision to action.
• Ownership clarity reduces hidden opex.
• The best organizations make routine decisions quickly and unusual decisions deliberately.

1. Workload Facts You Must Have

The vendor’s first move is to define your usage for you. Don’t let them.

• Total request volume per month, broken out by use case A single aggregate number is not enough. Know which workflows drive cost.
• Cost per task class (e.g., generation vs. classification vs. retrieval) If you cannot name your top three cost drivers, you cannot challenge the invoice.
• Latency p50/p95 by workflow, measured from your own instrumentation Vendor SLAs are measured at their edge, not yours.
• Percentage of spend attributable to this vendor vs. total AI budget Concentration creates leverage — for them. Know the number.
• Named owner of the vendor relationship on your side If no one owns it, no one negotiates it.

2. Architecture Leverage Check

Leverage is an architecture property. Answer these before you sit down.

• Is the vendor’s API called directly from product code, or through an abstraction layer ? Direct calls = switching costs measured in months. Abstraction = measured in days.
• How many distinct integration points does this vendor touch? Write the number. Fewer than five is manageable. More than ten is a dependency.
• What is the estimated engineering cost to swap this vendor? Get a real estimate, even a rough one. “Unknown” is not an answer.
• Do you have a secondary provider you have already integrated, even partially? Yes/No. If no, you have no credible threat.
• Does your data pipeline depend on vendor-specific formats or embeddings ? Format lock-in is often more expensive than API lock-in.

3. Evaluation Evidence

Vendors sell on benchmark claims. Counter with your data.

• Do you have evals that measure model performance on your actual workload ? Yes/No. If no, you are buying on their terms by default.
• Which models have you tested against your task suite in the last 90 days? List them. If the answer is only theirs, you have no comparison point.
• What is your acceptable quality threshold, defined numerically? “Good enough” is not a threshold. A number is.
• Have you run a cost-per-correct-output comparison across providers? Price per token is a distraction. Price per correct result is the metric.
• Who owns your eval framework and can demo it in the meeting if needed? Named person, not a team.

4. Exit Credibility

A vendor that believes you cannot leave does not negotiate. Make them uncertain.

• Do you have a documented migration plan, even a sketch? It does not need to be final. It needs to exist.
• What is your contractual notice period to exit? Know this before they remind you of it.
• Have you identified which vendor you would move to first if pricing increased 40%? Name them. Vague alternatives are not alternatives.
• Is there a sunset timeline for any features that are vendor-exclusive today? If yes, the vendor knows your dependency has an expiration date.
• Can your team absorb a two-week migration sprint without derailing the roadmap? Yes/No. Honest answer only.

If you cannot fill in the workload numbers, you are not done preparing — you are about to negotiate against someone who has already modeled your spend. If you have no eval data, you will accept their performance claims by default. If there is no exit plan, any number they name is essentially a take-it-or-leave-it offer. The meeting itself is the wrong place to discover these gaps. Thirty minutes with this worksheet before you walk in is worth more than any negotiation tactic once you are in the room.

AI roadmaps fail when ambition is treated as sequencing. Dependencies slip, rollback gets expensive, and the team discovers the missing work only after the launch date is already spoken for.

A survivable roadmap is not a prettier Gantt chart. It is a dependency-aware budget for uncertainty.

Roadmaps Fail at the Edges

The core mistake is treating the roadmap like a statement of intent instead of a statement of sequencing.

AI work fails at the edges:

• data access is slower than expected
• model behavior is less stable than expected
• review cycles take longer than expected
• vendor changes arrive earlier than expected

If your roadmap does not account for those edges, it is not a plan. It is a confidence exercise.

Most teams only find out those edges are missing after the launch date is already public.

The fix is to move the hidden work into the plan before the promise is made.

Budget the Dependency Chain

Every AI feature has a dependency chain:

• data availability
• context assembly
• model routing
• evaluation
• deployment
• fallback

If any one of those links is not ready, the feature will not survive real use.

If the chain is incomplete, the roadmap is lying by omission.

The most honest roadmap is the one that writes the chain down first. That slows the conversation, but it also keeps the team from selling a feature that depends on work nobody has budgeted.

Slower conversations are cheaper than broken launches.

Make Rollback a First-Class Requirement

Good roadmaps assume the first version will be wrong.

That means every AI initiative should answer four questions:

• How do we turn this off?
• How do we know it is hurting us?
• How fast can we revert?
• What manual path exists if the model degrades?

If those answers are fuzzy, the roadmap is overconfident.

If you cannot turn it off quickly, you have shipped a liability with a product label .

Roadmaps should not only describe the happy path. They should budget for the probability that the first version is wrong, the vendor changes terms, or the model regresses under load.

That is not pessimism. It is operational seriousness.

WIP Limits Matter More Than Hope

A roadmap that promises too many parallel AI experiments is usually a roadmap that does not respect WIP.

The more novel the work, the lower the WIP should be.

Concurrency feels productive until it multiplies rework.

Strong teams set rules like:

• no more than one high-risk AI launch per squad at a time
• no feature ships without evaluation coverage
• no vendor migration without a fallback path
• no roadmap item enters “done” until the operational notes exist

That may sound strict. It is. Novel work punishes loose concurrency.

What a Survivable Roadmap Looks Like

Survivable roadmaps are dependency-explicit, rollback-aware, and honest about capacity.

A roadmap is not a promise. It is a bet with visible failure modes.

If the failure modes are invisible, the roadmap is pretending.

You do not need a roadmap that impresses the room. You need one the organization can execute without pretending the hard parts are somebody else’s problem.

Key Takeaways

• AI roadmaps fail at dependency and rollback boundaries.
• Treat the roadmap as a budget for uncertainty, not a wish list.
• Limit WIP, make rollback explicit, and require evaluation coverage before launch.
• The best roadmap is the one the organization can survive.

AI does not change the core job of technical leadership. It changes the cost of being vague. In 2026, the best leaders still do the same three things: set direction, remove friction, and keep production systems measurable. The difference is that AI makes every weak assumption show up faster.

The real mandate is throughput. Not more noise. Not more experimentation theater . Throughput.

The Leadership Pivot: Focus on Throughput

Organizations do not pay technical leaders to keep up with model releases. They pay them to improve organizational throughput .

That means reducing cognitive overhead, tightening verification, and making deployment paths boring enough that teams can move without drama. If you cannot measure what an AI workflow produced, or what it cost to produce it, you do not have an operating system yet. You have a prototype with invoices.

The leadership question is simple: are we removing blockers faster than we are adding complexity?

Decision-Making in Practice

AI work gets messy when teams debate tools before they define the outcome.

Good leaders force the conversation back to first principles:

• What business metric should change if we ship this?
• What latency budget do we actually have?
• What happens when the model is wrong?

Those questions cut through a lot of noise. They keep the team from turning architecture meetings into opinion contests about vector databases , prompt styles, or the latest agent framework.

If the answer to any of those questions is fuzzy, the work is not ready for serious implementation.

Define “Good Enough” and Measure It

Reliability is not just accuracy. It is consistency, cost, and the ability to catch degradation before customers do .

Sometimes a smaller, cheaper model is the right answer. Sometimes the frontier model is worth the price. The point is not to be religious about either option. The point is to define the bar, test against it, and choose the system that meets it with the least operational pain.

Your job is not to build a perfect AI system. It is to build one where failure is bounded, expected, and visible.

The Cultural Shift

Technical leadership still has a change-management problem. Engineers will worry about ownership, safety, and the volatility of the ecosystem. Those concerns are real.

The right response is not debate for its own sake. It is instrumentation.

Stop arguing in design docs about whether a model will work. Build the telemetry that shows whether it works. Stop treating every new framework like a strategy reset. Run small, contained experiments that either produce evidence or die cheaply.

The strongest teams are not the ones that sprint toward the newest beta API. They are the ones that can absorb change without losing control.

Final Take

AI rewards leaders who are disciplined about outcomes and ruthless about verification. If the team can move quickly, measure clearly, and recover cleanly, AI becomes leverage. If not, it becomes another source of drag.

Six weeks later, the channel had five messages in it, four of them automated. The procurement leads were still pulling PDFs manually and copying terms into a shared spreadsheet. One support engineer, who had quietly championed the project from the beginning, had reverted to her old database query because “the tool doesn’t know about the amendments.” The tool was still running. Nobody had officially abandoned it. It had simply become invisible.

This pattern is not unusual. It is almost the default.

What Actually Failed

The postmortem conversation usually centers on the wrong things — model choice, interface design, rollout timing. Those are symptoms. The root causes are structural.

The contract tool was built around a narrow slice of the negotiation workflow: surfacing base terms. But procurement work is not base terms. It is base terms plus amendments plus prior history plus the relationship context the lead carries in her head. The tool knew one layer of a five-layer problem. It looked complete in a demo because demos are controlled. Real work is not controlled.

The output trust problem arrived fast. In week two, the tool surfaced an incorrect payment term — technically correct in the original contract, superseded by a signed amendment it had not been given access to. The lead caught it before it caused damage, but she stopped relying on it after that. One unexplained wrong answer is enough to demote a tool from co-worker to footnote. The team had not built evaluation into the system , so there was no way to know how often this happened, which made the uncertainty worse, not better.

Nobody owned adoption after the launch. The engineer who built it moved to a different priority. The VP who celebrated it never checked sustained usage . When procurement leads developed workarounds, there was no one watching the signal and no one with a mandate to respond. The tool drifted.

When It Works

A different team at a professional services firm built something structurally simpler: a tool that drafted the engagement summary section of a client report, pulling from structured notes the consultant had already entered into their project management system. Narrow scope. No novel context required. One predictable output format, reviewed every time before it went anywhere.

The tool stuck. Not because it was more technically impressive — it was considerably less so. It stuck because it removed a specific, recurring task that consultants genuinely disliked, it used context they were already maintaining anyway, and the output was always human-reviewed before it mattered. The failure mode was visible and safe. The value was obvious the first time you used it and every time after.

The team lead reviewed usage weekly for the first two months and made three small adjustments based on what she saw. That ownership — unglamorous, persistent, post-launch — is what made the difference.

The Structural Difference

Both companies built AI tools for internal workflows . One failed quietly, one became a habit. The gap was not the model. It was not the interface. It was whether the tool was designed around how work actually moves or around what would look good in a demo .

Tools that survive are ones that fit a narrow, complete slice of a workflow, produce output that is either verifiable or bounded enough to trust, require no context the user does not already have, and have someone whose job it is to watch whether people are actually using them.

That last part is the one most teams skip. Usage is not a launch outcome. It is an operating responsibility.

An AI-native company is not a company that uses AI. It is a company whose operating model — decisions, ownership, interfaces, capital, and failure boundaries — has been built so AI compounds inside it instead of evaporating around it.

The model will change. The system around it should not.

This is a manifesto. It is opinionated, deliberately. Twelve tenets, four movements, one test. Borrow what works. Argue with the rest.

Movement I — Strategy

1. The operating model is the strategy

The model is the most expensive dependency in your stack. It is not the brain. The brain is everything you build around it: context assembly, retrieval, validation, retries, telemetry, fallback, escalation.

Two companies buy the same frontier model on the same Tuesday. One ships in six weeks with a deterministic fallback, a typed validator, and an eval gate on every PR. The other ships in six months with a notebook of “good prompts” and a Slack channel for incidents. Same model. Different company.

If your AI plan begins with “which model should we buy,” you are solving the easiest problem in the room. The moat is everything around the model.

2. Capital allocation is the first product decision

Great AI teams do not start with a roadmap. They start with a kill list . Capital is finite. Attention is finite. Support burden is finite.

Three questions before any AI initiative gets funded:

1. Does this increase margin, reduce risk, or improve speed?
2. Can we measure that effect within one to three quarters?
3. Do we own the fallback if the model or vendor changes?

If the answer to all three is not yes, the default is no.

The most common pattern across Series B–D companies that quietly stalled in 2024–2025: somewhere between $1M and $3M of engineering and infra burned on internal copilots that never crossed adoption threshold, plus a duplicate prompt orchestration layer because two teams built one in parallel. Neither project had a measurable failure mode. Both had a sponsor.

A four-dimension scorecard makes the next budget meeting honest:

• Adoption — are real users using it in a real workflow?
• Reliability — does it fail in bounded, observable ways?
• Margin — does it reduce cost or improve unit economics?
• Speed — does it shorten a real business cycle time?

If you cannot defend it with numbers, the project is not innovative. It is unpriced.

3. Decision latency is a P&L variable

Slow decisions look like caution. In practice, they are hidden expense. Every day a real decision sits unresolved, the business pays in delay, rework, and attention.

Headcount is an input. Throughput is an outcome . Adding the tenth engineer to a system that takes nine days to approve a deploy adds nine more days of waiting, not 10% more output.

Track four numbers with the same seriousness as revenue:

• time from issue raised to decision made
• time from decision made to action taken
• escalations per decision class
• decisions reopened after approval

Ambiguous ownership is the most expensive architecture in your company.

Movement II — Architecture

4. Build firewalls, not masterpieces

A statistical engine cannot be expected to behave like deterministic infrastructure. If your architecture only works when the model is correct 100% of the time, it is not architecture. It is wishful thinking with a demo budget.

Three failure modes, three firewalls. They are not the same thing and they are not solved by the same code:

• Inbound sanitization. What data is permitted into the prompt context. PII strippers, schema enforcers, retrieved-document trust scoring. This is also where indirect prompt injection — instructions hidden in a vendor PDF, a customer message, or a tool output — gets caught before it reaches the model.
• Outbound validation. A typed schema checker stands between the model and the operational database. Malformed JSON, out-of-range values, and policy-violating outputs are rejected at the boundary, not absorbed by downstream services.
• Operational fallback. Circuit breakers for vendor outages and rate limits. If the model returns invalid output three times in a row, the system degrades to a deterministic path — not a stack trace in front of the user.

Each of these is a separate piece of code with a separate owner, a separate test surface, and a separate failure mode. A “kill switch” that catches all three is a slide, not a system.

You cannot prompt your way out of entropy. You have to architect your way out of it.

5. Evaluation is the spine

If you cannot define an eval suite before shipping a feature, you do not understand the system well enough to ship it.

A five-level maturity ladder :

1. Vibes-based. Someone eyeballs prompts before release.
2. Spreadsheet. Suite exists, runs occasionally, blocks nothing.
3. CI/CD-integrated. Evals run on every PR. A failed gate stays failed.
4. Continuous telemetry. Production samples scored asynchronously. Incidents become regression tests.
5. Governance as moat. Evaluation shapes architecture before code. Margin, latency, and sovereignty tradeoffs are quantified, not asserted.

Below Level 3 is not a production system. It is a demo with a pager.

Level 4 is where most organizations get stuck, and the reason is rarely effort. Judge models drift, ground truth ages, sampling bias creeps in, and your asynchronous scoring quietly stops tracking the failure mode you cared about. Mature teams hold a small, hand-labeled golden set as the anchor, treat the judge model as a versioned dependency, and re-calibrate when either changes.

Eval portability is a year-two survival trait. If your eval suite is hand-tuned to one model’s tokenizer and one vendor’s output quirks, you have not built an eval suite. You have built a benchmark for the model you are about to be unable to leave.

6. Agentic systems run on a reliability contract

Agents are not magical workers. They are autonomous systems with more ways to fail. The reliability discipline gets stricter, not looser.

Every production agent answers five questions in one meeting, without hand-waving:

• what is it allowed to do?
• what is it explicitly not allowed to do?
• what metrics prove it is healthy?
• what happens when the model degrades?
• who can stop it, and how fast?

But the five questions are a meeting checklist. The contract is a published artifact with SLOs, blast-radius caps in dollars or rows or API calls, rollback latency targets, and a named owner per failure mode. Blast radius is the real design variable: data scope, action scope, time scope, permission scope, fallback scope.

Kill switches are not weakness. They are governance that can move faster than the failure. A useful test of any AI control: could an engineer follow this rule at 2 a.m. without calling a committee?

A roadmap that ships an agent without answers to these questions is a roadmap that has shipped a liability with a product label. Every initiative names how it turns off, how it knows it is hurting, how fast it reverts, and what manual path exists when the model degrades.

Companion: Agent Reliability Contract template . Rollback document template .

Autonomy without a reliability contract is just an incident waiting for a timeline.

Movement III — Economics & Externals

7. Unit economics live at the workflow, not the model call

Teams fixate on tokens because tokens are visible. The real bill sits around the model: retries, context assembly, human correction, support escalation, and the work of proving the output is acceptable.

Route by value and by risk. Trivial work stays cheap and local. High-stakes work earns expensive inference and stronger checks. A finance-aware leader can answer, without hand-waving:

• what each class of request costs to serve, end to end
• where the rework happens
• what failure costs when the model is wrong
• which parts of the workflow justify premium inference

The cost question nobody owns until it explodes: when product ships a feature that 10x’s tokens, who pays? If the answer is “we’ll figure it out,” you have not designed an operating model. You have deferred a fight.

Compute placement is part of this calculation, not a separate one. For high-frequency agentic workloads, a chain of round-trips across regions and vendors compounds into real latency tax and real egress cost. Local-first, hardware-aware patterns earn their place where the workload mix justifies them — and create a worse outcome where it does not. Measure first, place compute second.

A cheaper model that fails gracefully beats an expensive model that fails silently.

8. Sovereignty is an architecture constraint

Privacy is not a feature you bolt on before an enterprise contract closes. It is the shape of the system.

A sovereign system controls the full lifecycle of every piece of data — where it lives, who can access it, how long it persists, and what happens when someone asks you to delete it. In practice, four concrete patterns:

• Customer-managed keys. BYOK or hold-your-own-key. If your cloud provider holds the only copy of the encryption key, “we cannot access your data” is a policy promise, not a verifiable claim.
• Regional routing with storage isolation. EU data does not leave EU infrastructure. The application layer handles the routing. The deployment pipeline ships multi-region.
• Scoped, short-lived access. No ambient credentials. Service-to-service tokens with explicit grants and automatic expiry.
• Immutable audit trails. Append-only, tamper-evident logging of every access, transformation, and movement.

“We use AWS” is not an answer to “where does my data live.” Sovereignty is about specificity.

The compounding bill arrives when you try to add this later. The discount arrives when you build it in early and close enterprise contracts without an architectural retrofit.

9. The threat model is the manifesto

An AI manifesto without a threat model is marketing copy. Four risks every operator names explicitly:

• Indirect prompt injection. Instructions hidden in retrieved documents, tool outputs, and user uploads — not just in the user’s direct prompt. Treat every retrieved string as potentially adversarial. Validate before it reaches the model. Strip before it reaches the agent.
• Silent quality drift. The model returns slightly worse reasoning. The tone shifts. The retrieval starts ignoring critical documents. There is no stack trace. Only asynchronous production scoring, anchored to a golden set, catches this before customers do.
• Vendor and model lock-in by accident. Fine-tunes, preference data calibrated to one model family, and prompts hand-tuned to a specific tokenizer compound. By year two, your “swappable” model is a six-month migration. Discipline preserves optionality: prompt abstraction, eval portability, vendor-neutral preference data, and a quarterly review of what would break if the vendor changed terms tomorrow.
• Agent blast radius creep. Permissions accumulate. The agent that summarizes documents quietly gains write access to your billing API because someone needed it once. Audit scope quarterly. Treat agent permissions like database credentials, not like configuration.

Threat modeling is not a one-time exercise. It is the bill of materials your system runs on.

Movement IV — People & Failure

10. Interfaces beat titles

Most AI hiring plans try to fix an interface problem with resumes. They rarely work.

A working leadership system is not a roster of senior titles. It is a decision map. Four owners with explicit decision rights and explicit escalation paths:

• Product — user outcomes, adoption, business tradeoffs.
• Platform — safe defaults, deployment paths, observability, paved roads.
• Applied AI — workflow behavior, routing, prompting, retrieval, evaluation quality.
• Governance — risk boundaries, sovereignty controls, escalation thresholds.

The titles can be anything. The interfaces cannot be ambiguous. If the answers depend on who is online that day, the system is not operational.

The same logic governs platform teams. A platform exists to make repeated decisions disappear into the default path — identity, routing, eval harnesses, logging, safe deployment, fallback behavior. The moment platform becomes a queue that has to bless every use case, the queue is the product and waiting is the cost. A platform should remove waiting, not become a waiting room.

Hiring works after the operating contract is clear, not before. New hires scale the current operating model, good or bad. Org debt is interface debt with better branding.

11. Anti-fragility requires portability discipline

Resilience is surviving the shock. Anti-fragility is using the shock to remove the next one.

Fragility hides in the org chart and in the stack. One engineer who knows the routing. One vendor whose terms changed last week. One fine-tune that took six months to train and would take six months to migrate. That is not an organization or a system. That is a single point of failure wearing a department badge or a model card.

Four design choices build strength:

• Modular ownership. No critical function depends on one person’s memory. Deputies are named.
• Resettable interfaces. A model, vendor, or workflow can be swapped without a rewrite. This is not free. It requires prompt abstraction, eval portability, vendor-neutral preference data, and a regular drill where the team actually proves a swap is possible.
• Fast learning loops. Every failure produces a tighter eval, a better fallback, or a clearer operating boundary.
• Cross-training on the boring parts. Alerts, evals, fallback logic, access boundaries. The unglamorous work is what keeps the organization elastic.

A short anti-fragility check:

• Can you swap a model without rewriting the product?
• Can you lose a key engineer without losing the system?
• Can you absorb a vendor price increase without panic?
• Can you turn a production incident into an improved control?

If any answer is no, the organization is more brittle than it thinks. The most expensive lie an AI organization tells itself is that the model is swappable when nobody has tried.

12. The year-two test

A lot of AI organizations look healthy in month three and brittle by year two. The model did not fail. The operating model did. Prototype energy is easy to create. Durable coordination is not.

The single question that separates the two:

Can the AI system survive a senior person going on vacation for two weeks?

If the answer is “not really,” the organization is still running on hidden tribal knowledge.

If the answer is “yes, with documented ownership, a published reliability contract, an eval suite that blocks releases, and a fallback path the on-call engineer can execute at 2 a.m.,” the company is moving from prototype to production.

That is the only year-two test that matters. Everything else in this manifesto is in service of passing it.

What this manifesto is not

It is not a prediction about which model wins. It is not a framework for replacing engineers with agents. It is not a defense of any vendor, any cloud, or any stack.

It is a statement about how serious companies organize for AI when the easy money, the demo budgets, and the hype cycles are done — and only the operating model is left to do the work.

The model will change.

The system around it should not.

Law Zava writes about the operating model behind serious AI execution. Companion artifacts: Agent Reliability Contract template · Rollback document template · Eval Suite starter kit . The canonical reading path is at /blog .

Most AI dashboards count motion, not progress. They record pilots, prompts, and meetings, then call that momentum. If the scorecard cannot show adoption, reliability, margin, or cycle-time improvement, it is a prop. A board should be able to read it and know whether the business is better off.

The Theater Problem

AI reporting drifts toward vanity metrics because vanity metrics are easy to collect and hard to argue with.

The usual suspects:

• number of pilots launched
• number of prompts written
• number of models tested
• number of meetings held
• number of slides in the board update

None of those is useless on its own. The problem is that none of them answers the only question that matters: what improved because we shipped this?

A Better Executive Scorecard

A serious AI scorecard should be small enough to remember and strong enough to force a decision.

Start with four dimensions:

1. Adoption — are real users using it in a real workflow?
2. Reliability — does it fail in bounded, observable ways?
3. Margin — does it reduce cost or improve unit economics?
4. Speed — does it shorten a real business cycle time?

If a project does not move at least one of those numbers, it is not strategic. It is a lab exercise with a budget.

The point is not to build a perfect dashboard. The point is to make it impossible to hide weak outcomes behind busy activity.

What to Report Weekly

A weekly AI review should be short, blunt, and decision-oriented.

Report:

• what shipped
• what users actually did with it
• what broke
• what it cost
• what decision changed because of the data

That last bullet matters. Progress reporting without decisions is performance art.

A team can launch five experiments in a week and still have no strategy. Strategy shows up when the evidence sharpens the next choice.

Keep the Dashboard Honest

There are two reliable ways AI dashboards lie.

First, they drift toward lagging metrics only. By the time the board sees the number, the product problem is already old.

Second, they reward volume instead of signal. A busy roadmap can still be a weak roadmap.

Keep the dashboard honest by requiring every metric on the top page to map to one of three board outcomes :

• margin expansion
• risk compression
• execution-speed advantage

If a metric does not help the board understand at least one of those outcomes, it belongs lower in the stack or not at all.

A line worth keeping: if the scorecard cannot survive finance review, it is not strategy.

Key Takeaways

• Measure adoption, reliability, margin, and speed.
• Weekly reviews should force decisions, not decorate slides.
• Tie every visible metric to margin, risk, or execution speed.
• If the dashboard cannot survive finance review, move it off the first page.

In 2026, build vs. buy is not a taste question. It is an operational cost question. Are you prepared to own the telemetry, the fallback paths, and the failure modes that come with the stack? Buying gives you speed and leaves the analytics with someone else. Building gives you control and hands you the overhead.

The Myth of the Headline Price

Most teams compare API pricing to GPU rental and stop there. That is the wrong first-order model.

Token price is the easiest number to quote and the least useful number to trust. The real bill shows up in the work around the model:

• Telemetry & Evals: If you self-host, you must build the pipeline that captures, scores, and reviews output. Vendor APIs may bundle some of this, but then they also own the metadata.
• Graceful Degradation: When the provider throttles you at peak, do you have local fallback? Hybrid systems buy resilience, but they also add systems-engineering work.
• Data Sovereignty : Sometimes the reason to build is simple: the data cannot legally leave your VPC. Once that is true, the token price stops mattering.

When to Buy (The Commodity Highway)

Buy when the AI capability is a feature, not the product.

If you are building an internal documentation chatbot, a support-ticket summarizer, or a semantic search overlay, buy the API. Do not spend engineering throughput standing up vLLM instances and chasing KV-cache optimizations for a problem that is not your moat.

The catch is lock-in at the integration layer. If your code imports vendor-specific classes directly, you will feel the squeeze when prices change or a model line is deprecated. Keep the provider behind an internal interface .

When to Build (The Crucible of Control)

Build when AI sits inside unit economics or inside a hard trust boundary.

You must build if:

1. Your margins depend on it. Billions of tokens a day can make the API tax the difference between a healthy product and a broken one.
2. You operate under zero-trust or residency constraints. In healthcare, finance, or defense, the data cannot touch a multi-tenant cloud edge.
3. You need hardware-level optimization. Sub-150ms tail latency usually means quantization, attention fusion, and serious control over the runtime.

That is the part teams underestimate. You are no longer building a prompt pipeline. You are operating a distributed, heavily constrained state machine. That takes engineers who understand memory bandwidth, not just prompting.

The Hybrid Default

The mature pattern in 2026 is a barbell.

Buy frontier models for complex reasoning, planning, and high-context zero-shot tasks. Build or host quantized, heavily tuned 8B models for the large volume of routing, formatting, and classification work that sits underneath the product.

The CTO’s job is not to choose a camp. It is to make the handoff between buy and build a config change, not a rewrite.

Most AI strategy decks are full of nouns and short on numbers. That is usually the tell. If a project cannot move margin, reduce risk, or shorten the path to an outcome, it is not strategy. It is activity with a steering committee.

Why Three Numbers Are Enough

Leaders overcomplicate AI strategy because they do not want to choose.

But every AI decision eventually lands in one of three buckets:

• Margin — does it improve unit economics?
• Risk — does it make the system safer or more controllable?
• Speed — does it shorten the path from decision to outcome?

That is the executive frame. Everything else supports it.

If a project cannot clearly improve at least one of those numbers, it does not belong near the top of the roadmap.

The Trap of Novelty Metrics

AI teams love the wrong metrics because the wrong metrics are easy to count.

Number of models tested. Number of pilots launched. Number of prompts written. Number of demos shown. Number of meetings held.

Those numbers can tell you whether work is happening. They do not tell you whether the company is getting more profitable, less exposed, or faster to act.

Build a Scorecard Around Outcomes

A serious AI scorecard is short.

1. Did margin improve?
2. Did risk go down?
3. Did cycle time shorten?

Everything else is instrumentation that helps answer those questions.

That does not mean you ignore adoption, reliability, or cost. It means you use them as inputs to the three executive numbers, not as substitutes for them.

The strongest boards and founders do not need twenty metrics. They need a few numbers that are hard to fake.

Make the Three Numbers Operational

The framework only works if the numbers are real.

For each AI initiative, define:

• the baseline
• the target
• the measurement cadence
• the owner
• the rollback path if the numbers move the wrong way

That keeps the conversation concrete and makes the project accountable.

A line worth keeping: if a strategy cannot change one of the three numbers, it is probably theater.

Key Takeaways

• Margin, risk, and speed are enough to evaluate AI strategy.
• Stop reporting novelty metrics as if they were outcomes.
• Give every project a baseline, target, owner, cadence, and rollback path.
• If the work does not change the numbers, the work is not strategic.

Great AI teams do not start with a roadmap. They start with a kill list. If a project cannot defend margin, risk, or speed, it does not deserve the next budget cycle. Capital is finite. Attention is finite. Support burden is finite.

The real mistake most companies make is treating AI spend as a separate class of spend. It is not. It competes with product work, platform work, hiring, and operational debt. If you cannot explain why an AI initiative deserves scarce capital, you are not allocating capital. You are subsidizing hope.

Capital Allocation Is the First Product Decision

Capital allocation is not a finance problem that happens to engineering. It is a technical leadership problem with finance consequences.

Every AI project consumes three things:

• engineering time
• infrastructure budget
• organizational attention

If the project does not improve one of three board-level outcomes — margin expansion, risk compression, or execution speed — it is likely a vanity project wearing a product costume.

That does not mean the project has to be immediately profitable. It does mean you should be able to state what gets better if the project works and what gets worse if it does not.

What Should Die First

The easiest place to make mistakes is the demo room. The second easiest is the budget meeting.

Stop funding these first:

1. Thin demos that do not survive workflow reality. If the user needs three manual edits after every response, you have built a presentation layer, not a product.

2. Duplicate platform work. If two teams are building separate prompt orchestration, evaluation, or routing layers, one of them should stop. Duplication feels like speed until the maintenance bill lands.

3. Ambiguous experiments with no owner. “We should explore AI” is not a strategy. It is a permission slip for drift.

4. Projects with no measurable failure mode. If nobody can say what counts as bad output, bad latency, bad cost, or bad adoption, the project cannot be managed.

There is a simple reason these projects linger: they are emotionally easy to defend. Nobody wants to kill a project that sounds innovative. But if you cannot defend it with numbers, the project is not innovative. It is unpriced.

The Kill-List Rubric

A good kill list is not a spreadsheet of personal dislikes. It is a decision system.

Before funding a new AI initiative, ask three questions:

• Does this increase margin, reduce risk, or improve speed?
• Can we measure that effect within one quarter?
• Do we own the fallback if the model or vendor changes?

If the answer to all three is not yes, the default should be no.

This is where a lot of teams get sentimental. They continue funding because the project has a sponsor, or because it already consumed sunk cost, or because it looks good in a board deck. Those are weak reasons to keep a system alive.

Strong reasons to keep funding an AI initiative usually look like this:

• it replaces high-volume manual work
• it improves decision quality in a regulated workflow
• it reduces customer wait time
• it protects a revenue stream that depends on fast, accurate responses

Notice that none of those reasons mention hype.

What to Keep Funding Instead

The highest-return AI investments are boring in the best way.

Fund the parts that make the system measurable and durable:

• retrieval and context quality
• evaluation harnesses
• fallback logic
• routing by task class
• observability around bad outputs and retries
• workflow-specific data collection

The point is not to chase the smartest model. The point is to build a system that can absorb model churn without forcing a rewrite every six months.

A useful line to keep in mind: if a system cannot be measured under load, it is still a pilot. Pilots are fine. Pilots just should not keep consuming production budget forever.

The Hard Part Is Saying No

The best operators are not famous for being aggressive spenders. They are famous for being disciplined about what they do not fund.

That discipline becomes a reputation asset. The founder who sees you delete a weak AI project starts trusting your judgment. The board member who sees you cut duplicate work starts trusting your signal. The engineering team that sees you protect their time starts trusting your priorities.

Capital allocation is how you tell the truth about what matters. If a project cannot defend margin, risk, or speed, it should not survive by momentum alone. Fund the systems that make AI measurable, recoverable, and cheap to operate. Cut the rest.

In 2026, a CTO’s AI strategy is not a model shortlist. It is an operating model for data, latency, evaluation, and failure. The model will change. The system around it should not.

If your AI plan still starts with “which model should we buy,” you are solving the easiest problem in the room. The moat is the pipeline that feeds context, the eval loop that catches regressions, and the fallback path that keeps the product standing when the model misses.

The Strategy Is the Infrastructure

The single biggest mistake engineering organizations make is treating the model as the brain. It is not. It is the most expensive dependency in the stack.

The brain is everything you build around it: context assembly, retrieval, validation, retries, telemetry, and rollback.

A CTO must focus ruthlessly on three pillars:

1. The Context Pipeline

The model is only as intelligent as the context you feed it. If Postgres, Cassandra, or Scylla takes five seconds to assemble structured context, encode it, and hand it to the orchestrator, your feature is already late before inference begins.

Strategy means architecting data replication, embedding generation, and caching so the latency budget stays intact for the inference layer. If your data infrastructure is not close to real time, your AI will not be either.

2. The Evaluation Framework

You cannot scale what you cannot measure. If your organization is still eyeballing model outputs before deployment, you are running a pilot, not a production system.

Leadership means demanding continuous evaluation. Every PR that touches an orchestration layer must be blocked by a CI pipeline that runs 500 deterministic evals against the new reasoning flow. Building that telemetry is the AI strategy.

3. Graceful Degradation and Fallbacks

LLMs fail. APIs throttle. Endpoints rotate. If a model hallucinates malformed JSON and your core application crashes, that is not an AI failure; that is an architectural failure.

A mature strategy wraps every AI interaction in circuit breakers. If the model fails three times, what is the deterministic fallback? If the cloud provider rate-limits you, where is the local, quantized 8B-parameter fallback model running in your own cluster?

Stop Chasing the Frontier

The frontier-model conversation is a distraction. Unless you are OpenAI or Anthropic, you do not win by having the smartest model. You win by having the tightest feedback loop, the cleanest data access, and the lowest cost per transaction.

A strong CTO designs for swapability : a single configuration commit, zero downtime, and telemetry that proves the new model performs 4% better on the exact workload that matters.

That is the strategy. Everything else is theater.

Headcount is an input. Throughput is an outcome. The best engineering organizations have stopped asking “how many engineers do we need?” and started asking “what’s blocking the engineers we have?” Teams that optimize for decision speed, defect containment, and execution clarity outperform teams twice their size. Hiring more people into a broken system just makes the system break faster.

The Metric Everyone Tracks and Nobody Questions

Every quarterly planning cycle, the same conversation happens. The roadmap is too ambitious for the team. The proposed solution is more headcount. The exec team approves some fraction of the ask. Six months later, the team is bigger but the roadmap is still slipping.

This pattern persists because headcount is easy to measure and feels actionable. You can put a number on a slide. You can point to it in a board meeting and say “we’re investing in engineering.”

But headcount measures capacity the way adding lanes measures highway throughput. It works up to a point, then coordination overhead offsets the capacity gain. The tenth engineer doesn’t add 10% more output. They add 10% more communication paths, 10% more code review load, and another person who needs context on every architectural decision.

The organizations getting this right have shifted to outcome metrics. Not “how many people do we have” but “how fast do decisions move from identification to resolution.” Not “how many PRs did we merge” but “what’s our change failure rate and how quickly do we recover.”

Staff Growth Versus Constraint Removal

Adding staff is an additive intervention. It puts more resources into the system. Constraint removal is a multiplicative intervention. It makes every existing resource more effective.

Consider a team of eight engineers where the average PR sits in review for 18 hours. Hiring two more engineers does nothing to fix the review bottleneck. It makes it worse because there are now more PRs competing for the same review bandwidth. But changing the review process, setting a 4-hour SLA, pairing reviewers with authors, and shrinking PR scope, can cut that 18 hours to 4 without adding a single person.

The same principle applies at every level. Slow deploys, unclear ownership, meetings that could be async documents, long approval chains. Each costs every engineer on the team hours per week. Multiply by team size and the waste is staggering.

If 20 engineers each lose 5 hours per week to process friction, that’s 100 engineer-hours, equivalent to 2.5 full-time engineers doing nothing but waiting. Removing the friction is cheaper than hiring, faster to implement, and doesn’t increase coordination costs.

AI tooling has made this dynamic sharper. A well-structured team with good tooling and clear ownership regularly outships teams twice its size. But a poorly structured team with AI tooling just generates more half-finished work faster. AI amplifies the system it operates in, good or bad.

The Operating System of a High-Throughput Team

High-throughput teams share three operational patterns that have nothing to do with individual talent.

Clear intent over detailed instructions. When an engineer picks up a task, they should know the outcome that matters, not the exact steps to get there. “Reduce P95 latency on the search endpoint below 200ms” is clear intent. “Refactor the search query builder to use connection pooling” is a solution masquerading as a task. The first lets the engineer use judgment. The second removes it.

Teams that operate on intent move faster because decisions happen at the point of most information, the engineer doing the work, rather than being routed through a manager who has less context. This requires trust, and trust requires that the intent is genuinely clear and that the engineer has the authority to make reasonable tradeoffs.

Delegated authority with explicit boundaries. Every recurring decision type should have a documented owner and a decision boundary. “The on-call engineer can roll back any deploy without approval” is a delegation. “Database schema changes require review from the data team” is a boundary. When these are written down and understood, decisions happen in minutes instead of hours.

The failure mode is implicit authority. Nobody knows who can make the call, so everyone escalates. The escalation chain adds latency to every decision. In a team of 15, this can mean that a simple operational decision takes a day instead of an hour because it bounces between three people who each assume someone else owns it.

Async-first communication . Synchronous communication, meetings, Slack pings expecting immediate response, tap-on-the-shoulder interruptions, is the most expensive coordination mechanism. It requires everyone to be available simultaneously and context-switch away from focused work.

Async-first doesn’t mean no meetings. It means meetings are for decisions that genuinely require real-time discussion. Everything else is a written document, a recorded decision in a ticket, or a code review comment.

A Weekly Operating Cadence

Decision tempo separates high-throughput teams from slow ones. A lightweight weekly cadence keeps the system self-correcting without drowning in noise.

Weekly: review leading metrics. Cycle time from commit to production, change failure rate, time to recover from incidents, review queue depth, and decision latency on open questions. Don’t track vanity metrics like lines of code or number of PRs.

Biweekly: connect signals to causes. Is cycle time creeping up? Is one team’s change failure rate spiking? Are the same types of decisions getting stuck repeatedly? The goal is systemic diagnosis, not individual blame.

Biweekly: pick one constraint to remove. “This sprint, we’re going to cut our deploy time from 45 minutes to under 10” is a decision. “We’re going to improve developer experience” is not. One thing, not five.

Continuous: execute, measure, repeat. Act on the decision, measure the result, and feed it back into the next weekly review. If cutting deploy time didn’t improve cycle time, the constraint was elsewhere. Move to the next one.

Incentives That Reward Impact Over Activity

Most engineering organizations accidentally incentivize busyness. The engineer who closes the most tickets gets praised. The team that ships the most features gets the biggest headcount allocation. The manager who runs the most meetings looks the most engaged.

Throughput-oriented incentives look different.

Reward engineers who eliminate recurring work, not just complete it. The engineer who automates away a manual process that costs the team 10 hours per week has created more value than the engineer who ships a new feature used by 50 people.

Reward teams that improve their own throughput metrics, not just output volume. A team that cuts its change failure rate from 15% to 3% has freed up enormous capacity that was previously spent on rollbacks, hotfixes, and incident response. That’s worth more than two new features.

Reward leaders who make themselves less necessary. The manager whose team operates smoothly when they’re on vacation has built a better system than the manager who’s cc’d on every decision.

A 12-Week Operating Reset

For teams experiencing delivery drag, a structured reset works better than a reorg.

Weeks 1-3: Measure. Instrument cycle time, change failure rate, review latency, and decision latency. Don’t change anything yet. Establish a baseline that everyone agrees on.

Weeks 4-6: Remove one constraint. Pick the biggest bottleneck revealed by the data. If review latency is the worst, fix the review process. If deploy time is the worst, fix the pipeline. One constraint at a time.

Weeks 7-9: Delegate and document. Write down the top 10 recurring decision types and who owns each one. Set decision boundaries. Remove one layer of approval from the most common workflow.

Weeks 10-12: Sustain. Establish the weekly review cadence. Compare throughput metrics to the week-1 baseline. Identify the next constraint. Make the cycle self-reinforcing.

Teams that complete this reset typically see 30-50% improvement in cycle time without adding staff. The improvement comes from removing friction that was invisible because everyone had adapted to it.

Board-Facing Metrics That Map Engineering to Business Risk

Boards understand risk and return. Translate engineering throughput into those terms.

Cycle time maps to market responsiveness. “We can respond to a competitor move in days, not months” is a strategic capability that boards care about.

Change failure rate maps to operational risk. “5% of our changes cause incidents” is a risk number a board can evaluate, especially when paired with the cost of those incidents.

Recovery time maps to resilience. “When something breaks, we fix it in under an hour” is a durability statement that affects customer trust and revenue protection.

Decision latency maps to organizational agility. “Strategic decisions take 2 days to reach execution, not 2 weeks” tells the board that the organization can adapt.

None of these metrics mention headcount. That’s the point. Headcount funds capacity. These metrics measure whether that capacity produces results.

Key Takeaways

Headcount tells you what you’re spending. Throughput metrics, cycle time, change failure rate, recovery time, decision latency, tell you what you’re getting.

The highest-leverage engineering work is constraint removal, not feature addition. Every hour of friction you eliminate pays dividends across every engineer on the team.

Stop asking “how many engineers do we need?” Start asking “what’s preventing the engineers we have from shipping?”

Then I ask: “How many of those interactions ended with the user getting the right answer?” Silence.

This is the metrics gap that keeps burning teams. Usage tells you people showed up. It tells you nothing about whether they left with what they needed. An AI feature can be heavily used and actively harmful at the same time. Users try it, get a wrong answer, correct it manually, and keep coming back because they’re optimistic. Your dashboard shows engagement. Your product is eroding trust.

What to Actually Measure

Three things. That’s it.

Did the output help? Not “was it generated.” Did it contribute to the user completing their task? Define what successful completion looks like for your specific workflow, then measure whether AI-assisted completions happen more often, faster, or with fewer errors than the baseline. If you can’t tie AI output to a task outcome, you’re measuring wind.

Was it correct? Combine automated checks with periodic human review. Automated checks catch format violations, hallucinated entities, and safety issues . Human review catches the subtle stuff: answers that are technically correct but misleading, or correct for the wrong version. Sample 5% of outputs weekly. That’s enough to spot trends before they become incidents.

Do users trust it? Trust is the leading indicator everyone ignores. Track it through implicit signals: how often users edit AI output before accepting it, how often they abandon a flow after seeing the AI response, and how often they re-prompt with the same question phrased differently. Rising edit rates or re-prompt rates mean trust is declining. By the time CSAT surveys catch this, you’ve already lost months.

The Dashboard That Fits on One Screen

Your AI scorecard should answer four questions at a glance:

1. Are people using it? (adoption, retention – the basics)
2. Is the output good? (correctness rate, safety rate from automated + human review)
3. Is it helping? (task completion rate, time to completion vs. baseline)
4. Do they trust it? (edit rate, re-prompt rate, abandonment rate)

Review weekly. Tie every metric to a decision. If a number moves and nobody changes anything, delete the number. Dashboards without decisions are theater.

When a metric dips, you should be able to trace it back to a model update, a retrieval change, or a product shift within the same week. If you can’t, your instrumentation is too coarse.

The Uncomfortable Truth

Most teams avoid quality metrics because they’re harder to collect and the numbers are less flattering than engagement counts. That’s exactly why they matter. The teams that measure task success and trust alongside usage are the ones whose AI features survive past the demo phase.

Measure what the user felt. Everything else is vanity.

Stop chasing model announcements. The teams that win in 2025 are the ones building evals, monitoring quality, and treating AI like infrastructure instead of magic. Discipline over heroics.

Every January, someone publishes a breathless AI predictions post. “This will be the year of AGI.” “Agents will replace developers.” “Multimodal everything.”

I’m not going to do that.

What I can tell you is what I see working with teams that are actually shipping AI to production. The pattern is clear: 2024 was the year everyone built demos. 2025 is the year those demos have to work.

The demo hangover

Here’s what happened to most AI projects last year. Someone built a prototype in a weekend. It was impressive. Leadership got excited. Budget appeared. Then the prototype hit real users, real data, and real edge cases, and everything got complicated.

I watched this play out at three different companies. Same story every time. The model was fine. The engineering around the model wasn’t.

Missing evaluation suites. No fallback paths. Prompts that drifted every time someone tweaked them. Cost tracking that amounted to “we’ll figure it out later.” The model was the easy part. Operating discipline was the hard part.

That’s the real trend for 2025. Not a new model. A new level of engineering rigor around models.

Reasoning gets interesting

Models that think before they answer are genuinely useful for a specific class of problems. Multi-step analysis. Code review. Debugging. Anything where you would rather wait 30 seconds for a correct answer than get a fast wrong one.

The trap is treating reasoning models as the default. They’re slower, more expensive, and overkill for 80% of requests. The smart move is routing: fast model for simple tasks, reasoning model for complex ones. I’ll write more about this in a couple of weeks.

Multimodal is real but boring

Image, audio, and text working together is no longer a research demo. It’s a feature. Internal tools are the clearest win – think document-processing pipelines that can read scanned forms, or support systems that understand screenshots.

The value isn’t in any single modality being amazing. It’s in combining them so the system has richer context. Boring. Useful. Exactly the kind of thing that makes money.

Evaluation-first development

The single biggest shift I keep pushing is simple: define success before you write the first prompt .

This sounds obvious. Almost nobody does it. Teams will spend weeks tuning prompts and then measure success by vibes. “It feels better.” “The CEO liked the demo.” That isn’t engineering. That’s hope.

What works: a fixed eval set, tested on every change, with clear pass/fail criteria. Treat prompts like code. Version them. Review them. Test them. I won’t ship a prompt change without running it against the eval suite. Period.

Governance stops being optional

Regulation is firming up. The EU AI Act is real. Enterprise clients are asking for audit trails, documentation, and risk tiers before they’ll sign contracts. If your AI system can’t explain what it does, what data it touches, and who’s responsible when it goes wrong, you’re in for a bad year.

This isn’t bureaucracy for its own sake. Good governance actually accelerates adoption because it turns “can we use AI for this?” from a six-week debate into a checklist. Risk tier low? Ship it. Risk tier high? Here’s exactly what you need before you ship.

Governance that blocks delivery is broken governance. Governance that makes yes safe and fast is a competitive advantage.

Agents: promising, overhyped

Agents that can execute multi-step tasks are improving fast. They’re also still brittle. Context changes break them. Domain boundaries confuse them. The failure modes are subtle and hard to detect.

The near-term play is constrained agents with explicit checkpoints. Not open-ended autonomy. Not “let the agent figure it out.” Clear scope, clear permissions, clear rollback. We learned this lesson with microservices a decade ago: autonomy without contracts is chaos.

What I’m ignoring

• Any roadmap built on vendor keynote slides instead of product outcomes.
• Prompt engineering tricks that can’t be tested, versioned, or reproduced.
• “Autonomous” systems with no permission model, no audit trail, and no kill switch.
• Anyone who says “just add AI” without specifying what success looks like.

What matters

The capabilities are real. The models will keep getting better. But the gap between “this works in a demo” and “this works in production at 3am on a Saturday” is where careers and companies are made.

Ruthless focus on the boring stuff. Evals. Monitoring. Cost tracking. Fallback paths. Governance. That’s the 2025 playbook.

The teams that treat AI like infrastructure – with the same rigor they bring to databases and deployment pipelines – will win. Everyone else will keep rebuilding demos.

The harder question: what should you actually do differently?

Here’s my short answer, based on a year of working on AI across multiple organizations and watching the gap between teams that shipped and teams that stalled.

Stop Experimenting. Start Measuring.

If you’ve been running AI “experiments” for more than a quarter without a clear evaluation framework, you aren’t experimenting. You’re procrastinating. Experiments have hypotheses, metrics, and endpoints. Pilots have owners, success criteria, and deadlines.

Pick two or three use cases closest to production. Define success in numbers, not narratives. Build an evaluation set. Ship to real users with monitoring. Learn from data, not opinions.

This isn’t glamorous. It’s effective.

Build the Operational Foundation

The teams that will move fastest in 2025 are the ones building the plumbing now. Not new models. Not new frameworks. Plumbing.

• An evaluation loop that runs regularly, not when someone remembers
• Cost tracking with per-feature attribution so you know where money goes
• Security controls for model access and data handling that satisfy your legal team
• Model-agnostic interfaces so you can swap providers without rewriting your stack

Every one of these is boring. Every one of these is a prerequisite for scaling anything in 2025. Through Q4, I’ve been helping teams set up exactly this kind of infrastructure, and the teams that have it in place are already iterating faster than teams that built flashy demos without it.

Governance Isn’t the Enemy

AI governance has a reputation problem. Engineers hear “governance” and think “bureaucracy that slows us down.” That framing is wrong.

Lightweight governance – clear ownership for use case intake, a simple review path for legal and security risks, a cadence for measuring value and retiring weak experiments – actually accelerates shipping. It removes the ambiguity that causes teams to stall waiting for implicit approval.

The companies that move fastest all have some version of this. Not a committee. Not a 50-page policy document. A clear owner, a simple process, and a regular review. That’s it.

What I’m Betting On

Personally, I’m betting that 2025 is the year AI stops being a separate initiative and becomes part of how software gets built. Not a team. Not a project. A capability that lives inside existing workflows, owned by existing teams, measured by existing standards.

The companies that treat AI as special will keep producing expensive demos. The companies that treat it as normal – same code review, same evaluation, same cost accountability, same ownership – will ship things that last.

Discipline over heroics. Same as always.

I see this pattern across telecom and fintech organizations. The demo-to-production gap isn’t a technology problem. It’s an organizational one.

The demo was the easy part

A POC can skip everything that makes enterprise software hard. It runs on a developer’s laptop with test data. It doesn’t need to handle real user volumes. During a demo, nobody asks about audit trails or data retention policies.

Then the project moves toward production and reality hits. Security wants a threat model. Legal wants to know where the data goes. The platform team wants to know who pays for compute. The data science team discovers the training data is messier than expected. None of this is surprising. These are the same problems every enterprise system faces, plus a few new AI-specific ones: model drift, prompt management, and probabilistic outputs.

The teams that get stuck are the ones that treated the POC as the starting line instead of a feasibility check.

Start boring, stay boring

The single best predictor of success I’ve seen is picking a first use case that’s low-risk and internal. Something where a human reviews the output before anything happens. Document summarization for internal teams. Draft generation for support responses that get edited before sending. Classification of inbound requests to route them to the right queue.

These aren’t exciting. That’s the point. You want a use case where a bad output is an inconvenience, not a liability. One where you can iterate on prompts and evaluate quality without a customer ever seeing an unpolished result.

I keep telling teams the same thing: your first AI feature should be invisible to customers. Ship it internally, prove it works, build the muscle memory for operating AI in production, then expand.

Build the platform before the pilots multiply

Here’s what happens when you don’t have a shared platform: every team builds its own integration. They pick different models, prompt patterns, and logging approaches. Six months later, you have eight AI features and no way to compare quality, manage costs, or enforce policies across them.

The fix is unglamorous. Build a thin shared layer early. It needs three things:

1. Centralized model access with authentication, rate limiting, and cost tracking.
2. A prompt registry so prompts are versioned, reviewable, and not buried in application code.
3. Evaluation tooling that every team can use to measure output quality against a golden set.

This doesn’t need to be perfect or fully featured. It needs to exist before the third team starts building their own AI integration. I’ve watched organizations try to consolidate after the fact. It’s painful and expensive.

Governance that enables instead of blocks

The worst governance models I see are designed by committee without input from the engineering teams that have to live with them. They produce a 40-page policy document, a six-week review cycle, and a strong incentive for teams to quietly build things without telling anyone.

Good governance is lightweight and fast. A one-page use case template. A clear risk-tier system: low risk gets self-service approval, high risk gets review. A standing meeting where legal, security, and engineering are in the same room instead of a months-long email chain.

One organization I worked with reduced its AI approval cycle from eight weeks to five days by switching from a document-based review to a 30-minute live walkthrough with all stakeholders. Same rigor. Fraction of the time.

The uncomfortable truth

Most enterprise AI projects don’t fail because the technology isn’t ready. They fail because the organization isn’t ready. The AI works fine in the demo. The procurement process takes four months. The data team can’t provide clean training data. The legal review has no precedent to follow, so it defaults to “no” until someone escalates.

If you want to ship AI in an enterprise, spend less time evaluating models and more time clearing organizational roadblocks. Get a budget owner. Get a security sponsor. Get data access sorted before you write the first prompt.

Process beats talent. Every time.

I know, I know. That sounds like something a consultant says to justify a strategy offsite. But having built AI features at a fintech company and watched a dozen teams try to do the same, the pattern is clear: teams that start with the model build demos. Teams that start with the problem build products.

The boring problem statement trick

Here’s a filter I’ve started using. Take your AI feature idea and describe it without mentioning AI, ML, LLMs, or GPT. If what’s left sounds like a useful product improvement, you probably have something. If what’s left sounds empty, you’re building a solution looking for a problem.

“We use GPT-4 to generate personalized financial insights” becomes “We show users relevant patterns in their spending.” The second version is a product. The first version is a technology demo. The second version also tells you what to measure: are the patterns actually relevant? Do users act on them?

At a fintech company, we almost fell into this trap. The initial pitch was “AI-powered transaction intelligence.” That means nothing. We reframed it as “reduce the time finance teams spend manually categorizing transactions from 4 hours to 20 minutes.” Now we had a measurable goal, a clear user, and a way to know if it worked.

Where AI actually helps

AI is good at tedious, pattern-heavy work where approximate answers are acceptable. Classification, summarization, drafting, extraction. It saves humans from the work they hate doing and are bad at doing consistently.

AI is bad at precision, accountability, and reasoning about edge cases. It’s bad at knowing what it doesn’t know. It’s bad at anything where “usually right” is worse than “always following a rule.”

The sweet spot for AI features in most products right now: draft-and-review workflows. The AI produces a first pass. The human reviews, corrects if needed, and approves. Both sides do what they’re good at.

The failure pattern: giving the model autonomy over decisions that matter. Auto-categorizing low-value transactions? Fine. Auto-approving expense reports over $10K? No. The line is about consequences. If the model gets it wrong, how bad is it? If the answer is “not great,” add a human checkpoint.

The five questions that kill bad ideas early

Before a team starts building an AI feature, I make them answer these. In writing. No hand-waving.

1. What user problem is this solving? Not “what could we do with AI” but “what is painful today, and who feels that pain?”
2. What does a wrong answer cost? If incorrect output has serious consequences, you need guardrails and fallbacks before you need features.
3. How do we measure success? Not usage. Not engagement. What concrete outcome improves? Time saved, errors reduced, revenue impacted.
4. What happens when the model is unavailable or uncertain? Your feature needs to work without AI. If it can’t, it’s too tightly coupled to a service you don’t control.
5. What’s our edge over someone else calling the same API? If the answer is “our prompts,” you don’t have an edge. Proprietary data, workflow integration, and feedback loops create defensibility. Prompt engineering doesn’t.

If any answer is weak, run a time-boxed experiment instead of a full build. Two weeks, clear success criteria, then decide.

The moat isn’t the model

I wrote about this in July, but it bears repeating because I keep seeing it: access to an API isn’t a competitive advantage. OpenAI’s API is available to everyone. So is Anthropic’s. So are the open-source models.

Your moat is the system around the model. The data you collect from user interactions. The workflow integration that makes ripping you out painful. The quality metrics that let you improve faster than competitors. The fallback system that means your product works even when the model doesn’t.

At a startup accelerator, I watched a batch of companies try to build moats around technology that was commoditizing in real time. The ones that survived had distribution or data advantages. The ones that died had clever technology and nothing else. Same pattern applies here.

Ship the ugly version

The final piece of advice is the least popular: ship something simple and ugly before you ship something impressive and polished. A feature that works reliably for 80% of cases with a clear fallback for the other 20% beats a feature that works brilliantly for 95% of cases and fails catastrophically for the other 5%.

Users forgive limitations when the product is honest about them. They don’t forgive confidence that turns out to be wrong.

Build the product. Not the demo.

The demo was impressive. You paste in a batch of transactions, the model categorizes them, and the output looks clean. The CEO loved it. The PM loved it. I loved it too, right up until we started testing edge cases.

A wire transfer labeled “SEPA CT REF-8847291 ACME GMBH” landed in “Entertainment.” A recurring subscription payment got labeled differently every time we ran it. And the model confidently categorized a clearly fraudulent transaction as “Regular business expense” without any hesitation.

This is the gap between demo and product, and it’s where most AI features die.

The thing nobody talks about: defining “good”

Before writing any production code, I made the team answer one question: what does a correct categorization look like, and what happens when it’s wrong?

For traditional features, this is obvious. A button either works or it doesn’t. A calculation is either right or wrong. For AI features, “right” is fuzzy. Is 85% accuracy good enough? Depends. If you’re categorizing expense reports for internal review, maybe. If you’re categorizing transactions for regulatory reporting, absolutely not.

We built an eval set of 200 transactions with hand-labeled categories. Not exciting work. Took the team three days. But that eval set became the foundation for every decision that followed: prompt changes, model selection, fallback logic, launch criteria.

The rule I enforce now: if you can’t write down what “good” looks like in concrete examples before you start building, you’re not ready to build.

Architecture for uncertainty

AI features sit inside the same product architecture as everything else. But they need extra layers to manage the fundamental uncertainty of probabilistic output. Here’s what the production feature stack actually looked like:

Input validation. Transactions go through a normalizer before the model sees them. Strip reference codes, standardize currency formats, expand abbreviations. The cleaner the input, the more consistent the output. This sounds boring. It improved accuracy by 8 points.

The model call. GPT-3.5-turbo with a tight prompt, structured JSON output, and a confidence score. We tried GPT-4 initially: better accuracy, but 10x the cost and 3x the latency. For this use case, 3.5-turbo plus good input normalization was the better trade-off.

Output validation. Every categorization gets checked against the valid category list. If the model returns a category that doesn’t exist (and it does, about 2% of the time), we fall back. If the confidence score is below our threshold, we fall back.

The fallback. This is the part most teams skip. Our fallback is a rules-based categorizer that handles ~40% of transactions using keyword matching and counterparty lookup. It’s not as good as the model, but it’s deterministic and always available. When the model is uncertain, the user gets the rules-based result with a flag saying “review suggested.”

Feedback loop. Users can correct categorizations. Those corrections feed back into our eval set and, eventually, into prompt improvements. This is the part that compounds over time.

Testing probabilistic systems

Unit tests cover the deterministic parts: input normalization, output validation, the rules-based fallback. Those work exactly like traditional testing.

Model behavior gets tested differently. We run the full eval set (200 transactions) on every prompt change and weekly against the live model. The metrics we track:

• Accuracy against our labeled set (target: >88%)
• Category distribution (catches when the model starts favoring certain categories)
• Confidence score distribution (catches when the model becomes less certain overall)
• Fallback rate (how often we’re bypassing the model)

We don’t assert on individual outputs. That’s a trap. The same input might get slightly different wording each time, and that’s fine as long as the category is right. We assert on aggregate metrics across the eval set.

One thing that bit us: OpenAI changed something in the model (not a version bump, just inference-time behavior) and our accuracy dropped 3 points overnight. We caught it because we run the eval set daily. Teams that don’t do this are flying blind.

Launching without embarrassment

We launched to 5% of users first. Internal users, specifically the finance team. They’re the harshest critics and the most forgiving audience: they understand the constraints and give precise feedback.

Two things came out of the soft launch that we hadn’t anticipated:

1. Multi-currency transactions confused the model more than we expected. A USD payment from a EUR account got categorized based on the currency, not the merchant. We added currency normalization to the input pipeline.

2. Users didn’t trust the output even when it was correct. Adding the confidence score as a visual indicator (“high confidence” / “review suggested”) dramatically improved trust. People are fine with AI output when they know the system is honest about its uncertainty.

After two weeks with the finance team, we expanded to 50%, then 100%. The feedback loop was running, the eval metrics were stable, and the fallback was handling edge cases gracefully.

What I’d tell another team

Define quality before you write code. Build the eval set first. It’s tedious, and it’s the most important thing you’ll do.

Design the fallback before the happy path. What happens when the model is wrong or unavailable? If that experience is terrible, your feature is fragile. If it’s graceful, you can ship confidently.

Instrument everything. Not just errors and latency. Quality metrics, confidence distributions, fallback rates, user corrections. You need to see the model’s behavior as a continuous signal, not a binary pass/fail.

Resist the demo. The demo is a trap. It shows the best case. Production shows the average case and every edge case you didn’t think of. Ship the demo only after you’ve built the scaffolding that makes the average case acceptable and the worst case survivable.

AI features are software. Treat them that way, with tests, rollout controls, monitoring, and healthy skepticism. The model is the least interesting part of the system. The interesting part is everything around it.

I’ve founded two startups. I’ve been through a deep-tech founder program and a startup accelerator. I know what a defensible business looks like and what a science fair project looks like. Most of what’s being built right now is the latter.

The wrapper problem

Here’s the test I apply: if OpenAI adds your feature to ChatGPT next Tuesday, do you still have a company? If the answer requires more than three seconds of thought, you’re a wrapper.

Wrappers are easy to build, impressive in demos, and worthless when the platform moves. I saw this exact pattern in mobile (2012-2013), chatbots (2016), and crypto (2021). The playbook is always the same: new technology enables easy prototypes, hundreds of startups launch, the platform or an incumbent absorbs the feature, and the startups die.

The AI version is faster because the prototyping is faster. You can build a convincing demo in a weekend. That’s the trap.

Where the actual value is

The startups that will survive have at least one of these:

Proprietary data loops. If every user interaction makes the product better in a way competitors can’t replicate, you have something. This is boring and slow to build. Good.

Deep workflow integration. If ripping you out requires a migration project, switching costs protect you. This means being embedded in existing processes, not sitting as a standalone tool people can ignore.

Domain expertise that reduces error rates. In regulated industries – finance, healthcare, legal – the model being 90% right is a liability, not a feature. The value is in the last 10%, and that requires domain knowledge the model doesn’t have.

Distribution. If you already have the customers and you’re adding AI to an existing product, you win against the startup that’s trying to acquire customers and build AI simultaneously.

Notice what’s not on the list: “better prompts.” Prompt engineering isn’t a moat. It’s barely a speed bump.

What I’d actually build

If I were starting a company today – and I think about this more than I should – I’d focus on the infrastructure layer. The tooling for evaluation, observability, and cost management. The picks-and-shovels play during a gold rush is a cliche for a reason: it actually works.

Or I’d go deep vertical in a domain I know. Fintech, specifically. (I’m biased – I’ve spent years in the space.) The financial services industry has massive data complexity, real compliance requirements, and budgets. An AI product that can navigate GAAP, regulatory reporting, and multi-currency reconciliation isn’t getting replaced by ChatGPT anytime soon.

The honest take

Most AI startups being funded right now won’t exist in 18 months. Not because AI isn’t real – it absolutely is. But because having access to an API isn’t a business. The companies that survive will be the ones that did something hard with the technology, not something easy.

If your pitch starts with “we use GPT-4 to…” you’ve already lost. Start with the problem. The model is a detail.

Stop pretending tech debt will fix itself. Put it in a registry, score it, and schedule it like real work. We did this at the fintech startup and it turned a chaotic backlog of “we should really fix that” into something we actually ship against every sprint.

Every engineering team I’ve worked with has some version of the same conversation. Someone mentions a gnarly part of the codebase. Everyone nods. Someone says “we should really clean that up.” Nobody does.

At the fintech startup, we hit a point where this was genuinely hurting us. Our fintech data pipeline had accumulated enough shortcuts and half-finished migrations that feature work was getting slower every quarter. Not dramatically. Just enough friction that estimates kept creeping up and nobody could point to exactly why.

So I built a system for it. Nothing fancy. But it changed how we think about debt.

What tech debt actually is

Tech debt is a future cost you created with a present decision. Sometimes that decision was smart – you shipped faster and the tradeoff was worth it. Sometimes it was accidental – the design was fine until requirements shifted. Either way, the cost is real.

The forms we see most often:

• Deliberate shortcuts. You knew it was a hack. You shipped anyway. Fair enough.
• Accidental debt. Looked fine at the time. Scale or new requirements proved otherwise.
• Environmental shifts. A dependency gets deprecated. A compliance rule changes. Not your fault, still your problem.
• Operational gaps. Missing monitoring, thin tests, no runbook. The kind of thing that bites you at 2am.

Our debt registry

Here is what actually worked for us. We created a debt registry – basically a shared list of known liabilities that lives right in our issue tracker. Not a separate doc. Not a wiki page nobody reads. Same board, same sprint planning, same visibility as feature work.

Each entry looks roughly like this:

id: TD-021
title: Legacy auth flow lacks rate limiting
area: auth-service
impact: Security, reliability
risk: High
effort: 3-4 weeks
owner: Platform
status: Proposed

The key insight was treating debt entries as first-class work items. When debt lives in a separate spreadsheet, it gets ignored. When it sits next to feature tickets in the same planning session, it gets discussed.

We review the registry every two weeks during sprint planning. Takes ten minutes. That alone changed how we worked.

Scoring: keep it dead simple

We tried complex scoring matrices. They didn’t survive contact with reality. What stuck was three dimensions and a formula you can do in your head:

• Impact. How much does this slow us down or degrade the product?
• Risk. What happens if it gets worse or fails outright?
• Effort. How much work to fix it?

priority = (impact * 2) + risk - effort

Each on a 1-5 scale. Is it perfect? No. But it forces you to compare things that otherwise feel impossible to rank. “The auth system is scary” versus “the build is slow” suddenly becomes a conversation with numbers instead of gut feelings.

When to act now versus later

Some things can’t wait:

• Security exposure. Full stop.
• The same root cause behind repeated incidents.
• A dependency about to hit end-of-life.
• Workarounds that tax every single release.

Everything else goes into the prioritized backlog. High impact, low effort items get picked up opportunistically. Large refactors get scheduled as dedicated work with clear milestones.

How we schedule paydown

We tried three approaches before settling on a hybrid:

Capacity allocation. We reserve roughly 20% of each sprint for maintenance and debt work. Non-negotiable. Product knows about it. This is the baseline.

Debt sprints. Every six weeks, we run a focused sprint on the highest-priority debt items. Engineers pick from the top of the registry. These sprints have been some of the most satisfying work the team does.

Opportunistic paydown. If you’re already in the code, improve it. Boy scout rule. We just ask people to tag the cleanup commits so we can track the effort.

Incremental migration over big rewrites

We learned this the hard way. Big-bang rewrites fail. They just do. Every large debt item at the fintech startup now follows the same pattern:

• Run old and new paths in parallel.
• Migrate the highest-traffic paths first.
• Set a cutover date. Remove old code on that date. No exceptions.

Feature flags make this safe:

if flags.new_auth_flow:
    return new_auth.authenticate(user)
return legacy_auth.authenticate(user)

The flag gives you a kill switch. Sleep better at night.

Selling debt work to non-engineers

This is where most teams fail. You can’t walk into a planning meeting and say “the auth code is messy.” Nobody cares.

What works: translate debt into business language. “Feature X takes twice as long to ship because every change requires manual regression testing in the auth module.” Now it’s a delivery speed conversation. Product managers understand delivery speed.

At the fintech startup, we started including debt impact in our sprint velocity reports. When the team could show that velocity dropped 15% quarter-over-quarter and tie it to specific debt items, getting time allocated stopped being a fight.

Prevention

A few guardrails go further than any cleanup sprint:

• Lightweight design reviews for anything touching critical paths.
• Code review checklists that ask about maintainability, not just correctness.
• Quarterly dependency upgrades so you never face a multi-year jump.
• Documentation expectations for anything another team will touch.

Closing

Tech debt isn’t a failure. It’s an accounting problem. You took a loan, now you need a repayment plan.

The registry changed everything for us. Not because it was sophisticated – it was a handful of fields in Jira. But because it made invisible costs visible. And once everyone can see the cost, the conversation about paying it down gets a lot easier.

Multiply pain by how often you touch it. Fix the top. Schedule the middle. Ignore the rest guilt-free.

The Problem With “We Should Really Fix That”

At the fintech startup we had a spreadsheet. Forty-seven items on it. Stuff ranging from “our deploy script is held together with duct tape” to “that one table column named data2.” Every retro, someone added more. Nothing came off.

The list was useless because everything on it felt important to whoever wrote it down. No ranking. No shared language for severity. Just a growing monument to good intentions.

So I built a framework. Dead simple. Two numbers.

Two Numbers, One Multiplication

For every piece of tech debt, score two things on a 1-to-5 scale:

Impact – how much does this actually hurt? A 5 means outages, data bugs, or features you literally can’t ship. A 1 means it annoys you when you see it but has zero customer-facing consequence.

Change frequency – how often does someone touch this code? A 5 means daily. A 1 means you forgot the directory existed.

Multiply them. That’s your priority score.

priority = impact x change frequency

• 15-25: Fix this soon. Like, put it in the next sprint soon.
• 8-14: Schedule it. Quarter planning, debt sprints, whatever your cadence is.
• 1-7: Accept it. Seriously. Move on.

Running the Numbers on Real Stuff

I’ll use examples close to what we actually triaged at the fintech startup.

Our user service had no tests and we were shipping changes to it three times a week. Impact 4, frequency 5. Score: 20. Obviously top of the list.

The deploy process was painful and slow. Impact 4, frequency 4. Score: 16. Right behind it.

We had a gnarly payment integration that scared everyone, but we only touched it once a quarter. Impact 5, frequency 2. Score: 10. Scary, but not urgent.

An admin dashboard that loaded slowly? Impact 2, frequency 2. Score: 4. Nobody cared enough. And that was the correct call.

The math removes the emotion. That’s the point.

Not All Debt Ages the Same

This matters and most people miss it. Some debt compounds. Every time you work around the problem, the workaround becomes the new baseline. The next person works around the workaround. You know exactly what I’m talking about.

Other debt just sits there. Ugly, stable, inert. The weird naming convention in a module nobody touches? It’ll be weird next year too, but it won’t be worse.

And some debt evaporates. We had a messy integration with a third-party API we were planning to drop. Spending time cleaning it up would have been pure waste.

When you’re prioritizing, ask: is this getting worse? Compounding debt should jump the queue even if the current score is middling.

Four Ways to Actually Pay It Down

Boy Scout rule. You’re already in the file. Leave it slightly better. Rename that variable. Extract that function. This handles the small stuff and keeps entropy from winning.

Protected time. Block real capacity for debt work. We did one day a week. Some teams do a cooldown sprint after each release. Doesn’t matter what rhythm you pick, what matters is that the time is sacred and not the first thing cut when a deadline looms.

Bundle it with features. If a feature touches a debt-heavy area, pad the estimate to include cleanup. Product managers accept this more easily than standalone debt tickets because the work is tied to something they already want.

Bite the bullet. Some debt needs a dedicated rewrite. A subsystem replacement. A migration. These need a business case, an owner, and a timeline. Not a Jira ticket that says “refactor payments” with no assignee sitting in the backlog for eight months.

Selling Debt Work Upward

Engineers talk about debt in terms of code quality. Leadership doesn’t care about code quality. They care about shipping speed, incident frequency, and hiring retention.

So translate. “This area has no tests and we ship to it constantly, which is why we’ve had three production incidents this quarter” lands differently than “we need to improve test coverage.” Same problem. Different framing.

At the fintech startup I started tying debt items to incident reports. When leadership could see a direct line between a piece of debt and a customer-facing problem, the prioritization conversation got a lot shorter.

Accepting Debt Is a Decision, Not a Failure

Here’s what changed once we had the scoring system: we stopped feeling guilty about the bottom of the list. A score of 4 means you’ve looked at it, evaluated it, and decided your time is better spent elsewhere. That’s not neglect. That’s judgment.

Low-score debt in a system you’re replacing? Ignore it. Cosmetic issues in stable code? Ignore them. Theoretical problems with no evidence of real pain? Keep an eye on them, but don’t burn a sprint.

The Takeaway

Forty-seven items became six that mattered. The rest we either accepted or scheduled for later with clear triggers for when to revisit. The team stopped arguing about what to fix because the math settled it.

Two numbers. One multiplication. That’s the whole framework.

I learned this the hard way at the fintech startup. We needed a serious infrastructure overhaul. I knew it. My team knew it. But our investors didn’t speak in uptime and throughput. They spoke in money. So I had to learn their language.

Talk money or don’t talk

Every infra pitch boils down to one of four things: it makes money, saves money, reduces risk, or lets us ship faster. That’s it. Pick the frame that fits and lead with it. Not with the technology. Not with the architecture diagram. With the business outcome.

At the fintech startup, we were heading into a period where marketing had big traffic projections. Our system couldn’t handle it. I didn’t pitch “we need to re-architect our data pipeline.” I pitched “we’ll lose paying users in Q3 if we don’t act now.” Different sentence. Same project. Completely different reaction in the room.

Risk is the easiest sell

When downside is obvious, risk framing writes itself. Quantify the probability of an outage. Estimate the cost. Multiply. That number gets attention fast because nobody wants to be the one who ignored a preventable disaster.

Growth constraints work too, but you need credible projections. Vague “we might need to scale” doesn’t cut it. Show the wall you’re about to hit and the revenue sitting on the other side.

Velocity framing is underrated. If your deploys take two hours and engineers sit idle waiting, that’s burned salary. Translate it. “We’re losing 40 engineer-hours a week to deployment bottlenecks” hits harder than “our CI/CD pipeline needs work.”

Do the math, even if it’s rough

A proposal without numbers is just an opinion. I’ve seen smart engineers lose budget requests because they couldn’t answer “what does this cost us today?” Document the current state in dollars and hours. Then describe what changes, in the same units.

Keep the ROI dead simple. Investment. Expected annual benefit. Payback period. If your assumptions are honest and explicit, rough math beats no math every single time.

Structure for skimmers

Executives don’t read proposals. They skim them. One paragraph up top: what you want, why, and what it’s worth. Then a short problem statement in business terms. Proposed solution with minimal technical detail. Quantified impact. Cost. Timeline. Risks and mitigations. Alternatives you considered and rejected.

That order matters. Bury the ask on page three and it never gets read.

Where I’ve seen pitches die

Being too technical. Your CFO doesn’t need to understand container orchestration. They need to understand what it changes for the business.

Skipping quantification. “Better reliability” means nothing. “$200K in avoided incident costs per year” means something.

Ignoring tradeoffs. If this delays a feature, say so. Explain why the trade is still worth it. Pretending there’s no cost makes you look naive.

Asking for everything at once. Phase it. Get a win. Use that win to fund the next phase.

And the killer: not following up. If you don’t report results after getting funded, you’ve just burned your credibility for the next ask.

Keep the conversation going

Don’t wait for budget season. Share metrics regularly. When an incident happens, quantify the damage. When delivery speeds up after an investment, show the numbers. Make infrastructure value visible all the time, so the next request feels like an obvious yes instead of a surprise ask.

Infrastructure is a business investment. Treat it like one and it gets funded. Treat it like a technical hobby and it doesn’t.