Your Real Token Price Is a Cache Hit Rate

This summer’s price cuts made good headlines, and one widely shared pricing tracker drew the conclusion the headlines invited: the mid-tier models have converged to within cents of each other on paper, and for chat-shaped workloads the comparison really is that simple now.

Agent workloads broke that comparison. An agent loop is not a stream of fresh prompts; it is the same long prefix (system prompt, tool definitions, accumulated transcript) resent on every iteration with a thin layer of new tokens on top. A twenty-step run can send two million input tokens of which forty thousand are new. Vendors price the two categories differently: a cached input token costs a fraction of a fresh one, on the order of a tenth on current price pages, though the ratio and the cache-write charge vary by vendor, and you should read yours. Two numbers then govern the bill: the share of your input that is stable prefix, and the hit rate you actually achieve on that share. Multiply them and you have the fraction of your input traffic that rides the discount. Everything else pays list price.

Scope this claim before leaning on it, because cost per governed workflow is still the number that governs , and in plenty of workflows output tokens, retries, or the human review step dominate while cache mechanics are a rounding error. The claim is narrower and sharper: for input-heavy agent loops, the workloads whose volume is why your bill outran the price cuts, the effective input price is set by that product of prefix share and hit rate, and once headline prices sit within cents of each other, it can decide which vendor is actually cheaper. A workflow at 95% stable prefix and a 90% hit rate can beat the same workflow at 40% on a nominally cheaper model. Whether it does is a one-line break-even check against the fresh-price gap, and the point is that nobody runs the check because nobody knows their hit rate.

They don’t know it because the hit rate is not a fact about the vendor; it is a property of engineering discipline. Caches key on exact prefixes, so anything that mutates the front of the context silently zeroes the discount: a timestamp in the system prompt, tool definitions that reorder nondeterministically, retrieval injected before the stable block instead of after. Cache windows expire on the vendor’s clock, so an agent that idles between steps can pay fresh price for the whole transcript again. In an agent loop, a cache miss is a price increase caused by context drift, and it is invisible in a bill that reports token totals. The usage APIs report cache reads and writes separately; if that split isn’t on a dashboard, the discount is unmanaged.

Then there is the number to know before you need it: the cold-start premium. The cache is provider-scoped. Your prompts and tool schemas travel to a second vendor; the warm cache state does not. The premium is real but it is not a fixed wall. It depends on how many distinct prefixes you run, how much traffic re-warms them, and the vendor’s cache window. High-traffic workflows with few prefixes re-warm in hours; long-tail workflows with thousands of session-specific prefixes keep missing indefinitely. Which is why the cold-start premium should be a measurement, not an estimate: replay a day of production traffic against your second-choice vendor and record cost per workflow, cold and steady-state. If you drill your backup at all , this number falls out of the drill for free.

Three lines on the cost dashboard, per input-heavy workflow. Cached share of input tokens, token-weighted: cached reads over total input. Blended input price, computed from fresh, write, and read charges, tracked against the headline price to show the gap you’re managing. And the measured cold-start premium on the alternate vendor, refreshed whenever the drill runs. The first two catch the one-line regressions, like the timestamp someone adds to a system prompt in March. The third prices your exit while it’s still a Tuesday-afternoon fact, not a mid-incident discovery. The price pages converged this summer. Your effective prices didn’t, and the distance between those two statements is a ratio you can start reporting this week.