The Benchmark You Didn't Build

Public benchmarks are useful, and the genre that tells you to ignore them is wrong. A model sitting twenty points behind the field on a task that resembles yours will not clear your bar either, so you can drop it without spending a cent. A leaderboard is a cheap negative filter and a rough ceiling on capability. What it cannot do is separate two strong candidates on the work that pays you: the gap between first and third place is usually inside the noise, and the test set has likely leaked into someone’s training run. Shortlist on the public number. Decide on your own. A benchmark you did not build is a benchmark someone else optimized for.

None of that is new, which is the actual problem. “Build your own evals” is the most recycled advice in this field, and plenty of teams take it, stand up a set of real cases, and quietly stop trusting it within a quarter. The failure is never collecting the cases. It is the part everyone skips: who scores them, on every change, without going broke or going blind.

The cost trap

Pull thirty to a hundred real cases from your logs and weight them toward the unhappy paths. That part is easy. Then someone has to grade the outputs, and if that someone is a human reviewer working every model, prompt, and routing change, the set runs once a quarter and becomes a ritual instead of an instrument . Re-running a hand-graded set on every change is the cost objection that kills most owned-eval programs. It is real, and “unglamorous and cheap” does not answer it.

Tiering does. Most cases have a checkable answer: a required field, a JSON shape that must parse, a string that must or must not appear, a number within tolerance. Encode those as programmatic assertions. They are deterministic, free, never drift, and run in CI on every commit. In practice sixty to eighty percent of a real case set collapses to assertions. Only the residue needs judgment: was the summary faithful, was the refusal correct.

For that residue, use an LLM as judge, but treat the judge as versioned code, not a vibe. Pin the judge model, freeze the rubric, anchor it with a few labeled examples in the prompt. Now the expensive layer costs cents per case and runs on every change. Humans grade the grader, not the traffic: keep a small gold set you labeled by hand, and on a schedule (and whenever the judge’s model version moves) measure judge-versus-human agreement. When agreement slips, fix the rubric before you trust another run. Auditing the judge instead of the model is what makes the cheap layer safe to lean on.

The noise trap

A set of thirty cannot tell a four-point regression from sampling noise. At an eighty percent pass rate over thirty cases, the standard error on the aggregate is about seven points, so any single-digit move is meaningless. You will chase ghosts or ship regressions while admiring a stable number. The aggregate is the wrong statistic. Run the old and new model on identical cases and read the per-case flips: which cases passed before and fail now. Three clean regressions out of thirty is a signal even when the headline barely moves, because each case is compared to itself rather than to an average. Run each case a few times so model nondeterminism does not masquerade as a change. Pairing, not volume, gives a small set teeth.

What it buys

The payoff is narrow and concrete. A new model tops a public set, clears your shortlist, and your eval shows two regressions on the exact request class that quietly generates support tickets. You hold it back, with receipts, and you can tell the board why the number did not move on your terms instead of the vendor’s. That is the whole return: not a better score, an argument you can defend. Separate it from cost per correct answer , which is a different decision, and the eval set becomes the one measurement a vendor cannot tune.