How to build an eval that survives production

We spend a lot of these pages arguing that you should measure the decision rather than the deployment — that a green adoption number tells you almost nothing about whether a system is any good. It is a fair challenge to turn back on us: fine, then what does a good measurement actually look like? This is the constructive half of that argument. When we build an evaluation for a system that informs a consequential decision, here is what we are trying to get right, and why each choice matters more than it appears to.

None of it is exotic. Most of it is the refusal to skip steps that are tedious and easy to skip. That refusal is the whole craft.

Start from the decision, not the model

A good eval doesn’t begin with the model’s outputs. It begins with the decision the system exists to support: what action does its output trigger, who is accountable for that action, and what would make the action right or wrong? Write that down first, in plain language, before any metric is chosen. Weak evals skip this and default to whatever the model conveniently emits — fluency, similarity to a reference answer, a benchmark score — and then spend months optimizing a number that was never connected to whether the decision improved. If you can’t state the decision and the outcome that counts as good, you aren’t ready to measure anything yet. You’re only ready to generate a number.

Define “good” as an outcome with a cost on being wrong

“Accuracy” is rarely the right target, because it treats every error as equal, and almost no real decision does. A good eval names the two ways the system can be wrong and weights them the way the business actually feels them. Approving what you should have denied and denying what you should have approved are different mistakes with different costs; so are a missed fraud and a falsely accused customer, a false alarm and a silent failure. Encode that asymmetry directly in the score. An eval that optimizes balanced accuracy on a decision where one error costs ten times the other is optimizing — precisely, confidently — for the wrong world.

Build the set from the cases that matter, not a random sample

The instinct is to evaluate on a representative sample. For consequential systems that instinct is wrong, because representative samples are dominated by easy cases, and easy cases are not where the system earns or loses its keep. A model can score 95 percent overall by being 99 percent right on the routine majority and 60 percent right on the contested, high-stakes minority — and that minority is the entire reason you bothered to measure. Oversample the hard cases on purpose: the ambiguous, the adversarial, the expensive, the ones a careful human would slow down on. Score them as their own number and look at it first. A blended figure that hides the consequential slice isn’t a measurement; it’s camouflage.

Anchor to ground truth you would defend

A score is only as trustworthy as the truth it’s measured against. The best ground truth is reality itself: did the prediction come true, did the flagged claim turn out fraudulent, did the approved loan perform. Where reality is too slow or too costly to wait for, the next best is human adjudication you would stand behind — experts judging cases under a written protocol, with their rate of agreement measured, so you know how solid the “truth” actually is before you grade a model against it. Be honest about the cases where ground truth is genuinely contested, and design for that rather than papering over it. The failure to avoid is the quiet one: comparing the model to a convenient proxy and quietly promoting the proxy to truth.

Measure the system you’ll run, under the conditions it’ll face

Evaluate on inputs that look like production, not like the pilot — messy, incomplete, out of distribution, occasionally adversarial. Inject the cases you are afraid of: the manipulated input, the format you’ve never seen, the edge the demo skipped, because those are the ones that will actually arrive. An eval built on clean, curated data certifies a system that exists only in the slide deck. The point of the exercise is to find the failures before your customers do, which means the eval has to contain them on purpose.

Run it forever, not at launch

The most expensive mistake is treating the eval as a gate you pass once. The system learns, the world shifts, the inputs drift, and the thing you validated in spring is not the thing running in autumn. A good eval is live instrumentation: a held-out stream, continuously sampled and re-adjudicated, with alarms on shifts in the output distribution and the outcome rates. You are not certifying a system; you are watching one that moves. Budget for that the way you would budget for any monitoring that matters — because the alternative is learning about the drift downstream, in a number nobody thought to connect to the model.

Decide the response before you see the number

An eval that produces a result but no committed action is theater, and a surprisingly common kind. Before the score comes back, decide what each outcome triggers: at what level do you ship, hold, roll back, or route to a human? Write the thresholds down in advance, when nobody is yet invested in a particular launch date — because thresholds set after the number arrives have a reliable way of bending to meet it. The discipline of a good eval was never the measurement. It’s the decision you pre-committed to make when the measurement disappoints you.

Cheap enough to run, legible enough to trust

Two practical constraints quietly decide whether any of this survives. If the eval is so expensive that it runs once a quarter, it will not catch drift, and people will route around it. If it is so opaque that the accountable decision-maker can’t read the result, it will not be trusted, and people will overrule it on instinct. A good eval is cheap enough to run on every meaningful change and legible enough that the person who owns the decision can look at the output and know what to do — including, when it’s warranted, the decision to stop. Precision no one can afford or understand isn’t precision. It’s just a second system to maintain.

The work is the refusal to skip steps

Read back over the list and notice that almost none of it is technically hard. Naming the decision, costing the two errors, hunting down the cases that matter, defending the ground truth, committing to the thresholds before the number lands — these are not feats of machine learning. They are the unglamorous, skippable steps, and skippable steps get skipped, especially under a deadline, especially when a single headline number is what got asked for. The reason the eval discipline gap is so common is not that good evaluation is a mystery. It’s that a real measurement is more work than a reassuring one, and the reassuring one is usually what’s due Friday. A measurement worth trusting is simply the one you built deliberately — before you needed it to say yes.