Measurement
AI evaluation (eval)
Also called: eval, model evaluation
Definition
An AI evaluation, or eval, is a repeatable test that measures whether an AI system produces a better outcome than the alternative it replaced, against a defined baseline.
An eval is not a demo and not a dashboard. It is a structured comparison: a fixed set of real cases, a scoring method decided in advance, and a baseline that could plausibly win. Evals answer the only question worth funding — did this change the result — rather than the easy questions of whether the system is live or whether people use it. Without one, every AI program is a belief, not a measurement.
What separates an eval from a benchmark
A benchmark scores a model against a public dataset that nobody in your company is paid to care about. An eval scores a system against your work: your claim notes, your submissions, your exception queue. Benchmarks tell you which model is generally strong across a wide range of public tasks that may bear little resemblance to your operation. Evals tell you whether this specific deployment, running on this specific data, earns its cost in your operation this quarter. A vendor's benchmark score is marketing collateral; your eval result is the number a CFO can actually act on, because it was built from the cases you would otherwise have paid a person to decide.
The three claims an eval keeps apart
Operational means the system is live and has not fallen over. Adopted means people use it and the usage line climbs on a dashboard somewhere. Outcome means the decision it touches is measurably better than before, on a metric someone outside the build team agreed to in advance. Only the third is worth paying for, and only the third is hard to fake — most measurement programs quietly stop at the second because it is the easiest one to report favorably. An eval forces the conversation past adoption and into whether the underlying decision actually improved, which is the question sponsors meant to ask at the start.
Who should own the eval
Evals lose credibility fast when the team that built the system also grades it, because incentives and blind spots both point the same direction. The strongest programs assign scoring to a group with no stake in the launch — risk, actuarial, or an independent operations reviewer — and keep the case set away from the builders until scoring is complete. That separation is uncomfortable and slower than self-grading, but it is the only version of the number that survives a skeptical board question or an auditor's request to see how the result was produced.
In practice
- Write the failure number before you build: the result that would make you turn it off
- Choose a baseline that could beat you — the rule, the spreadsheet, the experienced adjuster
- Freeze a case set and score it the same way every release, so drift is visible
- Report lift against the baseline, not activity against zero
- Assign scoring to a team with no stake in the launch
