Measurement
Eval discipline gap
Definition
The eval discipline gap is the distance between knowing that an AI system is being used and knowing whether it produced a better outcome than what it replaced.
The gap opens because adoption is cheap to measure and pleasant to report, while outcome is slow, difficult, and often unflattering. Organizations end up able to state usage to two decimals and unable to state value to any precision at all. It persists for structural reasons — vendor incentives, promotion for shipping, presence-shaped leadership questions — not because anyone is careless. Closing it requires changing what gets asked in review, not just what gets measured after the fact.
Why it survives good intentions
Every force that produces the gap is a sane local response. A vendor demo proves capability, never lift, because that is the easiest thing to show a buyer in forty-five minutes. Teams are rewarded for rollouts, not for the cleaner result of running an honest test and shutting a project down before it wastes another quarter. And measurement genuinely is hard when nobody instrumented the old process either, so there is no baseline waiting to be compared against. Exhortation does not close a structural gap; a different default habit does, applied consistently across every program regardless of how promising it looks at launch.
How to tell if you have one
Ask any AI program owner for the number the project was meant to move, the baseline it was compared against, and the result to date. If the answer arrives in the vocabulary of activity — models in production, seats filled, tickets handled, prompts sent — the gap is open and probably wide. A program with real discipline can answer in one sentence: the metric, the comparison, and the delta, with a source you could go check. Most cannot produce that sentence on the spot, which is itself the diagnosis, regardless of how much the program has spent or how many systems it has shipped.
What closing it actually requires
Closing the gap is a governance change before it is a technical one. It means funding measurement in the original budget rather than as an afterthought, staffing an owner whose job is explicitly to know the outcome number, and rewarding a documented shutdown as visibly as a documented launch. None of this requires new tooling; most organizations already have the data warehouse and the analysts to build an eval. What is missing is the standing expectation that every AI spend answers the outcome question on a schedule, the same way a capital project answers a payback question before it is approved.
In practice
- Replace 'are we doing AI' with 'is this decision better because of AI' in review agendas
- Fund measurement in the same budget line as the build, never as a later phase
- Make shutting a project down on evidence a reportable win
- Name one owner per program whose job is the outcome number, not the rollout
