Grading the wrong exam

The study everyone is quoting has more or less settled the question of enterprise AI ROI for a great many boardrooms this year: the overwhelming majority of pilots produce no measurable return. We are not going to dispute the direction of it, though we would hold the precise figure more loosely than most of the people quoting it do — a number that travels this fast is rarely doing so on the strength of its methodology. Enormous sums went into work that produced demos, dashboards and a considerable amount of internal enthusiasm, and never touched a line on any statement. The executives who read the figure and started asking harder questions about what their programs had actually delivered did the right thing, and did it faster than we expected. Our argument is not with them. It is with the figure they were handed to ask those questions with, which turns out to answer a different one — and, as we will get to, quietly discourages the very work that would improve the number. We have argued adjacent things ourselves — the absence of real measurement is close to the whole of our view on why so much of this work disappoints.

But the figure is measuring the wrong object, and the way it is wrong matters more than whether it is right.

What enterprise AI ROI measures

The statistic grades pilots as investments. Money in, money out, the difference is the verdict. That is a reasonable way to grade a deployment. It is the wrong way to grade a pilot, because the honest function of a pilot is not to return money. It is to return information.

Consider a pilot that ran for a quarter and established, clearly and early, that the workflow could not support automation — the inputs were too irregular, the exceptions were the job, and the people doing it were making judgment calls nobody had ever written down. Judged as an investment, that pilot failed. It goes in the denominator. Judged as information, it may be the most valuable thing the organization bought that year, because it prevented a far larger commitment to something that was never going to work. The number records that as a loss. It counts learning as loss.

The numerator is worse

What the statistic counts as success is the more serious problem. A pilot that produced a clean, measurable throughput gain lands in the numerator — a win, a case study, a thing to scale. But throughput against what? Most pilots are graded against whatever number was already on the dashboard, and dashboard numbers are proxies chosen for convenience, years ago, by people solving a different problem. A pilot that moves one of them has demonstrated that automation pursues the metric it was given, which was never in doubt. It has not demonstrated that the metric was the goal. Some of the most expensive mistakes we have watched up close were pilots that scored beautifully and were scaling the wrong behavior faster — and the cleanest example is the automation that books its saving in one quarter while the correction work it quietly removed surfaces, uncosted, three quarters later.

So the failures in that ratio include the pilots that taught the organization the most, and the successes include the pilots that will cost it the most. That is a strange instrument to run a capital allocation process on.

Most pilots asked nothing

The real indictment is harsher than the return framing, and more damning. Most pilots produce no information either.

They were built without a question. Someone secured budget, picked a workflow that was politically available rather than analytically interesting, stood the thing up, and ran it for a quarter. At the end there was a readout. It seemed to work. People liked it. Sentiment was positive, let us expand. What came back was a vibe, and a vibe is not an answer, because no question was ever asked. Nobody wrote down in advance what they would have to observe to conclude the thing should be scaled, or what they would have to observe to conclude it should be killed. Absent those two sentences, every possible outcome is compatible with continuing.

That is not a failed investment. It is a failed experiment, which is a different defect and a far more fixable one. An organization that loses money on a pilot has lost money. An organization that learns nothing from a pilot has lost the money and the option.

The chill it creates

Then there is the damage the statistic is doing on its own account. Organizations frightened by it are quietly getting out of the pilot business and asking instead for deployments with a guaranteed return — a business case up front, a committed number, a sponsor who has staked something on it working. The predictable consequence is that only the safest and most obvious automations get attempted. Document handling. Ticket routing. The work whose payback can be modeled in a spreadsheet before anyone goes near it.

Those are fine, and they are also, almost by definition, where the value is thinnest — because value concentrates in a small number of consequential decisions, and consequential decisions are precisely the ones nobody can guarantee a return on in advance. Demand certainty before exploration and the consequential cases never get examined at all. The figure scares organizations away from exactly the exploration that would fix the problem it describes.

Pre-register the question

The correction is not more pilots, or fewer. It is writing the question down before the pilot starts.

Two sentences, agreed in advance by the people who will read the result: what we would have to observe to conclude this should be scaled, and what we would have to observe to conclude it should be killed. Both have to be falsifiable. Both have to be observable inside the window the pilot actually has. And the kill condition has to be one somebody would genuinely act on, which is the part that gets negotiated away when it is written afterward. A pilot that can only succeed is not a pilot. It is a procurement with a longer runway.

The rest is making sure the measurement outlives the demo. An evaluation that still works once the system is in production is what turns a quarter of anecdote into something you can act on twice — the first time when you decide whether to scale, the second time when the conditions change and the original answer quietly stops being true.

The figure is not wrong so much as it is grading the wrong exam. Most pilots did fail, and they failed to return an answer rather than to return money — the missing return is downstream of the missing answer, not the other way around. An organization that fixes its returns without fixing its questions will only get better at funding the automations it could already justify. Fix the questions and the return stops being the interesting number, because you will know each time what you bought: either a capability worth scaling, or a decision you no longer have to make blind. Both are worth the quarter. Only one of them shows up in the statistic.